* SYNOPSIS
Detect calibration boards in observed camera images

#+BEGIN_EXAMPLE
$ mrgingham /tmp/image*.jpg

# filename x y level
/tmp/image1.jpg - -
/tmp/image2.jpg 1385.433000 1471.719000 0
/tmp/image2.jpg 1483.597000 1469.825000 0
/tmp/image2.jpg 1582.086000 1467.561000 1
...


$ mrgingham /tmp/image.jpg |
  vnl-filter -p x,y |
  feedgnuplot --domain --lines --points --image /tmp/image.jpg

[ image pops up with the detected grid plotted on top ]


$ mrgingham /tmp/image.jpg |
  vnl-filter -p x,y,level |
  feedgnuplot --domain
              --with 'linespoints pt 7 ps 2 palette'
              --tuplesizeall 3 --image /tmp/image.jpg

[ fancy image pops up with the detected grid plotted on top, detections
  colored by their decimation level ]
#+END_EXAMPLE

* INSTALLATION
As of today (2022-02-27), mrgingham is included in the bleeding-edge versions of
Debian and Ubuntu. So if you're running at least Debian/testing or the
not-yet-published Debian 12 (bookworm) or Ubuntu 22.04 (jammy), you can
simply

#+begin_src sh
apt install mrgingham libmrgingham-dev
#+end_src

to install the standalone tool and the development library respectively. For
older Debian and Ubuntu distros, packages are available in the mrcal
repositories. Please see the [[http://mrcal.secretsauce.net/install.html][mrcal installation page.]]

If running any other distro, you should build mrgingham from source. First, make
sure all the build-time dependencies are installed. These are listed in the
[[https://salsa.debian.org/science-team/mrgingham/-/blob/master/debian/control][=debian/control=]] file:

#+begin_example
Build-Depends: ...,
 libopencv-dev,
 libboost-dev,
 pkg-config,
 mawk,
 perl,
 python3-all,
 python3-all-dev,
 python3-numpy
#+end_example

On a Debian-based distro you can:

#+begin_src sh
sudo apt install \
 libopencv-dev   \
 libboost-dev    \
 pkg-config      \
 mawk            \
 perl            \
 python3-all     \
 python3-all-dev \
 python3-numpy
#+end_src

Replace the package names with their analogues on other distros. Once the
dependencies are installed, you

#+begin_src sh
make
#+end_src

and then the tool can be invoked with

#+begin_src sh
./mrgingham
#+end_src

* DESCRIPTION
Both chessboard and a square grid of circles are supported. Chessboard are the
/strongly/ preferred choice, since the circles cannot produce accurate results:
we care about the center point, which we are not directly observing. Thus with
closeup and oblique views, the reported circle center and the real circle center
could be very far away from each other. Because of this, more work was put into
the chessboard detector. Use that one. Really.

These are both nominally supported by OpenCV, but those implementations are slow
and not at all robust, in my experience. The implementations here are much
faster and work much better. I /do/ use OpenCV here, but only for some core
functionality.

Currently mrgingham looks for a /square/ grid of points, with some
user-requestable width. Rectangular grids are /not/ supported at this time.

** Approach
These tools work in two passes:

1. Look for "interesting" points in the image. The goal is to find all the
   points we care about, in any order. It is assumed that

   - there will be many outliers
   - there will be no outliers interspersed throughout the points we do care
     about (this isn't an unreasonable requirement: areas between chessboard
     corners have a solid color)

2. Run a geometric analysis to find a grid in this set of "interesting" points.
   This will throw out the outliers and it will order the output

If we return /any/ data, that means we found a full grid. The geometric search
is fairly anal, so if we found a full grid, it's extremely likely that it is
"right".

Once again: *the current implementation of mrgingham detects only complete
chessboards*.

*** Chessboards
This is based on the feature detector described in this paper:
https://arxiv.org/abs/1301.5491

The authors provide a simple MIT-licensed implementation here:
http://www-sigproc.eng.cam.ac.uk/Main/SB476Chess

This produces an image of detector response. /This/ library then aggregates
these responses by looking at local neighborhoods of high responses, and
computing the mean of the position of the points in each candidate neighborhood,
weighted by the detector response.

As noted earlier, I look for a square grid, 10x10 points by default. Here that
means 10x10 /internal corners/, meaning a chessboard with 11 /squares/ per side.
To ensure robust detections, it is recommended to make the outer squares of the
chessboard wider than the inner squares. This would ensure that we see exactly
10 points in a row with the expected spacing. If the outer squares have the same
size, the edge of the board might be picked up, and we would see 11 or 12 points
instead.

A recommended 10x10 pattern can be printed from this file: [[chessboard.10x10.pdf]].
And a recommended 14x14 pattern can be printed from this file:
[[chessboard.14x14.pdf]]. The denser chessboard containts more data, so fewer
observations will be required for convergence of the calibration algorithm. But
a higher-res camera is required to reliably detect the corners. A simple tool to
generate these =.pdf= files is included in the mrgingham repository. To make an
NxM chessboard figure do =make chessboard.NxM.pdf= in the mrgingham source tree.
=N= and =M= must be even integers. Note: today mrgingham requires that =N == M=,
but eventually this may be lifted, so this tool does not have this requirement.

*** Circles
*This isn't recommended, and exists for legacy compatibility only*

The circle finder does mostly what the first stage of the OpenCV circle detector
does:

1. Find a reasonable intensity threshold
2. Threshold the image
3. Find blobs
4. Return centroid of the blobs

This is relatively slow, can get confused by uneven lighting (although CLAHE can
take care of that), and is inaccurate: nothing says that the centroid of a blob
came from the center of the circle on the calibration board.

** Output representation
The =mrgingham= tool produces its output in a [[https://github.com/dkogan/vnlog/][vnlog]] text table. The columns are:

- =filename=: path to the image on disk
- =x=, =y=: detected pixel coordinates of the chessboard corner
- =level=: image level used in detecting this corner. Level 0 means
  "full-resolution". Level 1 means "half-resolution" and that the noise on this
  detection has double the standard deviation. Level 2 means
  "quarter-resolution" and so on.

If no chessboard was found in an image, a single record is output:

#+begin_example
filename - - -
#+end_example

The corners are output in a consistent order: starting at the top-left,
traversing the grid, in the horizontal direction first. Usually, the chessboard
is observed by multiple cameras mounted at a similar orientation, so this
consistent order is consistent across cameras.

However, if some cameras in the set are rotated, their observed chessboard
corners will not be consistent anymore: the first corner will be "top-left" in
pixel coordinates for both, which is at the top of the chessboard for
rightside-up cameras, but the bottom of the chessboard for upside-down cameras.
This situation is resolved with the =mrgingham-rotate-corners= tool. It
post-processes =mrgingham= output to reorder detections from rotated cameras.
See the manpage of that tool for more detail. Eventually the implementation
could be extended to be able to uniquely identify each corner, obviating the
need for =mrgingham-rotate-corners=, but we're not there today.

** API
The user-facing functions live in =mrgingham.hh=. Everything is in C++, mostly
because some of the underlying libraries are in C++. All functions return a
=bool= to indicate success/failure. All functions put the destination arguments
/first/. All functions return the output points in
=std::vector<mrgingham::PointDouble& points_out>=, an ordered list of found
points. The inputs are one of

- An image filename
- An OpenCV matrix: =cv::Mat& image=
- A set of detected points, that are unordered, and are a superset of the points
  we're seeking

The prototypes:

#+BEGIN_SRC C++
namespace mrgingham
{
    bool find_circle_grid_from_image_array( std::vector<mrgingham::PointDouble>& points_out,
                                            const cv::Mat& image );

    bool find_circle_grid_from_image_file( std::vector<mrgingham::PointDouble>& points_out,
                                           const char* filename );

    bool find_chessboard_from_image_array( std::vector<mrgingham::PointDouble>& points_out,
                                           const cv::Mat& image,
                                           int image_pyramid_level = -1 );

    int  find_chessboard_from_image_file( std::vector<mrgingham::PointDouble>& points_out,
                                          const char* filename,
                                          int image_pyramid_level = -1 );

    bool find_grid_from_points( std::vector<mrgingham::PointDouble>& points_out,
                                const std::vector<mrgingham::Point>& points );
};
#+END_SRC

The arguments should be clear. The only one that needs an explanation is
=image_pyramid_level=:

- if =image_pyramid_level= is 0 then we just use the image as is.

- if =image_pyramid_level= > 0 then we cut down the image by a factor of 2 that
  many times. So for example, level 3 means each dimension is cut down by a
  factor of 2^3 = 8

- if =image_pyramid_level= < 0 then we try several levels, taking the first one
  that produces results

** Applications
There're several included applications that exercise the library. The
=mrgingham-...= tools are distributed, and their manpages appear below. The
=test-...= tools are internal.

- =mrgingham= takes in images as globs (with some optional
  manipulation given on the cmdline), finds the grids, and returns them on
  stdout, as a vnlog

- =mrgingham-observe-pixel-uncertainty= evaluates the distribution of corner
  detections from repeated observations of a stationary scene

- =mrgingham-rotate-corners= corrects chessboard detections produced by rotated
  cameras by reordering the points in the detection stream

- =test-find-grid-from-points= ingests a file that contains an unordered set of
  points with outliers. It the finds the grid, and returns it on stdout

- =test-dump-chessboard-corners= is a lower-level tool that just finds the
  chessboard corner features and returns them on stdout. No geometric search is
  done.

- =test-dump-chessboard-corners= similarly is a lower-level tool that just finds
  the blob center features and returns them on stdout. No geometric search is
  done.

** Tests
There's a test suite in =test/test.sh=. It checks all images in =test/data/*=,
and reports which ones produced no data. Currently I don't ship any actual data.
I will at some point.

* MANPAGES
** mrgingham
#+BEGIN_EXAMPLE
xxx-manpage-mrgingham-xxx
#+END_EXAMPLE

** mrgingham-observe-pixel-uncertainty
#+BEGIN_EXAMPLE
xxx-manpage-mrgingham-observe-pixel-uncertainty-xxx
#+END_EXAMPLE

** mrgingham-rotate-corners
#+BEGIN_EXAMPLE
xxx-manpage-mrgingham-rotate-corners-xxx
#+END_EXAMPLE

* MAINTAINER
This is maintained by Dima Kogan <dima@secretsauce.net>. Please let Dima know if
something is unclear/broken/missing.
* LICENSE AND COPYRIGHT

This library is free software; you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Software Foundation; either version 2.1 of the License, or (at your option) any
later version.

Copyright 2017-2021 California Institute of Technology

Copyright 2017-2021 Dima Kogan (=dima@secretsauce.net=)
