# Automated test isolation bisection

As mentioned previously, it is possible to manually analyze test isolation
issues by interrogating the repository for which tests ran on which worker, and
then creating a list file with those tests, re-running only half of them,
checking the error still happens, rinse and repeat.

However that is tedious. inq can perform this analysis for you

```sh
  $ inq run --analyze-isolation
```

will perform that analysis for you. (This requires that your test runner is
(mostly) deterministic on test ordering). The process is:

1. The last run in the repository is used as a basis for analysing against -
   tests are only cross checked against tests run in the same worker in that
   run. This means that failures accrued from several different runs would not
   be processed with the right basis tests - you should do a full test run to
   seed your repository. This can be local, or just inq load a full run from
   your Jenkins or other remote run environment.

2. Each test that is currently listed as a failure is run in a test process
   given just that id to run.

3. Tests that fail are excluded from analysis - they are broken on their own.

4. The remaining failures are then individually analysed one by one.

5. For each failing, it gets run in one work along with the first 1/2 of the
   tests that were previously run prior to it.

6. If the test now passes, that set of prior tests are discarded, and the
   other half of the tests is promoted to be the full list. If the test fails
   then other other half of the tests are discarded and the current set
   promoted.

7. Go back to running the failing test along with 1/2 of the current list of
   priors unless the list only has 1 test in it. If the failing test still
   failed with that test, we have found the isolation issue. If it did not
   then either the isolation issue is racy, or it is a 3-or-more test
   isolation issue. Neither of those cases are automated today.

This cannot prove the absence of interactions - for instance, a runner that randomises the order of tests executing combined with a failure that occurs with A before B but not B before A could easily appear to be isolated when it is not.
