# Running Tests

inq is taught how to run your tests by interpreting your configuration file
(`inquest.toml`, `.inquest.toml`, or `.testr.conf`). For instance:

```toml
test_command = "foo $IDOPTION"
test_id_option = "--bar $IDFILE"
```

will cause `inq run` to run `foo` and process it as `inq load` would.
Likewise `inq run --failing` will automatically create a list file listing
just the failing tests, and then run `foo --bar failing.list` and process it as
`inq load` would. failing.list will be a newline separated list of the
test ids that your test runner outputs. If there are no failing tests, no test
execution will happen at all.

Arguments passed to `inq run` are used to filter test ids that will be run -
inq will query the runner for test ids and then apply each argument as a
regex filter. Tests that match any of the given filters will be run. Arguments
passed to run after a `--` are passed through to your test runner command
line. For instance, using the above config example `inq run quux -- bar
--no-plugins` would query for test ids, filter for those that match `quux` and
then run `foo bar --load-list tempfile.list --no-plugins`. Shell variables
are expanded in these commands on platforms that have a shell.

Having set up a configuration file, a common workflow then becomes:

```sh
  # Fix currently broken tests - repeat until there are no failures.
  $ inq run --failing
  # Do a full run to find anything that regressed during the reduction process.
  $ inq run
  # And either commit or loop around this again depending on whether errors
  # were found.
```

The `--failing option` turns on `--partial` automatically (so that if the
partial test run were to be interrupted, the failing tests that aren't run are
not lost).

Another common use case is repeating a failure that occured on a remote
machine (e.g. during a jenkins test run). There are two common ways to do
approach this.

Firstly, if you have a subunit stream from the run you can just load it:

```sh
  $ inq load < failing-stream
  # Run the failed tests
  $ inq run --failing
```

The streams generated by test runs are in `.testrepository/` named for their test
id - e.g. `.testrepository/0` is the first stream.

If you do not have a stream (because the test runner didn't output subunit or
you don't have access to the .testrepository) you may be able to use a list
file. If you can get a file that contains one test id per line, you can run
the named tests like this:

```sh
  $ inq run --load-list FILENAME
```

This can also be useful when dealing with sporadically failing tests, or tests
that only fail in combination with some other test - you can bisect the tests
that were run to get smaller and smaller (or larger and larger) test subsets
until the error is pinpointed.

`inq run --until-failure`` will run your test suite again and again and
again stopping only when interrupted or a failure occurs. This is useful
for repeating timing-related test failures.

## Re-running a previous run

`inq rerun <run_id>` re-runs exactly the tests of a previous run, in the
same order, forwarding the same `--`-style test arguments that the run
captured in its metadata. Without an argument it re-runs the latest run;
negative indices like `-2` are also accepted.

```sh
  $ inq rerun 7    # Repeat run 7
  $ inq rerun -1   # Repeat the latest run
```

This is useful for reproducing a flaky failure under the same conditions
or for re-running a suite after a code change without re-typing
`-- --pytest-flag --another` each time.

## Timeouts

inq supports several timeout mechanisms to deal with hanging tests:

### Per-test timeout (`--test-timeout`)

Kills the test runner if an individual test exceeds its timeout. The timeout
can be a fixed duration or computed automatically from historical test times:

```sh
  # Fixed 5 minute timeout per test
  $ inq run --test-timeout 5m
  # Auto-compute from historical times (3x the average)
  $ inq run --test-timeout auto
```

When a per-test timeout fires, inq kills the test runner process and
records the hung test as a timeout error. If the test command supports
filtering by test ID (`$IDOPTION`/`$IDFILE`/`$IDLIST` in the command),
inq restarts the runner with the remaining tests. Otherwise the run
stops.

### Restarting after a crash

If the test runner exits with a non-success status while one or more
tests are still in progress (a panic, segfault, or other abnormal
termination), inq treats this as a crash rather than a normal failing
run. The in-flight tests are recorded as errors ("test runner exited
while this test was running"), and — provided the test command supports
filtering by test ID — inq restarts the runner with the remaining
tests.

Restart-on-crash only fires when forward progress was made: if the
first test in a partition crashes the runner without any other test
completing, inq stops instead of looping. An ordinary failing run,
where all tests completed and the runner just exits non-zero because
of the failures, is not treated as a crash and does not trigger a
restart.

### Maximum number of restarts (`--max-restarts`)

Both timeout and crash restarts share a single budget, defaulting to
10 restarts per run. Override it with:

```sh
  $ inq run --max-restarts 3
```

Once the budget is exhausted, inq stops restarting and reports whatever
results it has so far.

### Overall run timeout (`--max-duration`)

Kills the entire test run if it exceeds the given duration:

```sh
  $ inq run --max-duration 30m
  $ inq run --max-duration auto
```

### No-output timeout (`--no-output-timeout`)

Kills the test runner if no output is produced for the given duration.
This catches tests that hang silently without producing any subunit events:

```sh
  $ inq run --no-output-timeout 60s
```

### Duration format

Durations are specified as a number with an optional unit suffix:
`30s`, `5m`, `1.5h`, `300` (seconds by default).

### Configuration file

All timeout settings can also be set in the configuration file:

```toml
test_timeout = "auto"
max_duration = "30m"
no_output_timeout = "120s"
```

CLI flags take precedence over configuration file values.