# rsync testsuite

This directory holds rsync's automated regression tests. Ideally every code
change or bug fix comes with a test that would have caught the problem.

The tests are Python scripts named `testsuite/*_test.py`, driven by the
`runtests.py` harness at the top of the tree (the old shell-based `runtests.sh`
is gone). Shared helpers live in `testsuite/rsyncfns.py`. A handful of C helper
programs (`tls`, `getgroups`, `trimslash`, …) are built alongside `rsync` and
used by some tests. Coverage notes are in [COVERAGE.md](COVERAGE.md).

## Writing tests

Favour readability — a test is also documentation of the behaviour it pins, so
prefer clarity over cleverness:

* When a test writes an `rsyncd.conf`, write it as a triple-quoted f-string so
  the actual config is readable top-to-bottom, with module parameters indented
  with plain spaces. Don't build it from adjacent string literals full of `\n`
  (and `\t`) escapes. The daemon's parser accepts space-indented parameters.
* Better still, use the structured helpers in `rsyncfns.py` when a stock config
  will do: `write_daemon_conf(modules, globals)` (per-test modules/params) or
  `build_rsyncd_conf()` (the four standard modules). They also handle the
  root-only `uid`/`gid` lines for you (needed so a `use chroot = no` daemon run
  as root can read a root-owned module).
* For config that varies (e.g. those root-only `uid`/`gid` lines), interpolate a
  single optional block that expands when needed and is an empty string
  otherwise, rather than splicing pieces together:

  ```python
  root = get_testuid() == get_rootuid()
  ids = f"uid = {get_rootuid()}\ngid = {get_rootgid()}" if root else ""
  conf.write_text(f"""\
  pid file = {base}/rsyncd.pid
  use chroot = no
  {ids}
  log file = {base}/rsyncd.log

  [m]
      path = {mod}
      read only = yes
  """)
  ```

## Running the tests

### Via make

Run from the build directory:

- **`make check`** — build the helper programs and run the whole suite in
  parallel (`CHECK_J`, default 8) against the just-built `./rsync`. You do **not**
  need `make install` first; indeed you generally should not install before
  testing. Use `make check CHECK_J=1` to run serially.
- **`make check29`** / **`make check30`** — the same, forcing protocol version 29
  or 30.
- **`make installcheck`** — run the suite against the *installed* binary (e.g.
  `/usr/local/bin/rsync`). Per the GNU standards this does not search `$PATH`.
  Handy for testing a distribution build.
- **`make check-progs`** — (re)build just the C helper programs the tests need,
  without running anything.
- **`make coverage`** / **`coverage-tcp`** / **`coverage-all`** — generate an HTML
  coverage report (needs `./configure --enable-coverage` and `gcovr`);
  `coverage-all` merges runs across protocol versions and the tcp transport.

### Via runtests.py directly

`make check` just drives `runtests.py`; run it directly for finer control. It
defaults `--rsync-bin` to `./rsync`, so run it from the build directory (or pass
`--rsync-bin` / `--tooldir`):

```sh
./runtests.py                 # all tests
./runtests.py chmod-temp-dir  # a single test by name
./runtests.py 'xattr*'        # a glob of test names
```

Useful options:

- `-j N`, `--parallel N` — run up to N tests at once
- `--use-tcp` — run daemon tests against a real `rsyncd` on `127.0.0.1` (the
  default runs them over a stdio pipe). **Read the security warning below before
  using this on a shared machine.**
- `--protocol VER` — force a protocol version
- `--preserve-scratch` — keep each test's scratch dir afterwards
- `--log-level N`, `--always-log` — more verbose output / show logs for passing tests too
- `--stop-on-fail` — stop after the first failure
- `--timeout SECS` — per-test timeout (default 300)
- `--timing` — after the run, list the tests by wall-clock, slowest first, with
  the serial sum and the floor set by the single longest test
- `--race-timeout SECS` — budget a TOCTOU race test may spend trying to win its
  race. These are the suite's slowest tests: a race test is a negative oracle,
  so it passes by spending its *whole* budget (5–15s each by default). Lowering
  this speeds the suite up and weakens the oracle in equal measure.
- `--valgrind`, `--valgrind-opts OPTS` — run rsync under valgrind
- `--rsync-bin PATH`, `--tooldir DIR`, `--srcdir DIR` — locate the binary / build / source dirs
- `--expect-skipped LIST` — see skip enforcement below

### Security warning: `--use-tcp`

> **⚠️ Do not use `--use-tcp` on a machine with untrusted local users.**
>
> `--use-tcp` starts a real `rsync` daemon listening on a loopback TCP port
> (`127.0.0.1` / `::1`) and **deliberately configures insecure test scenarios**
> (daemon modules without authentication, unsafe options enabled, etc.). Loopback
> addresses are reachable by *every* local user, so for as long as the tests run,
> any other user on the machine can connect to that daemon and exploit those
> deliberately-insecure modules — potentially reading or writing files with the
> privileges of the user running the tests (which is **root** if you run the suite
> as root).
>
> Only run `--use-tcp` where there are **no possible local users who might try to
> exploit it** — a single-user workstation or a dedicated, isolated CI machine.
> The default stdio-pipe transport carries no such risk: it talks to the daemon
> over a private pipe with nothing listening on the network, so prefer it on any
> shared or multi-user host.

### Results and exit codes

Each test prints one result line — `PASS`, `FAIL`, `ERROR`, `SKIP` (with a
reason), or `XFAIL` (an expected failure) — and the run ends with a
`passed / failed / skipped` summary. Per-test exit-code convention:

| code | meaning |
|------|---------|
| 0    | pass    |
| 1    | fail    |
| 2    | error   |
| 77   | skip    |
| 78   | xfail   |

`runtests.py` exits non-zero if any test fails. Some tests need root or another
precondition and otherwise `SKIP` — read the individual test scripts for details.

**Skip enforcement:** on a full run, set `RSYNC_EXPECT_SKIPPED=a,b,c` (or
`--expect-skipped a,b,c`) and the run fails if the set of skipped tests does not
match. This is how the CI workflows pin each platform's expected skip set. An
`@FILE` entry reads a skip list (one test per line) instead, and several may be
composed: the workflows use
`@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt`. Keeping the
lists one-name-per-line is what stops two branches that each add a skipping test
from conflicting -- see `testsuite/skiplist/README.md`.

### Scratch dirs and debugging

Each test runs in `testtmp/<name>/`. On failure the scratch directory is left in
place (also `--preserve-scratch`); including its logs in a bug report is helpful.

### Preconditions

You need `python3`, `/bin/sh`, and the normal build toolchain. The ACL/xattr
tests need the `acl` and `attr` tools (`getfacl`/`setfacl`,
`getfattr`/`setfattr`) and skip if they are absent. Some tests need root.

These tests also run in CI via GitHub Actions (see `.github/workflows/`).

## Fleet testing (fleettest.py)

`testsuite/fleettest.py` builds the committed HEAD of an rsync checkout on a
fleet of remote machines over ssh and runs the suite under both transports
(stdio-pipe and `--use-tcp`) in parallel, reporting only the *unexpected*
results. It is a fast local pre-flight for the GitHub CI matrix: each target
mirrors a `.github/workflows/*.yml` job — its configure flags, and the
`RSYNC_EXPECT_SKIPPED` list parsed straight from the workflow.

Because every run includes a `--use-tcp` pass, the fleet stands up the insecure
loopback test daemon on each target — so only point it at machines with **no
untrusted local users** (see the [security warning](#security-warning---use-tcp)
above).

The fleet — which machines, and how to reach and build on each — is described in
a JSON file. Copy the bundled example (it is git-ignored) and edit it for your
hosts:

```sh
cp testsuite/fleettest.json.example testsuite/fleettest.json   # then edit
# (or symlink it, or point elsewhere with --fleet PATH)
```

The config is looked up in order: `~/.fleettest.json` first, then
`testsuite/fleettest.json`, unless overridden with `--fleet PATH`.

Each entry names an ssh host (`null` to run locally), the workflow it mirrors,
and its configure flags, plus optional per-target settings (`make`, `privilege`,
`env_prefix`, …). See the comments in `fleettest.json.example`.

A target with `"nonroot": true` does an extra pass, after the main (root) run,
that reruns the privilege-sensitive tests as the unprivileged ssh user. Which
tests those are is **not** listed in the fleet config — a test opts in by
setting a module-level `fleet_nonroot = True`, so the set is maintained in the
test files and new privilege-sensitive tests join automatically with no
fleet-config change.

A target with `"protocols": [30, 29]` runs one extra stdio-pipe pass per listed
version, each forcing that older wire version with `runtests --protocol=N` — the
fleet analogue of a workflow's `check30`/`check29` steps. Each pass takes the
`RSYNC_EXPECT_SKIPPED` spec from the workflow's own `check30`/`check29` step, so
a lane with extra protocol-gated skips (`check29` adds
`@testsuite/skiplist/proto29.txt`) is enforced correctly. They show up as
`protoNN` columns in the report (and `--timing` breakdown); targets that don't
set `protocols` show `-` there.

Run it from inside a checkout (it builds the current directory's HEAD; use
`--repo PATH` for another tree):

```sh
python3 testsuite/fleettest.py                       # whole fleet, both transports
python3 testsuite/fleettest.py --list                # list configured targets
python3 testsuite/fleettest.py --targets NAME[,NAME]
python3 testsuite/fleettest.py --fleet other.json --transport pipe
python3 testsuite/fleettest.py --timing              # per-target wall-clock breakdown
python3 testsuite/fleettest.py --keep-on-fail        # keep logs + tree where it broke
python3 testsuite/fleettest.py --full-tcp            # whole suite in the tcp pass too
```

`--timing` adds a per-target breakdown after the report — total wall-clock plus
the push / build / pipe / tcp / protoNN / nonroot phases, sorted slowest-first. Targets
run in parallel, so the whole run is gated by the slowest one; the phase columns
show whether that target's hold-up is the push, the build, or a test pass. It
also passes `--timing` down to each target's `runtests.py`, so the captured
output attributes a slow pass to individual tests.

The `tcp` pass runs **only the tests that can reach the daemon transport**, since
it follows a full pipe pass over the very same build. `--use-tcp` is observable
through exactly one path — `RSYNC_TEST_USE_TCP` is read once in `rsyncfns`
(`USE_TCP`) and acted on once, in `start_test_daemon()` — so a test that never
gets there produces an identical result twice. That drops 186 of the 340 tests
and roughly a third of the pass's work; the count skipped is always printed.
Pass `--full-tcp` to sweep the whole suite there anyway. The narrowing applies
only when both transports run: under `--transport tcp` that pass is the only
one, so it runs the whole suite regardless.

`--keep-on-fail [DIR]` makes a failure inspectable without repeating the run.
For every target that came back with anything unexpected it writes the full
build and per-transport output to `DIR/<run_id>/<target>/` (default
`./fleettest-logs`) and keeps that target's remote run dir, with the scratch
trees its failing tests left behind. Targets that came back clean are swept as
usual. This matters most for the race tests, which may not fail the same way
twice — and because a re-run costs a full configure + build on every machine.

Each run gets its own randomly-named build dir on every target
(`<builddir>-<run_id>`), so two or three runs can share the same fleet without
interfering. The dir is removed when the run ends — on success or failure, and
best-effort on Ctrl-C/kill; pass `--keep` to retain it for inspection. A hard
kill (`SIGKILL`), or a signal arriving mid-push, can leave a stray
`<builddir>-<id>` behind; sweep leftovers with
`python3 testsuite/fleettest.py --cleanup` (scope it with `--targets`, and only
run it when no other fleet runs are active, since it removes *all* matching run
dirs on the selected targets).

Each target must be provisioned with the build toolchain its workflow installs
(autoconf, automake, a C compiler, perl, a python3 markdown module such as
cmarkgfm or commonmark unless the flags pass `--disable-md2man`, and the dev
libraries its configure flags enable). A missing piece shows up as `BUILD-FAIL`.

## Differential regression hunting (abdiff.py)

`testsuite/abdiff.py` is a developer tool — **not** a `*_test.py`, so `runtests.py`
ignores it. It hunts *regressions* by running the **same benign transfer** with
two rsync binaries (`A` = the build under test, `B` = a baseline) and comparing
the OUTCOME. The oracle is: for a benign input, a correctness/behaviour change
between the builds must be **invisible**, so A and B must produce an identical
result. Any divergence is a regression candidate to investigate and, if real,
minimize into a `*_test.py`.

It compares exit code, stderr (error markers + normalised text), `--stats`
"Literal data", the destination tree (content + full metadata: mode/uid/gid/
mtime/size/symlink target/xattrs/ACLs/hardlink grouping), the `--itemize` list,
and — with `--cost` — peak process-group RSS (a resource-regression oracle that
functional comparison misses). A **stability gate** runs each binary several
times and escalates on a candidate diff; nondeterministic scenarios are
quarantined `FLAKY`, never reported as regressions.

Run it from the build directory (so `./rsync` and `old_versions/` resolve):

```sh
testsuite/abdiff.py                       # default: ./rsync vs old_versions/rsync_3.4.1
testsuite/abdiff.py --sweep all -j5       # broad single pass, 5-way parallel
testsuite/abdiff.py --loop --timelimit 3600 --cost   # hunt for an hour, resource oracle on
testsuite/abdiff.py --list --sweep all    # list scenarios without running
```

Each finding is classed `DIFF` (regression candidate), `ALLOW` (an intentional,
documented behaviour change listed in the tool's allowlist), `BETTER` (A succeeds
where B fails), `FLAKY`, or `TIMEOUT`. Findings are printed and appended to a
per-run `abdiff-log_<TIME>.txt` (and the curated `--findings` log).

Key options: `-j N` parallelism; `--sweep NAME|all`; `--loop` (endless
random + systematic-combo stream) bounded by `--timelimit SECS`; `--cost`
(+`--scale N` for the large-tree fixtures); `--repeat N` (stability samples);
`--rsync-a`/`--rsync-b` the two binaries. Run **as root** to fold in the
owner/device/specials/fake-super and chroot-daemon sweeps automatically.

Transport lanes (a feature broken only over the wire is invisible to a local
copy): local, an ssh split (`support/lsh.sh`), a stdio-pipe daemon, a **real TCP
daemon** (bound port + greeting/handshake, and an auth challenge-response
variant), and the restricted **rrsync** wrapper (`support/rrsh.sh`). rrsync's
behaviour ships in the *script*, so pair each binary with its own version's
rrsync via `--rrsync-a`/`--rrsync-b` (give B's rrsync, e.g. one extracted from
that release's `support/rrsync`).

Cross-version baselines are the static binaries already in `old_versions/`;
`old_versions/build_static.sh` builds more from a git tag (and you can grab a
matching `support/rrsync` from the same tag for the rrsync lane).
