Make the test suite faster #174

This was generated by AI from a grilling session with the maintainer.

Problem Statement

The full test suite is too slow for the agent sessions inside a Run. With three Runs on a 4-core machine, a full cargo test took about 15–16 minutes (load 14–26). That is longer than the 600s Bash timeout the sessions run under, so the headless instruction to "run tests in the foreground" cannot be followed. Sessions then improvise waits that go wrong (#180), and each Run pays that cost again after every Repair. Every new Run worktree also starts with a cold build of about 10 minutes before the first test runs. CI, which every Run waits on at least once, pays the same costs.

Measured on 2026-09-30 (4 cores, load 14–26):

• Cold cargo test --no-run: 616s wall, 329s user + 53s sys. Top units: aws-lc-sys build script 372s (via reqwest → rustls), moxcms 113s, tokio 75s, image 56s, pxfm 54s (image/moxcms/pxfm via axoupdater → axoasset).
• Warm rebuild after editing src/: about 13s. The integration test crates never relink, because they use only the built binary.
• Test run: spec_run 282s, merge_run 140s, release 110s, run_notification 98s, ci 56s, the rest under 40s each.
• The time goes on starting processes, not on sleeps. Every scenario runs the real binary end to end, and it starts many git calls and Python fake gh/claude calls. The harness polls CI every 10ms, and each poll starts the fake gh again. Tests that loop over cases or over Repair cycles multiply this. Python startup is very sensitive to load (python3 -c pass: 0.48s wall for 0.04s of CPU).
• There is no [profile.dev], so builds use full debuginfo: a 108MB binary and a 1.6GB target dir per worktree.

Solution

Make the suite cheaper to build and to run, and measure the result in a way that doesn't depend on machine load. A session can then run the full suite in the foreground again, and a fresh Run worktree becomes testable in a few minutes rather than ten. The full suite stays the local gate for sessions; this spec does not narrow what they run.

Done means all three:

1. A warm full cargo test finishes in ≤ 3 minutes wall on an idle machine.
2. The warm full suite's total CPU time (user + sys) is at most half of a baseline recorded before any change.
3. A cold cargo test --no-run finishes in ≤ 5 minutes wall on an idle machine.

User Stories

1. As an agent session in a Run, I want the full test suite to finish within one foreground Bash call, so that I never have to wait on a backgrounded test run.
2. As an agent session in a Run, I want a fresh worktree to build its test targets in a few minutes, so that my first test run doesn't use up most of my time.
3. As an agent session in a Repair, I want re-running the suite after a fix to be quick, so that I can verify each fix before pushing.
4. As the maintainer running several Runs at once, I want each suite run to use less CPU, so that parallel Runs slow each other down less.
5. As the maintainer, I want less disk used per Run worktree, so that many worktrees can live side by side.
6. As the maintainer, I want the speed-up measured in CPU time as well as wall time, so that I can tell a real gain from a quiet machine.
7. As the maintainer, I want a recorded baseline and after-numbers in the PR, so that I can see what each change bought.
8. As the maintainer, I want the shipped binary to keep verifying TLS the way it does now, so that switching the crypto provider changes build time only, not behavior.
9. As the maintainer, I want self-update (axoupdater) to keep working, so that trimming dependencies doesn't break thirdshift update.
10. As a contributor, I want the end-to-end tests to keep checking the same behaviors, so that a faster suite doesn't mean weaker coverage.
11. As a contributor, I want argument-parsing cases covered by fast tests, so that adding a case doesn't add a full end-to-end Run.
12. As a contributor, I want the test fakes for gh and claude to stay easy to read and extend, so that new scenarios stay cheap to write.
13. As a contributor, I want backtraces from a failing test to still show file and line, so that cheaper debug builds don't make failures harder to find.
14. As CI, I want shorter build and test jobs, so that a Run's CI wait is shorter.

Implementation Decisions

Apply the levers cheapest first, measuring after each. Once all three targets are met, the rest are optional, but record what was skipped and why.

• Baseline first. Before any change, record on the same machine: cold cargo test --no-run wall time, warm full cargo test wall time, and user + sys (e.g. /usr/bin/time), with the load average at the time. Report before and after in the PR.
• Cheaper debug builds. Add a dev profile with debug = "line-tables-only" for the crate and no debuginfo for dependencies. Backtraces keep file and line; debugger locals are given up.
• rustls crypto provider: ring, not aws-lc. Configure reqwest/rustls so aws-lc-sys is no longer built at all (check cargo tree -i aws-lc-sys comes back empty). TLS verification behavior stays the same.
• axoupdater's image stack. Drop image/moxcms/pxfm only if axoupdater/axoasset offer a feature flag for it. If not, leave the dependency alone and note it in the PR as a follow-up (an upstream change or a replacement updater). Don't fork or replace axoupdater here.
• Fewer processes per scenario (lever 1). The CI poll interval is already overridable through the existing environment variable the harness sets. Raise the harness value from 10ms to about 100ms, and adjust the CI grace period the harness sets if needed so CI-wait scenarios still see their states. No production change is needed.
• Fewer end-to-end scenarios (lever 2). Where a test loops over cases that differ only in what they exercise below the Run (e.g. every position --email can take in the arguments), keep one end-to-end scenario and move the case list to unit tests of the argument parsing. The same goes for loops over signals or Repair and budget cycles, where one end-to-end case proves the wiring and the rest can be checked below the binary. Argument parsing has no unit tests today, so this adds that one new seam.
• Cheaper fakes (lever 3, only if still needed). Replace the Python fake gh/claude with something cheaper to start, e.g. a small Rust helper binary built with the tests, keeping their current contract: state and records in the JSON files the harness names, and the same argv handling.
• Don't merge the integration test files into one crate: they never relink when src/ changes, so there is little to gain.

Testing Decisions

• A good test here checks external behavior: the real binary run end to end against the fakes, asserting on exit code, stderr, git state, and the requests the fakes record. It doesn't check how the Run does it internally.
• The end-to-end harness stays the main seam. Every behavior an end-to-end test covers today must still be covered end to end or by a unit test at the level where the cases differ.
• The only new seam is unit tests of argument parsing, alongside the existing unit-test modules in the crate (prior art: the in-crate unit tests for config, email, notification, progress and worktree).
• If the fakes are rewritten, every existing scenario must pass unchanged against the new ones. They are the fakes' test.
• The measurements (baseline and after) are the acceptance test for speed. Include them in the PR.

Out of Scope

• Limiting how many builds run at once across Runs, and a Spec run's default parallelism (filed separately).
• The flaky 10s "the agent never started" deadline in the harness (filed separately).
• Narrowing what agent sessions run (targeted tests instead of the full suite).
• Language-neutral speed-ups in thirdshift itself, e.g. a worktree-setup hook or cache sharing for the repos it runs on.
• Changes to the CI workflow (caching, splitting jobs), beyond what it gains from a cheaper build and suite.
• Replacing or forking axoupdater.
• Merging integration test crates.

Further Notes

• Grilled on 2026-09-30 with the maintainer. Settled: the main aim is agent sessions' wait, with CI second; scope is this repo's own suite; sessions keep running the full suite; the time on an idle machine and the halved CPU time are both targets.
• Related: Sessions wait on a backgrounded command by its own task, not by process name #180 (sessions waiting on a backgrounded command by process name), which this makes rarer but doesn't replace.