Skip to content

Make the test suite faster - #210

Merged
JacobStephens2 merged 25 commits into
mainfrom
issue-174
Sep 30, 2026
Merged

JacobStephens2 merged 25 commits into
mainfrom
issue-174

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Sep 30, 2026 •

Copy link
Copy Markdown
Owner

Summary

This makes the test suite cheaper to build and run, so an agent session can run the full cargo test in one foreground Bash call.

 Cargo.toml
+  [profile.dev] debug = "line-tables-only"; dependencies debug = false   # #202
+  comment: why aws-lc and the image stack stay (axoasset pins them)      # #203
 tests/support/mod.rs
-  THIRDSHIFT_POLL_MS=10
+  THIRDSHIFT_POLL_MS=100                                                  # #205
 tests/*.rs   loops over --email positions, signals, labels, spellings, key sources
-  one Run per case
+  one Run per behavior; the cases move to unit tests                     # #204, #206
+    src/args.rs (new seam: argument parsing), src/interrupt.rs, src/run.rs (Budgets),
+    src/resend_key.rs, src/spec_run.rs, src/config.rs, scripts sourced by tests/release.rs
 tests/fakes/
-  gh, claude            (Python, ~0.5s to start under load)
+  fakes.rs gh.rs claude.rs json.rs   (one std-only Rust executable, same contract;
+                                      built with rustc on first use, cached by source hash)  # #207

The review found a few things, which this PR also fixes:

  • The fakes now share one git runner and one reader for their FAKE_* paths.
  • gh fake … is now handled by fake_command.
  • A failed Run whose Run notification also fails to send is tested end to end again. Collapsing its loop had dropped that case.

Evidence

Machine: 4 cores. None of these runs was on an idle machine.

Measurement Target Baseline (55bcf6e) After (#207)
Warm full cargo test, wall ≤ 180s 268s 107s / 129s / 113s
Warm full cargo test, user + sys ≤ 350s 479 + 222 = 701s 57 + 131 = 188s
Cold cargo test --no-run, wall ≤ 300s 127s 108s

I ran the warm suite again at this PR's head, after the review fixes: 127s wall, with 75s user + 156s sys = 231s CPU. All 25 test binaries pass.

The #207 runs also passed with a python3 stub first on PATH that fails if called, so nothing calls Python any more. The per-ticket measurements are in the comments on #174.

Merge Danger

Door: two-way

The only production-side changes are:

  • the dev profile
  • environment_key split out of from_environment in src/resend_key.rs, with the same behavior
  • scripts/release*.sh now run main only when executed, not when sourced, and is_plain_version is extracted

Blast Radius: tests

The shipped binary is unchanged: release builds keep their profile, and the TLS stack and axoupdater are untouched. The integration tests now need rustc on PATH (or $RUSTC) rather than python3. Dependency frames in dev backtraces lose file and line.

Follow-up

Unaddressed findings

Standards

  • Primitive Obsession: the gh fake's state is untyped Json reached by string keys. Kept on purpose: it keeps the Python fake's JSON contract, which the harness writes and reads.
  • Global static LOCK in tests/fakes/gh.rs, released by run_hook. Kept: it mirrors how the Python fake let go of the lock while a hook ran, and the doc comment explains why.
  • environment_key exists only as a test seam. Kept: it is small and well named.

Spec

  • "Configure reqwest/rustls so aws-lc-sys is no longer built at all": not done. It can't be done with a feature flag (see Follow-up). The spec allows skipping a lever once the targets are met.
  • The Repair and budget-cycle end-to-end tests still run every round. Each is already a single Run: the cap only runs out after the real rounds, so those rounds are the behavior under test. The counting also has unit tests in src/run.rs.
  • email.always with no key, or with a key only in the Credentials, has no end-to-end test. Not re-added: main.rs turns email.always into the same NotificationAsk as --email, and config.rs's unit tests cover that. From there the key lookup is the same path, which is tested end to end.
  • SIGINT to the Spec run alone has no end-to-end test. Not re-added: a unit test in src/interrupt.rs shows SIGINT sets the interrupt flag. The Spec run then always sends SIGTERM to the Ticket's Run, and that path is tested end to end.
  • New unit-test modules besides argument parsing, plus the sourceable release scripts. Kept: this is lever 2's "the rest can be checked below the binary". Argument parsing was the only seam that had no tests before.
  • The Rust fakes, about 1,860 lines including a JSON module, are larger than the Python fakes. Kept: they cut the suite's CPU time by about two thirds. Each fake stays one file with its contract documented at the top.
  • The timings were not measured on an idle machine. Every target is met even under load, so an idle machine would only do better.

Closes #174

Tickets

🤖 Generated with Claude Code

JacobStephens2 and others added 24 commits September 30, 2026 20:29
Line tables for the crate and no debuginfo for dependencies: the debug
binary drops from 108MB to 27MB and target/ from 1.5GB to 838MB, and
backtraces keep file and line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use line-table debuginfo for dev builds
axoupdater needs axoasset's remote feature, which always pulls in image
and turns on reqwest's default-tls (rustls with aws-lc). No flag drops
either, and asking reqwest for ring would build ring alongside aws-lc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two --email looping tests in run_notification ran nine full Runs
to cover flag position, bare flag against an address, the Issue URL
never taken as an address, and merge before and after. args now has
unit tests for every one of those cases, and each end-to-end test is
cut to one Run showing the parsed arguments reach the Run notification.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Say why aws-lc and axoupdater's image stack stay
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unit tests for argument parsing; one Run per --email end-to-end test
Each looping test in spec_run, run_notification and release keeps one case
end to end and moves the rest to unit tests where the cases differ: the
interrupt's signals, the Spec run's Unready labels, the Resend key lookup,
and the release scripts' version, summary and bump-PR steps, which the tests
now call by sourcing the scripts. The Repair and upstream-move budget tests
keep their real rounds, say so, and gain unit tests of the counting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Poll CI every 100ms in the test harness
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collapse the looping end-to-end scenarios to one Run per behavior
The warm suite missed both of #174's targets (249s wall, 650s user + sys
against a 701s baseline), and the Python fakes' own CPU was about 352s of
that: 4,940 gh calls and 381 claude calls. The fakes are now one std-only
Rust executable, built from tests/fakes.rs with rustc the first time a
test needs it and cached in CARGO_TARGET_TMPDIR by a hash of its sources.
It acts as gh or claude by the name it is called by, under the same
contract: the same state and record files, argv handling and output.

tests/fakes.rs is also a Cargo test target, so clippy and rustfmt cover
the fakes and the JSON layout is checked against Python's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pull the repeated pieces of the fakes into one place each: the --jq
refusal, the "not open" check, the branch tip on origin, record appends
and the lock file. Move the JSON tests next to the JSON module. Build the
fakes without -D warnings, so a lint a newer rustc adds can't fail the
suite: clippy covers them through the fakes test target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the Python fake gh and claude with a Rust helper
…led send after a failed Run

The fakes now share one git runner and one reader for the FAKE_* paths,
and `gh fake …` is handled by `fake_command`. A failed Run whose
notification also fails to send is tested end to end again, as the
collapsed loop covered it before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacobStephens2
JacobStephens2 marked this pull request as ready for review September 30, 2026 23:02
main's release script now waits for the release workflow run (#215), which
taught the Python fake gh `run list --commit`, `run view`, `release view`,
run attempts and $FAKE_MERGE_HEAD/$FAKE_MERGE_SHA. issue-174 replaced that
fake with tests/fakes/gh.rs, so those changes are ported there and the
Python fake stays deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JacobStephens2
JacobStephens2 merged commit 9dd7d34 into main Sep 30, 2026
13 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-174 branch September 30, 2026 23:13
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make the test suite faster

1 participant