[JacobStephens2]

Summary

A Scenario's Runs now start under a guardian that outlives a killed harness and ends everything the Run started.

 test harness (libtest)
-└── thirdshift                     # harness's process group; reparented to pid 1 on SIGKILL
-    └── fake codex / git / detached command (own groups/sessions) → pid 1, poll forever
+└── launcher (forked in pre_exec, exits at once)
+    guardian: subreaper, own process group, ignores INT/TERM/HUP/QUIT
+    ├── polls the Scenario's lease pipe (only the harness holds the writer)
+    └── thirdshift                 # still in the harness's process group, same Command/stdio
+        └── fakes, held git, detached commands → adopted by the guardian when orphaned
+
+ lease EOF (Scenario dropped, or harness dies by any signal)
+   guardian: SIGKILL + reap every child until ECHILD, then exit
+   Scenario drop waits (bounded by WAIT_BOUND) for every guardian

• tests/support/child_lifetime.rs: new OwnedRuns / OwnedRun, alongside the existing FixtureChild. The guardian is forked rather than exec'd, so it allocates nothing and runs only async-signal-safe calls. It reports the Run's pid and raw wait status over a socket. It keeps the Run unreaped until the test has read its status, as a Child does, so HeldRun::signal/kill never hit a reused pid.
• tests/support/mod.rs: run, run_with_input, run_from, run_with_env, run_until/run_from_until (and run_and_signal*), read_to_file and run_copy_replaced_midway spawn through OwnedRuns.
• tests/scenario_lifetime.rs: lifecycle tests with a subreaping supervisor.
• docs/testing.md: the new "Scenario Run lifetime" contract, beside the setup PTY and release ones.

Confirmed cause: Scenario started thirdshift as a plain child of the harness, and thirdshift starts each child in a process group of its own. A SIGKILLed harness never unwinds, so nothing ended those children. They were reparented to pid 1 and kept polling.

Evidence

Execution evidence

Lifecycle test: cargo test --test scenario_lifetime

• Before (this test against main's tests/support/{mod,child_lifetime}.rs): test result: FAILED. 0 passed; 6 failed, naming survivors, e.g.

  git 3709279 survived its fixture:
  thirdshift 3709228 survived its fixture:
  agent 3709660 survived its fixture:
  

• After: test result: ok. 6 passed; 0 failed; 2 ignored

It covers:

• A held fake agent plus a detached command that traps INT and TERM, and a held fake git.
• Each with SIGKILL of the harness alone and of its whole process group.
• Normal completion and assertion unwinding.

The issue's two reproductions: a throwaway script runs the test binary in its own process group under setsid, with a private TMPDIR. Once the marker exists, it SIGKILLs the group (or only the harness), waits, then lists processes whose cwd is inside that TMPDIR.

• Before (main):
  • process_tree a_command_started_during_the_stop_grace_period_is_killed_too, group kill: FAIL: 3 fixture processes survived (fake codex with ppid 1, bash -e claude-script.sh, …).
  • The same test, harness-only kill: FAIL: 4 fixture processes survived (thirdshift harness codex … with ppid 1, fake codex, its script, sleep 120).
• After: PASS: no fixture process survived, for both that test and pickup_run two_machines_racing_for_two_ready_issues_each_take_one, in both kill modes.

Full suite wall time: precompiled cargo test --no-fail-fast on a 4-core Ubuntu 24.04 host. Compilation is excluded.

	Wall time	Result
Before (main's support code)	702.9 s	3957 passed, 3 failed (pre-existing process findings::* flakes, see below)
After	711.4 s	3966 passed, 0 failed

scripts/affected-tests main selects the full Rust suite, and cargo nextest is unavailable here, so the full suite ran as cargo test.

Regressions fixed along the way:

• process findings::merged_fix_is_rechecked_after_a_later_fix_fails: a first version let wait return before the guardian reaped the Run. A second Run then saw the first's zombie pid as alive. The release is now synchronous: the guardian reaps the Run, then closes the socket.
• triage_run: run_with_input lost stdin because Child::wait on the launcher closes it. The pipes are now taken before that wait.

Pre-existing flakes, seen on main too:

• process findings::*: main's support code failed merged_fix_is_rechecked_after_a_later_fix_fails in 3 of 3 whole-binary runs.
• update console::* ETXTBSY at tests/update/console.rs:43: 1 of 6 runs on main. These tests write an executable in-process with fs::copy and exec it while a sibling test forks.
• The guardian's forks close inherited close-on-exec descriptors before forking again, so this change does not widen that window.

Merge Danger

Door: two-way

The change is test-only and touches no production code.

Blast Radius: test-suite

• Every Scenario Run now goes through the guardian, and Scenario drop waits up to WAIT_BOUND for cleanup. A descendant stuck in uninterruptible sleep now fails the test after 120 s instead of leaking.
• The guardian is Linux-only, like the existing child-lifetime helper. The Rust suite runs only on Linux in CI.
• A failed exec of thirdshift now shows as the Run exiting rather than a spawn error, because std's exec-error pipe is closed before the extra forks.

Fixtures with the same gap (out of scope)

These commands don't go through these Scenario methods, so their children can still outlive a killed harness:

• Console tests via Console::start_command(scenario.command(..)) (tests/console/*.rs).
• tests/update/console.rs:130.
• machine.command() in tests/console/fleet.rs and tests/console/other_accounts.rs.
• tests/eval_arms.rs:130.

Terminal Runs keep the setup PTY contract: immediate child only.

Unaddressed findings

Standards

All six Standards findings were addressed:

• Bounded drop.
• A doc note on the two guardians.
• A Report enum and a GuardianFds struct.
• Removed the pid placeholder and the redundant toggles.
• Doc wording and the terminal Runs.
• SAFETY notes, plus only unreaped adopted pids are killed.

Spec

• 3: no lifecycle test for the run/output() path. Declined. The acceptance criteria ask only for held-Run cases. Every Scenario method shares OwnedRuns::spawn, and the full suite's 3966 passing tests exercise run, run_with_input and read_to_file stdio through it. The triage_run stdin regression above was caught that way.
• 4: unrequested synchronous reap, and drop blocks. No change. It preserves the old Child semantics (see the process regression above), and the drop is now bounded by WAIT_BOUND.
• 8: depends on /proc/thread-self/children. No change. FixtureChild already depends on the same interface (/proc/self/task/*/children).

Security

None. The change is test-support code only: the guardian's lease, report socket and done pipe are anonymous, close-on-exec descriptors, and it signals only its own unreaped children, so no lower-trust principal or trust boundary is involved.

Review coverage: both reviewers read all four changed files; none left unread.

Closes #1912

Built with claude 2.1.314 · opus · default effort

Security review outcome

No unaddressed introduced findings.