Skip to content

#40 test: deterministic reclaim test; sweep of timing margins in tests/ - #41

Merged
gregoryfoster merged 5 commits into
mainfrom
40-deterministic-reclaim-test
Oct 8, 2026
Merged

gregoryfoster merged 5 commits into
mainfrom
40-deterministic-reclaim-test

Conversation

@gregoryfoster

@gregoryfoster gregoryfoster commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Closes #40. Test-only: no deploy (tests/ isn't a runtime path).

The strand test: no wall clock

The hand-off's trace confirmed. With a 0.6 s in-process extraction, the old test fails with ['cmd-1', 'cmd-1', 'cmd-2']: step 1 took longer than 500 ms, so the second walk reclaimed cmd-1 again. It never failed in 30 runs under CPU load, so the flake is rare.

Now XCLAIM … IDLE 60000 JUSTID, run as admin, gives both entries 60 s of idle time. They stay with the dead consumer, and JUSTID means no delivery is counted. The threshold is reclaim_min_idle_ms=30_000, and the sleep(0.6) is gone. Scratch Redis and CI are both 7.0.15, which accept IDLE.

  • It passes with a 2 s slow step 1, where the old test broke at 0.6 s.
  • It still proves its point. With the mutation count=1 → count=2 in Consumer.reclaim (claim both entries at once), it fails ['cmd-1'] == ['cmd-1', 'cmd-2'].

Sweep of tests/

Fixed:

Test Margin Fix
test_drift.py::test_every_call_shares_one_deadline 1.5×: a 0.2 s delay against a real 0.3 s deadline. Overhead over 0.1 s fails the first call A fake monotonic for the deadline. The second call gets 0.1 s against a 0.5 s stall (5×), and the first call has no timing in it. Mutation timeout=REQUEST_TIMEOUT_SECONDS (no shared deadline) → fails
test_deploy.py::test_refuses_while_another_deploy_holds_the_lock A race, not a margin: it polled for the lock file, but flock(1) creates the file before it holds the lock The test holds the lock itself (fcntl.flock). Mutation: deploy.sh skips flock -n → fails

Listed, safe (≥ 2× or one-sided):

Site Why safe
test_consumer.py test_run_stops_promptly: sleep(0.5) Only lets the loop run. Any length passes. The stop bound is 5 s against a 100 ms block (50×)
test_consumer.py broker-name retry: poll 10 ms, asyncio.timeout(10) A bounded condition wait
test_consumer.py stop-during-reclaim reclaim_min_idle_ms=0: no clock
test_main.py e2e: two polls, 100 × 0.1 s Bounded condition waits. The test takes about 2.6 s against 10 s per wait (about 4×)
test_child.py test_timeout_kills_the_child: timeout_s=2, < 10 5×
test_child.py terminal interrupt: 5 s pid poll, 1.5 s child sleep A bounded wait for startup. SIGINT lands within one 50 ms poll of the pid write, against 1.5 s (30×)
test_contain.py undumpable: 10 s ready poll A bounded condition wait
test_checkin.py test_a_stall_is_cut_off: 0.5 s stall against a 0.1 s timeout 5×, and load only lengthens the stall
test_deploy.py CI wait: < 15 s against a 30 s poll The test takes 0.18 s

Verification

  • ruff check, ruff format --check: clean. Full suite: 554 passed, 4 skipped.
  • On co-processor, under 4 CPU-hog processes on 2 vCPU: the strand and deadline tests 50/50, then tests/test_consumer.py 50/50. The hogs' 900 s timeout ran out near the end of that second loop, so its last few runs had no load.

Review (CR --fix)

Item Action Result
1 Fixed test_drift.py: the clock now reads 9.875, so the call has 0.125 s left, and the test matches read timeout=0.125). match="Timeout" had let a left / 2 mutation pass; it fails now
2 Fixed test_main.py e2e: both waits are now bounded at 30 s by asyncio.timeout, and a processor run that exits is named at once. A failed start used to fail the test 21 s later with an unpacking error, then a ProcessLookupError from the SIGTERM; now it fails in 1.7 s with "processor run exited before its loop was up"

Gate after CR 2: ruff clean, uv lock --locked ok, full suite 554 passed / 4 skipped, scratch Redis clean. Both touched tests passed 20/20 in a loop.

🤖 Generated with Claude Code

gregoryfoster and others added 5 commits October 8, 2026 13:43
…sleep against 500 ms

A slow step 1 (over 500 ms) let the second walk reclaim cmd-1 again: facts
['cmd-1', 'cmd-1', 'cmd-2'], reproduced with a 0.6 s extraction. Both entries now
start 60 s idle (JUSTID: no delivery counted) against a 30 s threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…5x margin

0.2 s delay against a real 0.3 s deadline: an overhead past 0.1 s failed the
first call. A fake monotonic leaves the second call 0.1 s against a 0.5 s stall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It polled for the lock file, which flock(1) creates before it holds the lock:
a deploy in that window took it. fcntl.flock in the test closes the window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…25 s left

match="Timeout" passed any short bound (a left / 2 mutation passed); 9.9 left
0.0999…, which no message can match exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A start past 10 s issued the command before the group (created at `$`), and
the test failed 20 s later on an unpack and a ProcessLookupError. Now 30 s
bounds, an exited process named at once, and no SIGTERM to a process gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_an_entry_that_raises_does_not_strand_the_ones_behind_it (0.6 s sleep against 500 ms idle), and since #34 a flake blocks deploys

1 participant