Repository navigation
Description
Problem
Four outside-session e2e tests fail under heavy load on epic/525-session-adoption at bb10fff. They passed 12 of 12 run alone, and the full suite passed on a quiet box.
g1_outside_claude_session_id_test.go:108: list held 1 of the 2 planted ended rows
g1_outside_adopt_test.go:591: kind bg: no row in {...}
g1_outside_ended_test.go:500: ended rows = [], want only the target's own row
g1_outside_sessions_test.go:494: no WebSocket text frame within 5s
Seen once, in the full suite (-race -parallel 32) while eve's Playwright suite ran on the same 8-CPU box (load average about 20). Three tests are on the epic branch only; TestOutsideEndedRowGrantsNoAuthority is also on main.
Root cause
Under load a probe misses its 10 s bound, and the machine goes unreachable. Three of the four failure dumps log outside machine status changed ... machine_status: unreachable. The test helper osRig.probe() returns at the first snapshot whose lastProbeAt moved. A failed probe also moves lastProbeAt, so the test reads a snapshot without the probe's rows. Relay behaves as designed: the next probe recovers.
Fix
Test-only. osRig.probe() waits for a probe that completed: each machine's lastProbeAt moved and its status is not unreachable. After a failed probe it moves the clock again, within the same bound. Tests that expect an unreachable machine keep their own wait.
Found while working #532.
Contract
- Branch
fix/573-probe-waitoffepic/525-session-adoption; the PR targets that branch withCloses #573. - Ownership:
e2e/features/g1_outside_sessions_test.go,e2e/features/g1_outside_claude_session_id_test.go. Nothing underinternal/orcmd/. osRig.probe()andosRig.probeFrame()return only a snapshot from a completed probe: every machine'slastProbeAtmoved and its status is notunreachable. After a failed probe they move the clock again, within the same overall bound.TestOutsideReportForExitedProcessRemovesRow: find why its hook wait missed under load. A test-side fix stays in this ownership. A product cause goes back to the coordinator as a question.- No assertion is removed or weakened; no skip is added; no bound is lengthened.
- Waits: the WebSocket snapshot frames; no sleep.
- Repro: run the four tests with
-count=3while CPU burners load every core (e.g.for i in $(seq 16); do yes >/dev/null & done). Red before the fix, green after. Kill the burners after.
Activity
barelyworkingcode commentedon Oct 21, 2026
Coordinator approval: proven bug, 2026-10-21. Seen in a real suite run on the epic head; the fix is test-only and restores the tests' intended wait. Retro list.
Contract
fix/573-probe-waitoffepic/525-session-adoption; the PR targets that branch withCloses #573.e2e/features/g1_outside_sessions_test.go,e2e/features/g1_outside_claude_session_id_test.go. Nothing underinternal/orcmd/.osRig.probe()andosRig.probeFrame()return only a snapshot from a completed probe: every machine'slastProbeAtmoved and its status is notunreachable. After a failed probe they move the clock again, within the same overall bound.TestOutsideReportForExitedProcessRemovesRow: find why its hook wait missed under load. A test-side fix stays in this ownership. A product cause goes back to the coordinator as a question.-count=3while CPU burners load every core (e.g.for i in $(seq 16); do yes >/dev/null & done). Red before the fix, green after. Kill the burners after.barelyworkingcode commentedon Oct 21, 2026
Fixed by #574, merged into epic/525-session-adoption.