Skip to content

Outside-session e2e tests read a failed probe's snapshot under load #573

Closed

Description

@barelyworkingcode

Problem

Four outside-session e2e tests fail under heavy load on epic/525-session-adoption at bb10fff. They passed 12 of 12 run alone, and the full suite passed on a quiet box.

g1_outside_claude_session_id_test.go:108: list held 1 of the 2 planted ended rows
g1_outside_adopt_test.go:591: kind bg: no row in {...}
g1_outside_ended_test.go:500: ended rows = [], want only the target's own row
g1_outside_sessions_test.go:494: no WebSocket text frame within 5s

Seen once, in the full suite (-race -parallel 32) while eve's Playwright suite ran on the same 8-CPU box (load average about 20). Three tests are on the epic branch only; TestOutsideEndedRowGrantsNoAuthority is also on main.

Root cause

Under load a probe misses its 10 s bound, and the machine goes unreachable. Three of the four failure dumps log outside machine status changed ... machine_status: unreachable. The test helper osRig.probe() returns at the first snapshot whose lastProbeAt moved. A failed probe also moves lastProbeAt, so the test reads a snapshot without the probe's rows. Relay behaves as designed: the next probe recovers.

Fix

Test-only. osRig.probe() waits for a probe that completed: each machine's lastProbeAt moved and its status is not unreachable. After a failed probe it moves the clock again, within the same bound. Tests that expect an unreachable machine keep their own wait.

Found while working #532.

Contract

  • Branch fix/573-probe-wait off epic/525-session-adoption; the PR targets that branch with Closes #573.
  • Ownership: e2e/features/g1_outside_sessions_test.go, e2e/features/g1_outside_claude_session_id_test.go. Nothing under internal/ or cmd/.
  • osRig.probe() and osRig.probeFrame() return only a snapshot from a completed probe: every machine's lastProbeAt moved and its status is not unreachable. After a failed probe they move the clock again, within the same overall bound.
  • TestOutsideReportForExitedProcessRemovesRow: find why its hook wait missed under load. A test-side fix stays in this ownership. A product cause goes back to the coordinator as a question.
  • No assertion is removed or weakened; no skip is added; no bound is lengthened.
  • Waits: the WebSocket snapshot frames; no sleep.
  • Repro: run the four tests with -count=3 while CPU burners load every core (e.g. for i in $(seq 16); do yes >/dev/null & done). Red before the fix, green after. Kill the burners after.

Activity

  1. added
    bugSomething isn't working
    flakeTest that fails intermittently; every rerun-to-green links one
    on Oct 21, 2026
  2. barelyworkingcode commented on Oct 21, 2026

    @barelyworkingcode
    OwnerAuthor

    Coordinator approval: proven bug, 2026-10-21. Seen in a real suite run on the epic head; the fix is test-only and restores the tests' intended wait. Retro list.

    Contract

    • Branch fix/573-probe-wait off epic/525-session-adoption; the PR targets that branch with Closes #573.
    • Ownership: e2e/features/g1_outside_sessions_test.go, e2e/features/g1_outside_claude_session_id_test.go. Nothing under internal/ or cmd/.
    • osRig.probe() and osRig.probeFrame() return only a snapshot from a completed probe: every machine's lastProbeAt moved and its status is not unreachable. After a failed probe they move the clock again, within the same overall bound.
    • TestOutsideReportForExitedProcessRemovesRow: find why its hook wait missed under load. A test-side fix stays in this ownership. A product cause goes back to the coordinator as a question.
    • No assertion is removed or weakened; no skip is added; no bound is lengthened.
    • Waits: the WebSocket snapshot frames; no sleep.
    • Repro: run the four tests with -count=3 while CPU burners load every core (e.g. for i in $(seq 16); do yes >/dev/null & done). Red before the fix, green after. Kill the burners after.
  3. barelyworkingcode commented on Oct 21, 2026

    @barelyworkingcode
    OwnerAuthor

    Fixed by #574, merged into epic/525-session-adoption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingflakeTest that fails intermittently; every rerun-to-green links one

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions