Skip to content

ci: cut Docker suite time with a lighter healthcheck and two parallel jobs - #19

Merged
mattmattox merged 4 commits into
mainfrom
claude/docker-suite-time
Oct 24, 2026
Merged

mattmattox merged 4 commits into
mainfrom
claude/docker-suite-time

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

The Docker suite runs in about a third of the time and no longer comes near its 1200s limit. All 33 tests still run.

CI run (same hour) Test step Job wall
main, c5d9df4 572s 658s
this branch, one job (test variant) 255s 339s
this branch, two jobs (2 runs) 143s / 135s (slower group) 239s / 212s

Main cause: the healthcheck ran ls every 2s on all 4 nodes. Each readdir takes the root directory lock and reads its whole 1MB block, and writers back off up to 1s while a reader holds it. With stat instead, every test got faster, including ones this PR doesn't touch (Rolling Restart 45s → 9–14s, Broadcast Bidirectional 25s → 1s).

Also:

  • A matrix splits the suite into two parallel jobs. run_test filters by TEST_GROUP, so the open PRs that add tests don't conflict. Unset runs everything.
  • Stress Single Node and Many Files run their loops in one docker exec instead of 300 and 200.
  • The orchestrator polls node health instead of sleeping 5s.

Kept: the journal tests' ~12s waits after a kill. A restarted node only reclaims its old slot once its heartbeat is over 10s stale, so those waits are coverage.

Merges cleanly with #13, #15, #16 and #17. The check names change to Multi-Node Docker Tests (group N); main has no required checks.

🤖 Generated with Claude Code

mattmattox and others added 4 commits October 24, 2026 13:25
The Docker job took 520-622s of test time on normal runs and hit the
1200s limit on a slow runner. A matrix now runs two groups on separate
runners: group 2 has the crash, journal, broadcast and restart tests,
group 1 everything else. run_test skips tests outside TEST_GROUP, so
the test list itself is unchanged, and an unset TEST_GROUP (local make
target) still runs every test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The orchestrator slept a fixed 5s before running tests. It now waits
until each node's healthcheck passes, up to 120s per node.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The healthcheck ran ls every 2s on all four nodes. Each readdir takes
the root directory lock and reads its whole 1MB block with 256 O_DIRECT
sector reads, competing with the tests for the lock and the disk. stat
still fails when the mount is gone or the daemon is dead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stress Single Node made 300 docker exec calls and Many Files 200. Each
loop now runs inside one exec and reports failures per iteration. The
file operations through the mount are the same.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit a361a42 into main Oct 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant