Repository navigation
ci: cut Docker suite time with a lighter healthcheck and two parallel jobs - #19
Merged
Merged
Conversation
The Docker job took 520-622s of test time on normal runs and hit the 1200s limit on a slow runner. A matrix now runs two groups on separate runners: group 2 has the crash, journal, broadcast and restart tests, group 1 everything else. run_test skips tests outside TEST_GROUP, so the test list itself is unchanged, and an unset TEST_GROUP (local make target) still runs every test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The orchestrator slept a fixed 5s before running tests. It now waits until each node's healthcheck passes, up to 120s per node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The healthcheck ran ls every 2s on all four nodes. Each readdir takes the root directory lock and reads its whole 1MB block with 256 O_DIRECT sector reads, competing with the tests for the lock and the disk. stat still fails when the mount is gone or the daemon is dead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stress Single Node made 300 docker exec calls and Many Files 200. Each loop now runs inside one exec and reports failures per iteration. The file operations through the mount are the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Docker suite runs in about a third of the time and no longer comes near its 1200s limit. All 33 tests still run.
Main cause: the healthcheck ran
lsevery 2s on all 4 nodes. Each readdir takes the root directory lock and reads its whole 1MB block, and writers back off up to 1s while a reader holds it. Withstatinstead, every test got faster, including ones this PR doesn't touch (Rolling Restart 45s → 9–14s, Broadcast Bidirectional 25s → 1s).Also:
run_testfilters byTEST_GROUP, so the open PRs that add tests don't conflict. Unset runs everything.docker execinstead of 300 and 200.Kept: the journal tests' ~12s waits after a kill. A restarted node only reclaims its old slot once its heartbeat is over 10s stale, so those waits are coverage.
Merges cleanly with #13, #15, #16 and #17. The check names change to
Multi-Node Docker Tests (group N); main has no required checks.🤖 Generated with Claude Code