Skip to content

test: add a multi-node soak driver with a model, faults and replay - #152

Merged
mattmattox merged 8 commits into
mainfrom
test/multinode-soak
Nov 14, 2026
Merged

mattmattox merged 8 commits into
mainfrom
test/multinode-soak

Conversation

@mattmattox

Copy link
Copy Markdown
Contributor

goclusterfs-soak runs a random workload on a 3-VM cluster and checks every node against an in-memory model. tests/soak/README.md covers the details and how to run it.

Parts

  • A static agent on each VM runs operations over one ssh session.
  • A bash controller sources the VM harness and owns the VMs.
  • Concurrent lanes run on their own files and snapshots, in directories they share.
  • After every op, each node must show the model within -window. Checks slower than the documented 6 s are logged.
  • Faults: power cut, SIGKILL of goclusterfs-mount, or SIGSTOP past the heartbeat timeout. The victim's in-flight ops drain first, and the other nodes keep working through the fault.
  • Hourly fsck runs on a quiesced cluster. Anything other than leaks and orphans is a failure.
  • ops.jsonl logs every op, fault and fsck; replay runs a log again in order.

It runs on .66 inside a systemd-run scope, under its own lock and ports, so normal VM runs still share the host.

What it found

Not in this PR

🤖 Generated with Claude Code

mattmattox and others added 8 commits November 14, 2026 09:15
goclusterfs-soak runs a random mix of operations from several lanes on a
3-VM cluster and checks every node against a model within the coherence
window. It cuts power, SIGKILLs or stops nodes past the heartbeat timeout,
runs fsck from a quiesced cluster every hour, logs every operation, and
replays a log in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… names are cut to 15 characters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mand line starts with ld-linux

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the coherence window

Operations cut short by the end of the run were reported as failures. The
window grows to 30s, because reads on the surviving nodes wait out a failed
node's detection, and every check slower than the documented 6s is logged
as a slow record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mattmattox
mattmattox merged commit 507b942 into main Nov 14, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant