Skip to content

fix(test): assert recovery backoff on the claims, not worker request arrival - #9591

Merged
waleedlatif1 merged 1 commit into
stagingfrom
fix/recovery-storm-backoff-timing
Nov 3, 2026
Merged

waleedlatif1 merged 1 commit into
stagingfrom
fix/recovery-storm-backoff-timing

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • The recovery-storm "gives up on a run whose recovered controllers keep failing" cases asserted the takeover backoff on when the fake worker received each takeover's POST. That arrival includes the variable work between a claim and its POST (stream lease, replay-ring read, seq lookup, claim transaction), and since these cases run concurrently the first takeover's lag is stretched while later ones are not, so the first gap intermittently read ~500 ms against a 700 ms floor
  • The product behavior is fine: planRecovery stamps claimedAt/notBefore at planning time and refuses to claim before the previous notBefore, and claimRunController is a compare-and-set on the controller token, so claims can never be closer than the jittered floor
  • The test now records the backoff each successful claim persisted (wrapping claimRunController the same way the file already wraps the replay buffer) and asserts the contract directly: attempts run 1..MAX+1, each claim's claimedAt is at or after the previous notBefore, and each floor is at least 0.8 × 1 s × 2^i. Both stamps come from the same planning clock, so there is no slack to tune

Type of Change

  • Bug fix (flaky test)

Testing

  • Ran recovery-storm.integration.ts locally against disposable Postgres 17 + Redis: 14/14 pass
  • Instrumented run: the first POST gap was below the first claim gap in every concurrent case, and only the first takeover's claim→POST lag grew under CPU contention (~46 ms idle → ~125 ms loaded) while later lags stayed ~10–40 ms
  • Mutation check: removing the notBefore wait in planRecovery fails all five cases on the new assertions; restored afterwards

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Nov 3, 2026 1:29am UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@greptile-apps

greptile-apps Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] This test-only change appears safe to merge.

Summary

Updates the recovery-storm integration test to check claim timestamps instead of worker request arrival.

  • The recovery-storm test checks backoff when each takeover is claimed.

Reviews (1) · Last reviewed commit: "fix(test): assert recovery backoff on th..." · Reviewed by Greptile

@waleedlatif1
waleedlatif1 merged commit 395a394 into staging Nov 3, 2026
64 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/recovery-storm-backoff-timing branch November 3, 2026 01:35

This branch was previously deployed

1 inactive deployment
Preview — ff0d0c7b Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant