Skip to content

Retry transient CI check reads with bounded interruptible backoff - #761

Merged
JacobStephens2 merged 2 commits into
mainfrom
issue-731
Oct 17, 2026
Merged

JacobStephens2 merged 2 commits into
mainfrom
issue-731

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Retry transient read-only CI observations on the same repository and full commit across the watch, Base comparison, and Check re-run reconciliation.

read checks
  transient failure → wait 1s → retry → wait 2s → final attempt
  permanent or unknown failure → causal error
  cancellation → stop reads and backoff

Closes #731

Evidence

  • Before: cargo test --bin thirdshift ci::tests::a_truncated_initial_check_read_recovers_without_losing_pending_or_red_checks -- --exact failed with EOF while parsing a value.
    After: the regression passes; all 36 CI seam tests, 14 GitHub executable-seam tests, and 23 CI integration tests pass. Coverage includes exhaustion at three attempts, cancellation, permanent errors, real pending/red states, and preserved Check re-run protections.
  • Full cargo test passed (1864 tests passed, 2 ignored). The final EOF regression also passes.
  • cargo check --all-targets, cargo fmt --check, and cargo clippy --all-targets -- -D warnings pass.

Merge Danger

Door: two-way

Revert to restore immediate failure on CI read errors; no persisted state or configuration changes.

Blast Radius: CI

Transient check-read failures can delay an observation by up to three seconds of backoff. Workflow mutations and Repair allowances are unchanged.

Test seams

  • CI watch_through / rerun_through through the existing Outside observation, wait, and progress seam: deterministic recovery, exhaustion, cancellation, and preserved CI facts and Check re-run protections.
  • GitHub checks_on through the existing executable seam: classify real gh exit/JSON failures while retaining diagnostics and interruption behavior.

Unaddressed findings

Standards

None. The independent Standards review found no violations or actionable smells.

Spec

None. The one P2 finding (ordinary transport EOF was not retried) was reproduced with the reviewer's exact command, python3 /tmp/issue-731-eof-repro.py (exit 101), fixed in e1e1028, and verified with the unchanged reproduction (pass). URL-prefixed EOF, bare EOF, and permanent HTTP 401 plus EOF remain in check_reads_classify_transport_and_api_failures_without_hiding_diagnostics as regression coverage. The Spec reviewer confirmed the fix.

Security

None. Reviewed all six changed files in ed01f04b742bd94ab8637478fa08bb51c50932b5...HEAD (HEAD e1e1028) and supporting CI, GitHub, polling, process interruption and Delivery code. No introduced security findings. All 36 CI seam tests and 14 GitHub executable-seam tests passed in an offline, network-isolated, resource-limited local sandbox.

Security review outcome

No unaddressed introduced findings.

Review coverage

Standards and Spec each read all six changed files, including the final patch. Unread changed files: none. Both reports, including each axis's final files-read list, are preserved in .thirdshift-review-PelNE4/standards.md and .thirdshift-review-PelNE4/spec.md in the session worktree.

Built with codex · gpt-6.1-sol · high

@JacobStephens2
JacobStephens2 merged commit 8b2af14 into main Oct 17, 2026
18 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-731 branch October 17, 2026 15:55
@JacobStephens2 JacobStephens2 mentioned this pull request Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry transient CI check reads within a bounded interruptible budget

1 participant