Skip to content

A Run the host ran out of disk space under hands its issue back, not agent-failed (#2092) - #2111

Merged
JacobStephens2 merged 4 commits into
mainfrom
issue-2092
Nov 12, 2026
Merged

JacobStephens2 merged 4 commits into
mainfrom
issue-2092

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

A Run whose failure was caused by the host running out of disk space now hands its issue back instead of charging the failure to it. Out of disk means No space left on device from the OS or git, during worktree creation, a Session trace write, a push and so on.

 claim_ending(failed run)
-  Failed                        # agent-failed if work is on origin
+  Ending::failed(error)
+    out_of_disk(error)?  -> OutOfDisk
+    otherwise            -> Failed (unchanged)

 Claim::end
+  OutOfDisk  -> release even with work on origin:
+                in-progress off, ready-for-agent back,
+                no agent-failed, no Recovery comment
+                (a Security fix still gets agent-failed)
  • failed_run::out_of_disk walks the cause chain. It returns true for an io::Error of kind StorageFull, or for a layer whose text contains No space left on device.
    • Chains containing a SessionFailure are never out of disk, because the agent can write its diagnostic.
    • Chains containing TicketsNotDone are never out of disk, because a Spec run's Ticket causes can mix a full disk with genuine failures.
    • Chains containing a Base fix's ChildEnded are never out of disk, because the parent reads the Base fix's cause back as text that can carry its agent's diagnostic.
  • No agent-failed label means the Failure limit doesn't count the failure.
  • The put-back ready-for-agent is a hand-back, so Pickup retakes the issue only after its settling window. Host limits: wait for free disk space instead of starting sessions that fail on a full disk #2091's disk limit then holds the session until space is free, so a full disk doesn't cause a rapid retry loop.
  • The Command log says what to do:
    • The release line reads …, without agent-failed: the host ran out of disk space; free disk space on it, then retry the Command.
    • The same advice follows the cause on stderr.
    • The Recovery lines gain Before retrying: the host ran out of disk space; ….
  • ADR 0030 records this as an amendment to ADR 0029's "whatever kind of Command failed" rule.
  • Updated docs: the glossary (Claim, Failure limit), the guide, and the hand-written site pages (site/index.html, how-it-works, architecture).

Evidence

Execution evidence

The reproduction is a fake-adapter scenario: a git wrapper on PATH fails git worktree add (or git push) with git's No space left on device message, on an issue whose Issue branch is already on origin.

  • Before (production code at main, the new tests applied): cargo test --test out_of_disk
    test a_continuation_whose_worktree_cannot_be_written_for_lack_of_disk_is_handed_back_ready ... FAILED
    test a_push_that_runs_out_of_disk_hands_the_issue_back_and_keeps_earlier_work_on_origin ... FAILED
    test a_failure_that_is_not_out_of_disk_still_labels_agent_failed ... ok
      left: ["bug", "agent-failed"]
     right: ["bug", "ready-for-agent"]
      left: ["agent-failed"]
     right: ["ready-for-agent"]
    
    stderr before: thirdshift: labelling #7 agent-failed, in place of in-progress, so it waits for the Day shift
  • After: cargo test --test out_of_disk
    test a_continuation_whose_worktree_cannot_be_written_for_lack_of_disk_is_handed_back_ready ... ok
    test a_push_that_runs_out_of_disk_hands_the_issue_back_and_keeps_earlier_work_on_origin ... ok
    test a_failure_that_is_not_out_of_disk_still_labels_agent_failed ... ok
    test result: ok. 3 passed; 0 failed; 1 ignored
    

Confirmed cause: run::claim_ending mapped every failure to claim::Ending::Failed. For a run with work on origin, such as a Continuation's Issue branch, Ending::Failed swaps in-progress for agent-failed. Nothing distinguished a host cause from an issue cause.

Inline tests:

  • claim::tests (47 passed): covers the release with work on origin and a Security fix that still gets agent-failed.
  • failed_run::tests (3 passed): covers ENOSPC as an io::Error and as git text, and agent-written diagnostics and TicketsNotDone excluded. The exclusion test failed against the first classifier.
  • run_ending::tests (26 passed): covers the advice line and the parent's read-back of the cause.

scripts/affected-tests main:

  • npm test --prefix site passed, and npm run docs:check --prefix site passed.
  • cargo test had two timing failures, each passing on rerun. Neither test goes near the Claim:
    • process::findings::merged_fix_is_rechecked_after_a_later_fix_fails also flakes on origin/main (1 of 6 runs).
    • triage_pass::recommendation_only_pass_reserves_without_claiming_or_publishing_and_notifies panicked with "the agent never started" after 120s, and passed on rerun.
  • Every binary cargo test skipped after that failure was rerun and passed.

Visual evidence

None needed. The site changes are prose inside existing paragraphs, with no layout, CSS or markup structure changed.

Merge Danger

Door: two-way

Reverting restores the old labelling. Issues already released keep ready-for-agent, which is the state the Day shift restored by hand on 2026-11-12.

Blast Radius: labelling

A real issue failure would escape agent-failed and the Failure limit only if its factory-owned cause text says No space left on device. Agent diagnostics, Spec-run Ticket causes and Base fix endings are excluded.

Unaddressed findings

Standards

  • command_log duplicates tests/process.rs:767. Declined. Each integration test binary only shares tests/support, and moving this helper there is unrelated churn.
  • Shotgun Surgery: the advice bypasses base_fix::Advice. Declined. Advice is the Base fix's labelled advice (Base check:/Retry with:). The out-of-disk advice is one constant, failed_run::OUT_OF_DISK, decided by one predicate, failed_run::out_of_disk / says_out_of_disk.
  • Primitive Obsession: substring predicate, no FailureCategory. Declined. By the time run_ending sees the cause it is already text (Cause::full). A new recorded category belongs with the Console display; see the Spec entry below.

Security

None. The Security review found one introduced finding and fixed it: a Base fix agent's diagnostic naming a full disk released the parent Run's Claim without agent-failed. Its regression test is tests/out_of_disk.rs::a_base_fix_agent_saying_the_disk_is_full_does_not_release_the_run_that_waited_on_it.

Spec

  • Session trace write scenario ("The same holds when a Session trace write … fails with no space left"). Declined for an integration scenario: no unattended seam yields ENOSPC on the trace write. The trace file is freshly named, and making its directory unwritable gives EACCES, not ENOSPC. The classifier's handling of an io::Error with StorageFull from any layer is covered by failed_run::tests::running_out_of_disk_is_recognised_anywhere_in_the_cause_chain. The push case is covered end to end and pins the exact Preserved: line.
  • Inbox / Console shows the failure as a host problem. Not implemented, and filed as Inbox: show a Run that failed because the host ran out of disk as a host problem, not an issue failure #2116 (needs-triage). It is not in the acceptance criteria. How the Inbox (ADR 0023) groups a machine-level failure across runs is a UI decision for the Day shift.

Review coverage: both reviewers read every changed file except site/docs/index.html, which is generated from docs/guide.md and verified with npm run docs:check --prefix site. The second commit's new file, docs/adr/0030-a-full-disk-is-the-hosts-failure-not-the-issues.md, and the later edits to src/recovery.rs were written in response to the review and were not re-reviewed.

Closes #2092

Built with claude 2.1.314 · opus · default effort

Security review outcome

No unaddressed introduced findings.

JacobStephens2 and others added 2 commits November 12, 2026 09:02
…agent-failed (#2092)

A failure whose cause chain says No space left on device (the OS's
ENOSPC or git's wording) ends the Claim with a new OutOfDisk ending: the
Claim is released even with work on origin, ready-for-agent goes back,
no agent-failed and no Recovery comment, so the Failure limit does not
count it. A Security fix still waits for the Day shift. The Command's
cause is followed by advice to free disk space.

Co-Authored-By: Claude <noreply@anthropic.com>
…very line, ADR 0030

A failed Harness session's diagnostic and a Spec run's Tickets not done
never count as running out of disk. The out-of-disk release shares the
ordinary release's labelling, the Recovery lines say to free disk space,
the tests use write_executable and pin the preservation line, and ADR 0030
records the amendment to ADR 0029.

Co-Authored-By: Claude <noreply@anthropic.com>
JacobStephens2 and others added 2 commits November 12, 2026 09:49
 security review)

The parent Run read a failed Base fix's cause back as plain text, so a
Base fix agent's session diagnostic naming "No space left on device"
made the parent's failure look like the host's: its Claim was released
with work on origin, without agent-failed, and the Failure limit did not
count it. A Base fix's ending is now a typed ChildEnded error that
out_of_disk never counts, and the out-of-disk advice follows the typed
check instead of matching the cause's text.

Co-Authored-By: Claude <noreply@anthropic.com>
@JacobStephens2
JacobStephens2 merged commit ca8065f into main Nov 12, 2026
24 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-2092 branch November 12, 2026 15:00
@JacobStephens2 JacobStephens2 mentioned this pull request Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A host out of disk space fails the issue: agent-failed and a Failure limit count for a cause unrelated to the issue

1 participant