Skip to content

Give invalid Triage recommendations one correction turn - #2028

Merged
JacobStephens2 merged 5 commits into
mainfrom
issue-2019
Nov 11, 2026
Merged

JacobStephens2 merged 5 commits into
mainfrom
issue-2019

Conversation

@JacobStephens2

@JacobStephens2 JacobStephens2 commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

A completed Triage investigation previously failed immediately when its recommendation had a shape or questionnaire error. Give it one correction turn while keeping validation strict.

completed investigation
  validate complete recommendation
  invalid -> one correction in the same worktree and launch policy
             keep a distinct Session trace and Model evidence
             validate the complete corrected recommendation
  check publication -> file/update issues or recommend only

The correction receives the original instructions, rejected output and exact error as separately identified evidence, and preserves the full investigation and intended issues/comments without investigating again. Execution and GitHub/publication failures do not trigger correction. Private Security-finding notification protections remain in force.

Closes #2019

Evidence

Execution evidence

  • Before: cargo test --test triage_correction an_unnecessary_questionnaire_gets_one_correction_and_keeps_the_investigation -- --exact --nocapture

    assertion `left == right` failed:
    thirdshift: the Triage questionnaire is required only for needs-info or ready-for-human issues: Handle an empty cache
      left: Some(1)
     right: Some(0)
    test result: FAILED. 0 passed; 1 failed
    

    After: the same command passed: 1 passed; 0 failed.

  • Confirmed cause: successful Harness output reached parse_recommendation, whose validation error propagated before publication or a correction could run. The minimized trigger is one otherwise complete issue with an unnecessary questionnaire.

  • cargo test --test triage_correction: 9 passed, including missing/unnecessary questionnaires, misplaced comments, mixed entries, one-turn limits, complete publication, recommend-only, existing-issue cardinality/Agent Brief/readiness, post-correction validation, private notifications and real Bubblewrap confinement.

  • cargo test --test triage_run --test triage_issue --test triage_spec_regressions --test triage_pass: 52 passed.

  • cargo test --bin thirdshift prompts_page: 14 passed after regeneration; npm run docs:check --prefix site passed.

  • PLAYWRIGHT_BROWSERS_PATH=/tmp/thirdshift-issue-2019-browsers CARGO_BUILD_JOBS=3 RUST_TEST_THREADS=3 scripts/affected-tests refs/remotes/origin/main: passed (68 site checks and the complete Rust suite, including all 100 integration test files).

The initial scripts/affected-tests main selected unrelated history because local main is checked out in another worktree at 27f547958b9e5f43bb2616f7e42286f35534fe0a; it stopped immediately on unavailable Go. The final selection uses the specified refs/remotes/origin/main base at de91156c424fa54e95845f1e10e46afaf0bdf406. That worktree and local main were left untouched.

Test seams

  • Real thirdshift triage CLI in Report and existing-issue modes, with fake Harness responses and fake GitHub publication.
  • Session traces and Model evidence retained by that CLI, and the real read-machine sandbox launch where available.

Visual evidence

Manual, Triage runs section, same scroll state.

Desktop, 1440×900

Before
After

Mobile, 390×844

Before
After

CI repair validation

Merged main at 482507a4 in 2d05b78a, including #2027's Ubuntu DNS recovery before musl dependency installation. The reconciliation also includes upstream version 0.29.0 and domain documentation; it changes no application sources, Rust tests or site pages. The feature diff against the merged base remains the same 13 files.

  • Before reconciliation, PLAYWRIGHT_BROWSERS_PATH=/tmp/thirdshift-issue-2019-browsers CARGO_BUILD_JOBS=3 RUST_TEST_THREADS=3 scripts/affected-tests refs/remotes/origin/main passed: 68 site checks and 4,242 selected Rust tests. Rust wall time: 1682.52 seconds, including compilation.
  • After reconciliation, node --test site/tests/docs.test.cjs passed 7 tests, CARGO_BUILD_JOBS=3 RUST_TEST_THREADS=3 cargo test --test update passed 10 tests, and the built binary's version output matched thirdshift 0.29.0.
  • python3 -m unittest discover -s scripts -p test_generate_release.py passed 3 tests; python3 scripts/generate-release.py --check and bash -n scripts/release-dns.sh passed. The imported DNS helper also passed four isolated probes with mocked getent and sudo: healthy DNS, recovery through public resolvers, refusal to change DNS outside CI, and failure when recovery still cannot resolve a host. Host DNS was not changed.
  • Attempted dist build --artifacts=local --target x86_64-unknown-linux-musl --print=linkage --output-format=json and the same command with --target aarch64-unknown-linux-musl. This Rocky Linux host cannot complete those artifact builds: x86_64 lacks x86_64-linux-musl-gcc; aarch64 lacks cargo-zigbuild. Both attempts exited 255. Local musl artifacts are not verified.

Review coverage

Independent Standards and Spec reviews against refs/remotes/origin/main (de91156c424fa54e95845f1e10e46afaf0bdf406) each raised 0 findings. Each reviewer read all 13 changed files, including generated output; no unread changed files or exclusions remain. Reports and counts are retained in .thirdshift-review-7L9Yo7/{standards.md,spec.md,summary.json} beside the Session evidence.

Unaddressed findings

Standards

None.

Spec

None.

Security

None.

Security review outcome

No unaddressed introduced findings.

CI notes

  • At the original head 4e5d9a49, the x86_64 musl job and aarch64 musl job failed before compilation: APT could not resolve azure.archive.ubuntu.com / ports.ubuntu.com while installing musl-tools. The feature diff changes neither release setup nor dependency inputs. The upstream reconciliation above supplies the existing DNS fix without relaxing any check.
  • The unrelated console job failed on the same Ubuntu DNS error during Playwright dependency installation; the site job was cancelled during dependency installation before npm test. Those setup steps are unchanged by the feature diff. No additional fixes were made for them.

Merge Danger

Door: two-way

Reverting restores immediate failure on invalid recommendations; no schema migration.

Blast Radius: Triage

An invalid recommendation can spend one additional Harness turn before publication or failure. Valid initial recommendations use no extra turn.

Built with codex codex-cli 0.167.1 · gpt-6.1-sol · xhigh

@JacobStephens2
JacobStephens2 merged commit 4a301fa into main Nov 11, 2026
16 checks passed
@JacobStephens2
JacobStephens2 deleted the issue-2019 branch November 11, 2026 18:07
This was referenced Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A Triage run discards its whole analysis over a questionnaire or field-shape mismatch

1 participant