Skip to content

A Triage run discards its whole analysis over a questionnaire or field-shape mismatch #2019

Closed

Description

@JacobStephens2

This was generated by AI during triage.

What happened

A Triage run whose session wrote a full recommendation fails as a whole because one field is slightly off. Five runs failed this way on 2026-11-10 (thirdshift 0.27.2, Claude Code, across three private repositories):

  • 4 runs: the Triage questionnaire is required only for needs-info or ready-for-human issues: <issue title>
  • 1 run: the Triage recommendation is not valid recommendation JSON: unknown field 'comments', expected one of 'title', 'body', 'category', 'state', 'confirmation_blocked', 'questionnaire'. The session nested comments inside an issue object instead of at the top level.

Where

parse_recommendation in src/triage.rs:~210:

  • ensure! requires a questionnaire exactly when the issue's final state is needs-info or ready-for-human, and forbids one otherwise.
  • serde rejects unknown fields.

Either mismatch discards the investigation and every proposed issue.

Expected

A recoverable shape error doesn't throw the analysis away. Possible approaches:

  • drop a questionnaire that isn't needed;
  • downgrade to needs-triage when a needed questionnaire is missing;
  • give the session one Repair turn quoting the error.

Then a failed run is kept for a real problem only.

Activity

  1. added
    bugSomething isn't working
    needs-triageMaintainer needs to evaluate this issue
    in-progressA Claim: thirdshift has taken this issue
    and removed
    needs-triageMaintainer needs to evaluate this issue
    on Nov 11, 2026
  2. JacobStephens2 commented on Nov 11, 2026

    @JacobStephens2
    OwnerAuthor

    This was generated by AI during triage.

    Agent Brief

    Confirmed on selected Base commit de91156. Keep #2019 as one standalone Ticket: give an invalid Triage recommendation one bounded correction turn before failing.

    Diagnosis and evidence

    • src/triage.rs:96-131 uses serde deny_unknown_fields. A scratch executable using these exact structs accepted a valid control and reproduced the reported unknown-field error when comments was nested inside an issue.
    • src/triage.rs:210-269 rejects missing, blank or unnecessary questionnaires after resolving each issue's effective state, including confirmation_blocked forcing ready-for-human.
    • src/triage.rs:566-572 propagates the parser error immediately. Publication and the 'Triage analysis retained' notification context occur later, so malformed recommendations never reach them. Raw Session traces survive; the investigation is unavailable through the normal recommendation/publication path rather than erased from all storage.
    • tests/triage_run.rs:464 confirms questionnaire mismatches currently fail in both filing and recommendation-only mode. tests/triage_issue.rs:325 confirms existing-issue validation rejects invalid recommendations without mutation.

    Required behavior

    1. When a successfully completed investigation returns a recommendation that fails parsing or questionnaire validation, allow exactly one correction turn. Supply the original investigation instructions, rejected final message and exact validation error. Preserve their distinction between trusted instructions and untrusted evidence. Request a corrected complete recommendation without repeating the investigation.
    2. Run correction within the same worktree and launch policy, including the read-machine sandbox when applicable. Give it a distinct Session-trace identity so the initial trace and Model evidence remain intact.
    3. Revalidate the complete corrected recommendation through the existing parser, existing-issue checks and publication checks before any publication. Preserve strict unknown-field rejection, questionnaire requirements, confirmation_blocked handling, readiness permission, category checks and comment-target checks. In particular, do not downgrade a missing questionnaire to needs-triage: existing-issue mode forbids that outcome.
    4. A valid initial recommendation uses no correction turn. A second invalid recommendation fails with useful diagnostics and references to retained analysis. Harness execution failures and GitHub/publication failures do not trigger this correction. Preserve private Security-finding notification protections.

    Validation

    Add regressions for unnecessary and missing questionnaires, misplaced comments, and a recommendation containing both valid and malformed issue entries. Verify successful correction preserves the complete investigation and all intended issues/comments; repeated malformed output stops after one correction; no publication happens before validation; recommendation-only remains mutation-free; and existing-issue cardinality and Agent Brief requirements still hold. Cover Report and existing-issue modes. Update prompt/documentation sources and regenerate derived pages if their contracts change. Run scripts/affected-tests main.

    Investigation results

    Both focused existing tests passed: cargo test --test triage_run questionnaires_are_required_only_for_issues_waiting_for_a_person -- --exact --nocapture, and cargo test --test triage_issue recommend_only_rejects_invalid_recommendations_without_mutation -- --exact --nocapture. The first exercises 16 mismatch cases. No source or Git-history changes were made.

  3. added
    ready-for-agentFully specified, ready for an AFK agent
    in-progressA Claim: thirdshift has taken this issue
    and removed
    in-progressA Claim: thirdshift has taken this issue
    ready-for-agentFully specified, ready for an AFK agent
    on Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions