Skip to content

Run the Security review on security.harness when it is set #910

Closed

Description

@JacobStephens2

This was generated by AI during triage.

A Security review always runs on its Command's Harness, so when that Harness's model refuses security work there is no way to get the review done unattended. Spec run #776 lost its Security review this way, and the Day shift wants to move toward Commands that use more than one Harness.

Evidence

Spec run #776 on stephens, Spec PR #842, 2026-10-20 (UTC):

  • 15:51-15:56: the Security review session ran on Codex (gpt-6.1-sol), the Command's Harness from harness.default = "codex". While it read the repository's thirdshift-security-audit skill files and console/reproductions/session-stop.py, the turn failed with OpenAI's "This content was flagged for possible cybersecurity risk. ... apply for Daybreak access ..." message.
  • thirdshift classified it correctly as Codex's cybersecurity safeguard refusal (Safeguard refusals fail a Security run, and each security session's Model is logged #547). As the glossary says, that held the result like an unaddressed Security finding, and the Spec run ended failed at 20:27 after its CI went green.
  • The Spec adds email/password accounts, login codes, authenticated Stop and a sudo install script, so it is exactly the change the Security review is for, and it reached the Day shift with no Security review.

[security] harness exists, but it chooses only a Security run's Harness. The Security review inside a Run, Spec run or Base fix ignores it.

Decision (Day shift, 2026-10-20)

Option (a): a Security review runs on security.harness when that is set. Update the glossary's one-Harness-per-Command rule accordingly. This is a first step toward letting one Command use different Harnesses for different sessions.

Rejected: (b) retry once on a fallback Harness after a refusal. It adds retry logic and still spends a refused session first.

Related

Activity

  1. JacobStephens2 commented on Oct 20, 2026

    @JacobStephens2
    OwnerAuthor

    This was generated by AI during triage.

    Agent Brief

    Category: enhancement
    Summary: Run the Security review on security.harness when it is set, so one Command can use a second Harness for its security session

    Current behavior:
    Every session of a Command (implement, Spec review, Security review, Resumes and Repairs, and those of its child Runs) runs on the one Harness the Command chose: its harness word, else harness.default, else Claude Code. [security] harness chooses the Harness only for a Security run. A Security review that the Command's Harness refuses holds the result, and nothing tries elsewhere.

    Desired behavior:

    • The Security review session uses security.harness when it is set, wherever a Security review runs: a Run, a Spec run on its Spec PR, or a Base fix that asked for one. Its Model and Effort come from that Harness's own [harness.<name>] section, as a Security run's do. When security.harness is blank or missing, the Security review uses the Command's Harness, as today.
    • A Command's harness word does not override security.harness for the Security review. The word chooses the Command's Harness; the User config's security choice wins for security sessions. A command-level override for the security session is out of scope.
    • Every other session is unchanged: implement, Spec review, Resumes, Repairs (including any started after the Security review) and child Runs keep the Command's Harness.
    • The Harness check covers the security Harness. A Command that will run a Security review on a different Harness checks that Harness (and its Model/Effort, as the existing check does) before any work starts, and fails before work if it can't run, as for the main Harness.
    • What it ran on is visible. The Security review's Session log, its .models.json, the Command log's session lines and the Run notification's "Session models" list name the security session's Harness, not the Command's. The "Built with" line keeps naming the Command's Harness. If several Harnesses ran, it may list the security one too, for example · security review on claude · claude-opus-5-5.
    • The Recovery command printed after a failure still reproduces the same choice. Since the choice comes from the User config, no new word is needed. Confirm that a Continuation's Security review again uses security.harness.
    • Docs and decision records:
      • The glossary's Harness entry now says a Command's sessions run on its Harness except the Security review, which runs on security.harness when set.
      • The Security review entry says the same.
      • docs/guide.md's [security] comment and Harness section are updated.
      • Setup's comment for security.harness now says it covers Security reviews as well as Security runs.
      • A new ADR records that a Command may use more than one Harness, starting with the Security review. It says why (refusals by the Command's Harness, Run the Security review on security.harness when it is set #910), and that per-session Harness choice is the direction.

    Key interfaces:

    • The Harness, Model and Effort resolution shared by Security runs (command word, then security.harness, then harness.default, then Claude Code). The Security review needs a variant that skips the command word.
    • Where a Run, Spec run or Base fix starts its Security review session, and the per-session Choice it passes to the Harness.
    • The Harness check before work starts.
    • Session model recording (.models.json) and the Run notification's session list.

    Acceptance criteria:

    • With harness.default = "codex" and security.harness = "claude", a Run's implement and Repair sessions launch Codex. Its Security review launches Claude Code with [harness.claude]'s Model and Effort. Shown with the test fakes for both Harnesses.
    • The same for a Spec run's Security review on its Spec PR, and for a Base fix that asked for one.
    • With security.harness blank or missing, the Security review runs on the Command's Harness (existing tests pass).
    • A command harness codex word with security.harness = "claude" still runs the Security review on Claude Code.
    • A security Harness whose check fails fails the Command before any work, with a cause naming that Harness.
    • The Session log, .models.json, Command log and Run notification name the Security review's actual Harness.
    • Glossary, guide, Setup comment and the new ADR are updated. cargo test, cargo clippy --all-targets -- -D warnings and cargo fmt --check pass.

    Out of scope:

    • Choosing Harnesses for other sessions (Spec review, Repairs) or a general per-session Harness map.
    • A command word to override the security Harness.
    • Retrying a refused Security review on another Harness.
    • Security runs' own Harness choice, which already uses security.harness.
  2. added
    ready-for-agentFully specified, ready for an AFK agent
    in-progressA Claim: thirdshift has taken this issue
    and removed
    ready-for-agentFully specified, ready for an AFK agent
    on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions