diff --git a/README.md b/README.md index 4e4425e93..7f2941d81 100644 --- a/README.md +++ b/README.md @@ -727,6 +727,26 @@ A skip exits `0` and sends no email, even with Run notifications on, so the sche On a repository that also has a [Pickup line](#a-pickup-run-on-a-schedule), put `--plan-only` on the Weeding line instead: the Architect run publishes the Architect plan and marks it ready, and a Pickup run builds it as an ordinary Ready issue, under the [Claim limit](#the-claim-limit), and merges it with `merge.always`, as in the [whole crontab](#a-pickup-run-on-a-schedule) below. +## Security runs + +`thirdshift secure` starts a report-only **Security run**: it audits the **Base branch** at its head on `origin` and records each **Security finding** privately as a draft repository security advisory. Start it from a clone on the Base branch, or name that branch from any checkout: + +```sh +thirdshift secure +thirdshift secure base main harness claude +thirdshift secure base main harness codex model gpt-6.1-sol effort high +``` + +The command takes the Harness, Model and Effort words and the shared Pass words (`merge`, `no-merge`, `base-fix`, `no-base-fix`, `parallel`, `email` and `no-email`, with or without dashes). This first step dispatches no fix and sends no Run notification; those options are reserved for later Security run work. It reproduces and fixes nothing, and never commits, pushes, publishes or closes an advisory. + +A Security run is skipped while another **Pass** on the repository is running on the machine, or while the repository has a **Ready issue**. A skip exits `0`, starts no session and keeps no Command log; its reason goes into the repository's **Activity log**, with the usual `activity.quiet_skips` behavior. Before starting work, thirdshift checks the chosen Harness and that **Node.js** is on `PATH`, since the embedded skill's report validators require it. + +The Security audit runs in a throwaway worktree detached at origin's Base branch head, leaving the Launch directory and local work untouched, including when `launch.pull` is set. Its Session prompt runs `thirdshift-security-audit` in full audit mode with the `quick` profile, auditing the whole repository except vendored and third-party code. It names a `SECURITY.md` or conventional threat-model document when present, reads compatible earlier runs and ends `incomplete` instead of asking for input. + +Each audit keeps a new output directory under `///audits/`, outside the worktree. The audit artifacts and private advisory descriptions contain evidence; keep the logs directory private. thirdshift runs both embedded validators before recording anything. A missing final line, an incomplete session or an invalid report fails the run and names its Session log. Command logs are in `commands/secure/`; Session logs are named `secure--security-audit.jsonl`; the Activity log records the start and ending as for other Passes. + +A finding's title becomes the advisory summary. Its description preserves its write-up, trace, evidence and validation plan, with the fingerprint and audited commit. thirdshift claims no severity, CWE or affected version range. It uses the package in the repository's manifest, or the `other` ecosystem when none is identified. Matching fingerprints in any advisory state are kept rather than recorded again. Rejected candidates stay in the local report and produce no advisory. Draft advisories require GitHub repository security manager or administrator access. Private-repository fallback is later work; this command fails if the advisory endpoint is unavailable, without publishing a finding as an issue. + ## Pickup runs `thirdshift pickup` starts a **Pickup run**: one pass, with no **Issue URL**, that takes the lowest-numbered **Ready issue** in the repository and runs it, so that an issue you have already marked ready needs no command typed for it. Start it from a clone of the repository, on the **Base branch**, or name the Base branch with `base `: diff --git a/prompts/README.md b/prompts/README.md index 6b0f38373..02283939e 100644 --- a/prompts/README.md +++ b/prompts/README.md @@ -25,6 +25,7 @@ With `harness codex`, each session runs `codex exec --json --dangerously-bypass- | Continuation, with an open pull request | Starts the implement session when the Run is a Continuation of an Issue branch whose pull request is open. | [continuation-pr.md](continuation-pr.md) | | Spec review | In a Spec run, starts the Spec review once every Ticket has landed on the Spec branch, before the Spec PR, a draft until then, is marked ready. | [spec-review.md](spec-review.md) | | Architecture review | Starts the Architecture review, the session an Architect run opens with, in a worktree at the head of the Base branch. The line naming the focus is left out when the command gives none. | [architecture-review.md](architecture-review.md) | +| Security audit | Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. | [security-audit.md](security-audit.md) | | Conflict Repair | Starts a Repair session when merging the Base branch into the Issue branch leaves conflicts. | [conflict-repair.md](conflict-repair.md) | | Conflict Repair, on Foreign commits | In a Merge run, starts a Repair session when merging Foreign commits from the Issue branch on origin into the local one leaves conflicts. | [foreign-conflict-repair.md](foreign-conflict-repair.md) | | Review Repair | In a Merge run, starts a Repair session once Foreign commits are merged into the Issue branch, to review them from the head the Run last knew as its own before they can be merged. | [review-repair.md](review-repair.md) | diff --git a/prompts/security-audit.md b/prompts/security-audit.md new file mode 100644 index 000000000..7f66f074c --- /dev/null +++ b/prompts/security-audit.md @@ -0,0 +1,21 @@ + + +# Security audit + +Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root. + +``` +/thirdshift-security-audit +Use the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile. +Audit the whole repository at commit ``, the head of Base branch `` on origin. All vendored and third-party code are out of scope. +Read the repository's threat-model document `SECURITY.md`. +Output directory: ``. +Audit root: ``. +Read compatible earlier runs under the audit root before planning this audit. +This session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing. +If a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking. +Write the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing. +End your final message with exactly `Security audit: complete` after writing valid artifacts, or `Security audit: incomplete` when incomplete. + +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. +``` diff --git a/site/prompts/index.html b/site/prompts/index.html index 14ccd800e..b1d95e1c8 100644 --- a/site/prompts/index.html +++ b/site/prompts/index.html @@ -155,6 +155,25 @@

Architecture review

Architecture review idea: <URL of the issue you filed> Architecture review already filed: <URL of the open issue that already covers it> +You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on. + + +
+

Security audit

+

Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root.

+

Sent by a Security run, before any unit

+ 
/thirdshift-security-audit
+Use the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile.
+Audit the whole repository at commit `<audited commit>`, the head of Base branch `<base>` on origin. All vendored and third-party code are out of scope.
+Read the repository's threat-model document `SECURITY.md`.
+Output directory: `<output directory>`.
+Audit root: `<audit root>`.
+Read compatible earlier runs under the audit root before planning this audit.
+This session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing.
+If a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking.
+Write the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing.
+End your final message with exactly `Security audit: complete` after writing valid artifacts, or `Security audit: incomplete` when incomplete.
+
 You run headless: nobody is watching, and ending your turn ends the session. Run tests and other long commands in the foreground, raising the command's timeout if needed. If a command is moved to the background, wait for that task by its own task id or output file, never by process names or patterns (`pgrep`, `ps | grep`, and the like): other sessions on this machine run the same commands. Never end your turn while a background task you depend on is still running: ending the turn kills it. Before ending your turn, stop every background task you no longer need, by its task id (with the `TaskStop` tool, if you have it): a task still running when your turn ends is taken as work you were waiting on.
diff --git a/src/architect.rs b/src/architect.rs index 9ad4609c7..724f9161b 100644 --- a/src/architect.rs +++ b/src/architect.rs @@ -227,7 +227,7 @@ pub fn run( } = match launch::start(base) { Ok(Start::Clear(launch)) => launch, Ok(Start::AlreadyRunning(running)) => { - logs::skipped(Pass::ArchitectRun, &running.0, &running); + logs::skipped(Pass::Architect, &running.0, &running); return Outcome::Skipped(Skipped::AlreadyRunning(running)); } Err(error) => return failed(error.into()), @@ -285,7 +285,7 @@ fn run_through(outside: &mut impl Outside, architect_run: &ArchitectRun) -> Outc match gates(outside) { Ok(Decision::GoAhead) => {} Ok(Decision::Skip(skipped)) => { - outside.skipped(Pass::ArchitectRun, &skipped); + outside.skipped(Pass::Architect, &skipped); return Outcome::Skipped(skipped); } Err(error) => return failed(error.into()), diff --git a/src/args.rs b/src/args.rs index 79d4710b1..a3a4abd1b 100644 --- a/src/args.rs +++ b/src/args.rs @@ -24,7 +24,8 @@ pub enum Command { /// `email-test`, with the address it was given, if any. EmailTest(Option), Architect(ArchitectArgs), - Pickup(PickupArgs), + Pickup(PassArgs), + Secure(PassArgs), Run(RunArgs), } @@ -49,15 +50,15 @@ pub struct ArchitectArgs { pub plan_only: bool, } -/// A Pickup run's arguments. +/// A Pickup run or Security run's shared Pass arguments. #[derive(Debug, PartialEq, Eq)] -pub struct PickupArgs { +pub struct PassArgs { /// The Base branch `base ` named, if given; without it, the /// branch checked out in the Launch directory is the Base branch. pub base: Option, /// The flags it shares with a Run: `email` and `no-email` for its own - /// Run notification, the rest for the Spec run or Run the Ready issue is - /// dispatched as. + /// Run notification, the rest for a run the Pass dispatches. The + /// report-only Security run reserves them until those stages are built. pub flags: Flags, } @@ -77,7 +78,7 @@ pub struct RunArgs { } /// Parse the arguments after the program name. `help`, `version`, `update`, -/// `setup`, `email-test`, `architect` and `pickup` are commands only as the +/// `setup`, `email-test`, `architect`, `pickup` and `secure` are commands only as the /// first argument. Otherwise it is a Run: one Issue URL, with each Run flag at /// most once, before or after it. /// `email` may be followed by the address to send the Run notification to, @@ -105,7 +106,8 @@ pub fn parse(args: &[String]) -> Result { }; } Some("architect") => return parse_architect(&args[1..]).map(Command::Architect), - Some("pickup") => return parse_pickup(&args[1..]).map(Command::Pickup), + Some("pickup") => return parse_pass(&args[1..], "pickup").map(Command::Pickup), + Some("secure") => return parse_pass(&args[1..], "secure").map(Command::Secure), _ => {} } let mut issue = None; @@ -178,18 +180,18 @@ fn parse_architect(args: &[String]) -> Result { }) } -/// Parse the arguments after `pickup`: its flags, each at most once, in any +/// Parse the arguments after `pickup` or `secure`: shared flags, at most once, in any /// order, and nothing else. `base` must be followed by the Base branch. -fn parse_pickup(args: &[String]) -> Result { +fn parse_pass(args: &[String], command: &str) -> Result { let mut base = None; let mut flags = Flags::default(); let mut args = args.iter().peekable(); while let Some(arg) = args.next() { if !take_pass_flag(&mut base, &mut flags, arg, &mut args)? { - bail!("unexpected argument after pickup: {arg}"); + bail!("unexpected argument after {command}: {arg}"); } } - Ok(PickupArgs { base, flags }) + Ok(PassArgs { base, flags }) } /// Take a Pass's shared words: its Base branch and the flags for its @@ -704,7 +706,7 @@ mod tests { } /// The Pickup run `args` parse to. - fn pickup_args(args: &[&str]) -> PickupArgs { + fn pickup_args(args: &[&str]) -> PassArgs { match parse_strs(args) { Ok(Command::Pickup(pickup_args)) => pickup_args, Ok(_) => panic!("{args:?}: not a Pickup run"), @@ -714,12 +716,12 @@ mod tests { #[test] fn pickup_takes_each_of_its_flags_with_or_without_dashes_in_any_order() { - let none = PickupArgs { + let none = PassArgs { base: None, flags: Flags::default(), }; assert_eq!(pickup_args(&["pickup"]), none); - let all = PickupArgs { + let all = PassArgs { base: Some("develop".to_string()), flags: Flags { goal: Some(Goal::Merged), @@ -773,7 +775,7 @@ mod tests { ] { assert_eq!(pickup_args(&args), all, "{args:?}"); } - let cautious = PickupArgs { + let cautious = PassArgs { base: None, flags: Flags { goal: Some(Goal::ReadyForReview), diff --git a/src/command.rs b/src/command.rs index 7423a3b20..de924585e 100644 --- a/src/command.rs +++ b/src/command.rs @@ -40,6 +40,8 @@ pub enum Ending { review: Result, dispatched: Option, }, + /// A Security run completed its audit, or failed. + Security(Result), /// A Pass was skipped before any work. Skipped(Skip), } diff --git a/src/github.rs b/src/github.rs index e8af0e71b..cafd268ac 100644 --- a/src/github.rs +++ b/src/github.rs @@ -12,6 +12,9 @@ use crate::issue::IssueUrl; use crate::labels::{Label, Labels}; use crate::process::{self, Control, Interruption}; +mod advisories; +pub use advisories::{DraftAdvisory, Package}; + #[cfg(test)] mod execution_tests; @@ -614,9 +617,13 @@ impl GitHub { .with_context(|| format!("{command} returned invalid JSON")) } fn output(&self, args: &[&str]) -> Result { + self.output_with_input(args, None) + } + + fn output_with_input(&self, args: &[&str], input: Option<&[u8]>) -> Result { process::output( Command::new("gh").args(args), - None, + input, Control { name: "gh", interruption: self.interruption, diff --git a/src/github/advisories.rs b/src/github/advisories.rs new file mode 100644 index 000000000..8c5938a80 --- /dev/null +++ b/src/github/advisories.rs @@ -0,0 +1,73 @@ +//! Private advisory operations. Publishing and closing belong to the Day shift. + +use anyhow::{Context, Result, bail}; +use serde::Serialize; +use serde_json::{Value, json}; + +use super::GitHub; + +#[derive(Clone, Debug, Serialize)] +pub struct Package { + pub ecosystem: String, + pub name: Option, +} + +#[derive(Clone, Debug)] +pub struct DraftAdvisory { + pub fingerprint: String, + pub summary: String, + pub description: String, + pub package: Package, +} + +impl DraftAdvisory { + pub fn already_recorded(&self, advisories: &[Value]) -> bool { + let marker = format!("Fingerprint: `{}`", self.fingerprint); + advisories.iter().any(|advisory| { + advisory["description"] + .as_str() + .is_some_and(|description| description.lines().any(|line| line == marker)) + }) + } +} + +impl GitHub { + /// Every state and every page, including closed and published records. + pub fn security_advisories(&self, repo: &str) -> Result> { + self.gh_api_items( + &format!("repos/{repo}/security-advisories?per_page=100"), + "", + ) + } + + /// The create endpoint creates a draft. No severity or version claim. + pub fn create_security_advisory(&self, repo: &str, draft: &DraftAdvisory) -> Result { + let body = serde_json::to_vec(&json!({ + "summary": draft.summary, + "description": draft.description, + "severity": null, + "cwe_ids": [], + "vulnerabilities": [{"package": draft.package, "vulnerable_version_range": null}] + }))?; + let output = self.output_with_input( + &[ + "api", + "--method", + "POST", + &format!("repos/{repo}/security-advisories"), + "--input", + "-", + ], + Some(&body), + )?; + if !output.status.success() { + // API errors can echo private evidence, so do not relay them. + bail!( + "creating a draft repository security advisory failed ({})", + output.status + ); + } + serde_json::from_slice(&output.stdout) + .context("creating a draft advisory returned invalid JSON") + } +} diff --git a/src/logs.rs b/src/logs.rs index c02b0d4f3..9b541922c 100644 --- a/src/logs.rs +++ b/src/logs.rs @@ -40,6 +40,7 @@ pub enum Begin<'a> { ArchitectRun, /// `thirdshift pickup`, a pass. PickupRun, + SecurityRun, } /// The work a command starts. @@ -53,20 +54,24 @@ pub enum Work<'a> { PickupRun(&'a IssueUrl), /// An Architect run, on its repository. ArchitectRun(&'a Repo), + /// A Security run, on its repository. + SecurityRun(&'a Repo), } /// A Pass, a command that may be skipped before it starts work. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Pass { - ArchitectRun, - PickupRun, + Architect, + Pickup, + Security, } impl Display for Pass { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { f.write_str(match self { - Pass::ArchitectRun => "Architect run", - Pass::PickupRun => "Pickup run", + Pass::Architect => "Architect run", + Pass::Pickup => "Pickup run", + Pass::Security => "Security run", }) } } @@ -170,7 +175,7 @@ impl Work<'_> { fn repo(self) -> Repo { match self { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => issue.repo(), - Work::ArchitectRun(repo) => repo.clone(), + Work::ArchitectRun(repo) | Work::SecurityRun(repo) => repo.clone(), } } @@ -179,8 +184,9 @@ impl Work<'_> { match self { Work::Run(_) => Kind::Run, Work::SpecRun(_) => Kind::SpecRun, - Work::PickupRun(_) => Kind::Pass(Pass::PickupRun), - Work::ArchitectRun(_) => Kind::Pass(Pass::ArchitectRun), + Work::PickupRun(_) => Kind::Pass(Pass::Pickup), + Work::ArchitectRun(_) => Kind::Pass(Pass::Architect), + Work::SecurityRun(_) => Kind::Pass(Pass::Security), } } @@ -188,7 +194,7 @@ impl Work<'_> { fn issue(self) -> Option { match self { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => Some(issue.number), - Work::ArchitectRun(_) => None, + Work::ArchitectRun(_) | Work::SecurityRun(_) => None, } } @@ -204,6 +210,7 @@ impl Work<'_> { Work::Run(issue) | Work::SpecRun(issue) => ("issue", format!("{}-", issue.number)), Work::PickupRun(issue) => ("pickup", format!("{}-", issue.number)), Work::ArchitectRun(_) => ("architect", String::new()), + Work::SecurityRun(_) => ("secure", String::new()), }; root.join("commands") .join(folder) diff --git a/src/logs/recording.rs b/src/logs/recording.rs index 9053021de..4adeb294a 100644 --- a/src/logs/recording.rs +++ b/src/logs/recording.rs @@ -83,6 +83,7 @@ impl Record { Begin::Run(issue) => (format!("starting on {}", issue.url), false), Begin::ArchitectRun => ("Architect run starting".into(), true), Begin::PickupRun => ("Pickup run starting".into(), true), + Begin::SecurityRun => ("Security run starting".into(), true), }; let now = self.outside.now(); self.stamp diff --git a/src/logs/recording/tests.rs b/src/logs/recording/tests.rs index 1949060d6..f0d90543a 100644 --- a/src/logs/recording/tests.rs +++ b/src/logs/recording/tests.rs @@ -431,8 +431,9 @@ fn output_and_start_without_a_factory_beginning_keep_no_command_log() { fn skip(memory: &Scripted, pass: Pass, reason: &str) { let mut record = Record::new(memory.clone()); record.begin(match pass { - Pass::PickupRun => Begin::PickupRun, - Pass::ArchitectRun => Begin::ArchitectRun, + Pass::Pickup => Begin::PickupRun, + Pass::Architect => Begin::ArchitectRun, + Pass::Security => Begin::SecurityRun, }); record.configured(&config(true)); record.eprint(reason); @@ -445,28 +446,28 @@ fn skip(memory: &Scripted, pass: Pass, reason: &str) { #[test] fn separate_recordings_suppress_skips_from_shared_history_until_their_kind_changes() { let memory = Scripted::default(); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::ArchitectRun, "a plan is open"); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Architect, "a plan is open"); + skip(&memory, Pass::Pickup, "idle"); // A Run's lines do not reset the Pickup run's skip. let mut run = Record::new(memory.clone()); run.begin(Begin::Run(&issue())); run.configured(&config(false)); run.started(Work::Run(&issue()), &harness()); run.ended("ready"); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::PickupRun, "at the Claim limit"); - skip(&memory, Pass::PickupRun, "at the Claim limit"); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Pickup, "at the Claim limit"); + skip(&memory, Pass::Pickup, "at the Claim limit"); + skip(&memory, Pass::Pickup, "idle"); let mut pickup = Record::new(memory.clone()); pickup.begin(Begin::PickupRun); pickup.configured(&config(true)); pickup.started(Work::PickupRun(&issue()), &harness()); // Even just an intervening start permits the same skip again. - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); pickup.ended("ready"); - skip(&memory, Pass::PickupRun, "idle"); - skip(&memory, Pass::ArchitectRun, "a plan is open"); + skip(&memory, Pass::Pickup, "idle"); + skip(&memory, Pass::Architect, "a plan is open"); assert_eq!( memory.file(ACTIVITY), "2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Architect run skipped: a plan is open\n2026-10-03 12:00:00 Run #7 started: commands/issue/7-20261003T120000-0400.log, on claude · opus · high\n2026-10-03 12:00:00 Run #7 ended: ready\n2026-10-03 12:00:00 Pickup run skipped: at the Claim limit\n2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Pickup run #7 started: commands/pickup/7-20261003T120000-0400.log, on claude · opus · high\n2026-10-03 12:00:00 Pickup run skipped: idle\n2026-10-03 12:00:00 Pickup run #7 ended: ready\n2026-10-03 12:00:00 Pickup run skipped: idle\n" @@ -487,14 +488,14 @@ fn skip_history_uses_the_last_64_kib_and_discards_a_cut_first_line() { .unwrap() .files .insert(PathBuf::from(ACTIVITY), seed.clone()); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); let expected = if prefix.is_empty() { seed } else { format!("{seed}{line}") }; assert_eq!(memory.file(ACTIVITY), expected); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); assert_eq!(memory.file(ACTIVITY), expected); } let memory = Scripted::default(); @@ -505,7 +506,7 @@ fn skip_history_uses_the_last_64_kib_and_discards_a_cut_first_line() { .unwrap() .files .insert(PathBuf::from(ACTIVITY), seed.into()); - skip(&memory, Pass::PickupRun, "idle"); + skip(&memory, Pass::Pickup, "idle"); assert_eq!(memory.file(ACTIVITY), format!("{seed}{line}")); } @@ -642,8 +643,8 @@ fn activity_read_and_append_warnings_reveal_quiet_output_and_reset_in_fresh_inst for read_failure in [false, true] { let memory = Scripted::default(); for (begin, pass, kind) in [ - (Begin::PickupRun, Pass::PickupRun, "Pickup run"), - (Begin::ArchitectRun, Pass::ArchitectRun, "Architect run"), + (Begin::PickupRun, Pass::Pickup, "Pickup run"), + (Begin::ArchitectRun, Pass::Architect, "Architect run"), ] { { let mut faults = memory.0.lock().unwrap(); diff --git a/src/main.rs b/src/main.rs index 0d25dcc55..eb20a2c0f 100644 --- a/src/main.rs +++ b/src/main.rs @@ -34,6 +34,7 @@ mod ready; mod resend_key; mod run; mod run_ending; +mod security; mod session; mod setup; mod skills; @@ -46,7 +47,7 @@ mod worktree; use std::process::ExitCode; use architect::Outcome; -use args::{ArchitectArgs, Command, PickupArgs, RunArgs}; +use args::{ArchitectArgs, Command, PassArgs, RunArgs}; use asks::Asks; use command::{Ending, failure}; use config::UserConfig; @@ -67,6 +68,8 @@ usage: thirdshift Run the factory on the iss thirdshift architect base [] Do either with as the Base branch, from a clone on any branch thirdshift pickup Take the lowest-numbered Ready issue in the repository and run it thirdshift pickup base Do that with as the Base branch, from a clone on any branch + thirdshift secure Audit the Base branch and record findings privately + thirdshift secure base Do that with as the Base branch thirdshift email-test [
] Send a test email through Resend, to check the email setup thirdshift setup Choose your defaults, then write the User config with every setting thirdshift update Update thirdshift to the latest release @@ -391,6 +394,7 @@ fn main() -> ExitCode { } Ok(Command::Architect(architect_args)) => return architect(architect_args), Ok(Command::Pickup(pickup_args)) => return pickup(pickup_args), + Ok(Command::Secure(args)) => return secure(args), Ok(Command::Run(run_args)) => run_args, Err(error) => return argument_error(format_args!("{error:#}")), }; @@ -476,7 +480,7 @@ fn architect(args: ArchitectArgs) -> ExitCode { /// The repository's Activity log records the skip, if it differs from the /// last, or the start and end of its work. With `activity.quiet_skips` set, a /// skipped pass prints nothing at all. -fn pickup(args: PickupArgs) -> ExitCode { +fn pickup(args: PassArgs) -> ExitCode { let ask = |config: &UserConfig| args.flags.notification(config); let (config, mut started) = match command::start(Begin::PickupRun, ask, About::PickupRun) { Ok(started) => started, @@ -499,6 +503,27 @@ fn pickup(args: PickupArgs) -> ExitCode { } } +/// A report-only Security run. Notification delivery is added by #539. +fn secure(args: PassArgs) -> ExitCode { + let (config, started) = match command::start( + Begin::SecurityRun, + |_| notification::NotificationAsk::Skip, + About::SecurityRun, + ) { + Ok(started) => started, + Err(failure) => return failure, + }; + let mut harness = args.flags.harness(&config); + let outcome = security::run(args.base.as_deref(), &args.flags, &config, &mut harness); + started.finish(match outcome { + security::Outcome::Skipped(skipped) => Ending::Skipped(command::Skip { + reason: skipped.to_string(), + urls: Vec::new(), + }), + security::Outcome::Audited(audited) => Ending::Security(audited), + }) +} + /// The end of a command other than a Run: the line that says how it went, /// or its error, on stderr. fn outcome(result: anyhow::Result) -> ExitCode { diff --git a/src/notification.rs b/src/notification.rs index 71a9baab6..a090ca5c8 100644 --- a/src/notification.rs +++ b/src/notification.rs @@ -55,6 +55,8 @@ pub enum About<'a> { ArchitectRun, /// A Pickup run, which is about the issue it takes, once it takes one. PickupRun, + /// A report-only Security run; its notification is added by #539. + SecurityRun, } /// What a Run notification's subject names. @@ -69,7 +71,7 @@ enum Subject { /// An Architect run, with the repository the Launch directory's `origin` /// names, if it names one. ArchitectRun(Option), - /// A Pickup run that has taken no issue yet, and so has nothing to tell. + /// A Pass whose notification subject has not been established yet. Pending, } @@ -102,6 +104,7 @@ impl RunNotification { // run's own preflight reports why. About::ArchitectRun => Subject::ArchitectRun(launch::repo().ok()), About::PickupRun => Subject::Pending, + About::SecurityRun => Subject::Pending, }; Ok(RunNotification { checked, diff --git a/src/pass.rs b/src/pass.rs index 7083d0c9a..d3495355f 100644 --- a/src/pass.rs +++ b/src/pass.rs @@ -14,7 +14,9 @@ use std::fmt::Display; use std::path::PathBuf; +use crate::github::DraftAdvisory; use anyhow::Result; +use serde_json::Value; use crate::asks::{Asks, Flags}; use crate::config::UserConfig; @@ -94,6 +96,14 @@ pub trait Outside { prompt: &str, conclude: impl FnOnce(&mut Self, Option<&str>) -> Result, ) -> (Result, Option); + /// Check Node.js before a Security audit needs its validators. + fn check_node(&mut self) -> Result<()>; + /// Audit the Base branch in a disposable checkout and validate its artifacts. + fn audit(&mut self, base: &str) -> (Result, Option); + /// Private records in every advisory state. + fn security_advisories(&mut self) -> Result>; + /// Record one finding privately as a draft. + fn create_security_advisory(&mut self, finding: &DraftAdvisory) -> Result; /// Run `dispatch` to its end, on the Harness the pass checked. fn dispatch(&mut self, dispatch: Dispatch) -> Ended; } @@ -183,6 +193,22 @@ impl Outside for LaunchAndGitHub<'_> { concluded } + fn check_node(&mut self) -> Result<()> { + crate::security::audit::check_node() + } + + fn audit(&mut self, base: &str) -> (Result, Option) { + crate::security::audit::run(self.launch.git(), self.repo, base, self.harness) + } + + fn security_advisories(&mut self) -> Result> { + GitHub::new().security_advisories(&self.repo.slug()) + } + + fn create_security_advisory(&mut self, finding: &DraftAdvisory) -> Result { + GitHub::new().create_security_advisory(&self.repo.slug(), finding) + } + /// The run's asks are the Architect plan's or the Ready issue's, from /// the command's flags and the User config, on the checked Harness. fn dispatch(&mut self, dispatch: Dispatch) -> Ended { @@ -221,12 +247,14 @@ mod in_memory { use anyhow::{Result, anyhow, bail}; use super::{Dispatch, Outside}; + use crate::github::DraftAdvisory; use crate::github::{Issue, ListedIssue}; use crate::issue::{IssueUrl, Repo}; use crate::labels::{Edit, Label, Labels}; use crate::logs::{Pass, Work}; use crate::ready::ReadyIssue; use crate::run::{Ended, Goal, Reached}; + use serde_json::{Value, json}; /// A call a pass made outside itself, in the order it made it. #[derive(Debug, PartialEq, Eq)] @@ -253,6 +281,14 @@ mod in_memory { Skipped(String), /// It checked its Harness, Model and Effort. HarnessCheck, + /// It checked Node.js for the Security audit validators. + NodeCheck, + /// It ran and validated the Security audit of this Base branch. + SecurityAudit(String), + /// It listed private advisories in every state. + AdvisoryList, + /// It recorded this fingerprint privately. + CreateAdvisory(String), /// It recorded that it started work: on this issue, for a Pickup /// run, or on its repository, for an Architect run. Started(Option), @@ -293,6 +329,10 @@ mod in_memory { ready: Option<(ListedIssue, bool)>, /// Whether the Harness check fails. harness_failing: bool, + node_failing: bool, + audit_findings: Vec, + audit_error: Option, + advisories: Vec, /// The Architecture review session's final message, if it has one, /// or the cause it fails with. review: Result, String>, @@ -317,6 +357,10 @@ mod in_memory { failing_edits: Vec::new(), ready: None, harness_failing: false, + node_failing: false, + audit_findings: Vec::new(), + audit_error: None, + advisories: Vec::new(), review: Ok(None), session_log: None, ending: None, @@ -391,6 +435,26 @@ mod in_memory { self } + pub fn node_failing(mut self) -> Self { + self.node_failing = true; + self + } + + pub fn audited(mut self, findings: Vec) -> Self { + self.audit_findings = findings; + self + } + + pub fn audit_failing(mut self, cause: &str) -> Self { + self.audit_error = Some(cause.to_string()); + self + } + + pub fn advisories(mut self, advisories: Vec) -> Self { + self.advisories = advisories; + self + } + /// Have the Architecture review session end with `final_message`. pub fn reviewed(mut self, final_message: &str) -> Self { self.review = Ok(Some(final_message.to_string())); @@ -502,7 +566,7 @@ mod in_memory { Work::Run(issue) | Work::SpecRun(issue) | Work::PickupRun(issue) => { Some(issue.number) } - Work::ArchitectRun(_) => None, + Work::ArchitectRun(_) | Work::SecurityRun(_) => None, }; self.calls.push(Call::Started(issue)); } @@ -530,6 +594,41 @@ mod in_memory { } } + fn check_node(&mut self) -> Result<()> { + self.calls.push(Call::NodeCheck); + if self.node_failing { + bail!("Node.js is required"); + } + Ok(()) + } + + fn audit( + &mut self, + base: &str, + ) -> (Result, Option) { + self.calls.push(Call::SecurityAudit(base.to_string())); + let audited = match self.audit_error.take() { + Some(cause) => Err(anyhow!(cause)), + None => Ok(crate::security::audit::Audited { + findings: std::mem::take(&mut self.audit_findings), + }), + }; + (audited, self.session_log.clone()) + } + + fn security_advisories(&mut self) -> Result> { + self.calls.push(Call::AdvisoryList); + Ok(self.advisories.clone()) + } + + fn create_security_advisory(&mut self, finding: &DraftAdvisory) -> Result { + self.calls + .push(Call::CreateAdvisory(finding.fingerprint.clone())); + let advisory = json!({"description": finding.description, "state": "draft"}); + self.advisories.push(advisory.clone()); + Ok(advisory) + } + fn dispatch(&mut self, dispatch: Dispatch) -> Ended { self.calls.push(match dispatch { Dispatch::ArchitectPlan { plan, base } => Call::DispatchPlan { diff --git a/src/pickup.rs b/src/pickup.rs index ee98fb439..1e244aacc 100644 --- a/src/pickup.rs +++ b/src/pickup.rs @@ -111,7 +111,7 @@ pub fn run( } = match launch::start(base)? { Start::Clear(launch) => launch, Start::AlreadyRunning(running) => { - logs::skipped(Pass::PickupRun, &running.0, &running); + logs::skipped(Pass::Pickup, &running.0, &running); return Ok(Outcome::Skipped(Skipped::AlreadyRunning(running))); } }; @@ -142,7 +142,7 @@ fn run_through( let ReadyIssue { listed, is_spec } = match gates(outside, repo, limit)? { Decision::Take(ready) => ready, Decision::Skip(skipped) => { - outside.skipped(Pass::PickupRun, &skipped); + outside.skipped(Pass::Pickup, &skipped); return Ok(Outcome::Skipped(skipped)); } }; diff --git a/src/prompt.rs b/src/prompt.rs index b5f8e71fe..9d8162d42 100644 --- a/src/prompt.rs +++ b/src/prompt.rs @@ -292,3 +292,24 @@ pub fn resume(killed: &[&str]) -> String { killed = killed.join("; "), ) } + +/// The Security audit's terminal protocol: only these final lines are read. +pub const AUDIT_COMPLETE_LINE: &str = "Security audit: complete"; +pub const AUDIT_INCOMPLETE_LINE: &str = "Security audit: incomplete"; + +pub fn security_audit( + base: &str, + commit: &str, + root: &std::path::Path, + output: &std::path::Path, + threat_model: Option<&str>, +) -> String { + let threat_model = threat_model + .map(|file| format!("Read the repository's threat-model document `{file}`.\n")) + .unwrap_or_default(); + format!( + "/thirdshift-security-audit\nUse the `thirdshift-security-audit` skill in full audit mode with report artifacts and the `quick` profile.\nAudit the whole repository at commit `{commit}`, the head of Base branch `{base}` on origin. All vendored and third-party code are out of scope.\n{threat_model}Output directory: `{output}`.\nAudit root: `{root}`.\nRead compatible earlier runs under the audit root before planning this audit.\nThis session commits, pushes, opens and publishes nothing, changes no repository source, and touches no deployed site. It reproduces and fixes nothing.\nIf a decision or missing prerequisite would require asking, record run_status incomplete with the reason, and end incomplete rather than asking.\nWrite the skill's report artifacts and run-metadata.json; mark run_status complete only when all required artifacts are written. Run both skill validators before finishing.\nEnd your final message with exactly `{AUDIT_COMPLETE_LINE}` after writing valid artifacts, or `{AUDIT_INCOMPLETE_LINE}` when incomplete.\n\n{HEADLESS}", + output = output.display(), + root = root.display() + ) +} diff --git a/src/prompts_page.rs b/src/prompts_page.rs index f5c2e8a7a..60935ff1c 100644 --- a/src/prompts_page.rs +++ b/src/prompts_page.rs @@ -19,7 +19,7 @@ use crate::prompt; use crate::session::{claude_args, codex_args}; use crate::skills::SKILLS; -use Sender::{ArchitectRun, Units}; +use Sender::{ArchitectRun, SecurityRun, Units}; /// A press unit on the home page, which each prompt and skill links back to. struct Unit { @@ -87,6 +87,8 @@ enum Sender { Units(&'static [Unit]), /// An Architect run, which the press units, a Run's, don't cover. ArchitectRun, + /// A Security run, before any dispatched Run. + SecurityRun, } /// A prompt as the page shows it: when it is sent, by whom, and its text @@ -162,6 +164,19 @@ fn prompts() -> Vec { sender: ArchitectRun, text: prompt::architecture_review(BASE, Some(FOCUS)), }, + Prompt { + id: "prompt-security-audit", + title: "Security audit", + when: "Starts the report-only Security audit in a throwaway worktree at origin's Base branch head. The threat-model line is included when a conventional document exists; artifacts stay under the repository's audit root.", + sender: SecurityRun, + text: prompt::security_audit( + BASE, + "", + std::path::Path::new(""), + std::path::Path::new(""), + Some("SECURITY.md"), + ), + }, Prompt { id: "prompt-conflict-repair", title: "Conflict Repair", @@ -532,6 +547,7 @@ fn sent_by(sender: &Sender) -> String { match sender { Units(units) => unit_links(units), ArchitectRun => "an Architect run, before any unit".to_string(), + SecurityRun => "a Security run, before any unit".to_string(), } } diff --git a/src/run.rs b/src/run.rs index 1b5e94866..a2dce0984 100644 --- a/src/run.rs +++ b/src/run.rs @@ -560,7 +560,9 @@ mod tests { let spec_run = match work { Work::Run(_) => false, Work::SpecRun(_) => true, - Work::PickupRun(_) | Work::ArchitectRun(_) => panic!("not a Run's work"), + Work::PickupRun(_) | Work::ArchitectRun(_) | Work::SecurityRun(_) => { + panic!("not a Run's work") + } }; self.calls.push(Call::Started { spec_run }); } diff --git a/src/run_ending.rs b/src/run_ending.rs index efc5974f0..97dfb0f2b 100644 --- a/src/run_ending.rs +++ b/src/run_ending.rs @@ -36,6 +36,21 @@ pub fn read(ending: &Ending) -> Result, &Skip> { Ending::Architect { review, dispatched } => { Ok(Account::of_architect(review, dispatched.as_ref())) } + Ending::Security(audited) => Ok(match audited { + Ok(recorded) => Account { + outcome: "findings recorded", + ended: Ok(recorded.to_string()), + interrupted: false, + pr_url: None, + advice: &[], + base_fix: None, + log: None, + ticket_lines: &[], + review: None, + urls: Vec::new(), + }, + Err(failed) => Account::of_failure(failed, "audit failed"), + }), Ending::Skipped(skip) => Err(skip), } } diff --git a/src/security.rs b/src/security.rs new file mode 100644 index 000000000..baf06c8f0 --- /dev/null +++ b/src/security.rs @@ -0,0 +1,242 @@ +//! Report-only Security runs: gate before work, audit origin's Base branch, +//! then record the validated findings privately for the Day shift. + +use std::fmt; + +use anyhow::Result; + +use crate::asks::Flags; +use crate::config::UserConfig; +use crate::failed_run::FailedRun; +use crate::github::ListedIssue; +use crate::harness::Choice; +use crate::issue::Repo; +use crate::launch::{self, AlreadyRunning, Launch, Start}; +use crate::logs::{self, Pass, Work}; +use crate::pass::{LaunchAndGitHub, Outside}; + +pub mod audit; + +pub enum Outcome { + Skipped(Skipped), + Audited(Result), +} + +pub enum Skipped { + AlreadyRunning(AlreadyRunning), + ReadyIssue(ListedIssue), +} + +impl fmt::Display for Skipped { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + match self { + Self::AlreadyRunning(running) => running.fmt(f), + Self::ReadyIssue(listed) => write!( + f, + "Ready issue #{} \"{}\" goes first: {}", + listed.issue.number, listed.title, listed.issue.url + ), + } + } +} + +pub struct Recorded { + pub created: usize, + pub existing: usize, +} + +impl fmt::Display for Recorded { + fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { + write!( + f, + "Security audit recorded {} new finding(s), {} already recorded", + self.created, self.existing + ) + } +} + +pub fn run( + base: Option<&str>, + flags: &Flags, + config: &UserConfig, + harness: &mut Choice, +) -> Outcome { + let Launch { + directory, + repo, + base, + } = match launch::start(base) { + Ok(Start::Clear(launch)) => launch, + Ok(Start::AlreadyRunning(running)) => { + logs::skipped(Pass::Security, &running.0, &running); + return Outcome::Skipped(Skipped::AlreadyRunning(running)); + } + Err(error) => return Outcome::Audited(Err(error.into())), + }; + run_through( + &mut LaunchAndGitHub { + launch: &directory, + base: &base, + repo: &repo, + harness, + flags, + config, + }, + &repo, + base.name(), + ) +} + +fn run_through(outside: &mut impl Outside, repo: &Repo, base: &str) -> Outcome { + match outside.ready_issue() { + Ok(Some(ready)) => { + let skipped = Skipped::ReadyIssue(ready.listed); + outside.skipped(Pass::Security, &skipped); + return Outcome::Skipped(skipped); + } + Ok(None) => {} + Err(error) => return Outcome::Audited(Err(error.into())), + } + Outcome::Audited(audit_and_record(outside, repo, base)) +} + +fn audit_and_record( + outside: &mut impl Outside, + repo: &Repo, + base: &str, +) -> Result { + outside.check_harness()?; + outside.check_node()?; + outside.started(Work::SecurityRun(repo)); + let (audited, log) = outside.audit(base); + let recorded = (|| -> Result { + let audited = audited?; + let mut recorded = Recorded { + created: 0, + existing: 0, + }; + if audited.findings.is_empty() { + return Ok(recorded); + } + let mut known = outside.security_advisories()?; + for finding in audited.findings { + if finding.already_recorded(&known) { + recorded.existing += 1; + } else { + known.push(outside.create_security_advisory(&finding)?); + recorded.created += 1; + outside.step("recorded a Security finding as a private draft advisory".to_string()); + } + } + Ok(recorded) + })(); + recorded.map_err(|error| FailedRun { + log, + ..error.into() + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::pass::{Call, InMemory, widgets}; + + #[test] + fn checks_prerequisites_then_audits_without_pulling() { + let mut outside = InMemory::default(); + let outcome = run_through(&mut outside, &widgets(), "main"); + assert!(matches!(outcome, Outcome::Audited(Ok(_)))); + assert_eq!( + outside.calls, + vec![ + Call::ReadySearch, + Call::HarnessCheck, + Call::NodeCheck, + Call::Started(None), + Call::SecurityAudit("main".to_string()) + ] + ); + } + #[test] + fn a_ready_issue_precedes_every_prerequisite_and_audit() { + let mut outside = InMemory::default() + .ready(7, false) + .node_failing() + .harness_failing(); + assert!(matches!( + run_through(&mut outside, &widgets(), "main"), + Outcome::Skipped(Skipped::ReadyIssue(_)) + )); + assert!(matches!( + &outside.calls[..], + [Call::ReadySearch, Call::Skipped(_)] + )); + } + + #[test] + fn each_failed_prerequisite_prevents_later_operations() { + for (mut outside, expected) in [ + ( + InMemory::default().harness_failing(), + vec![Call::ReadySearch, Call::HarnessCheck], + ), + ( + InMemory::default().node_failing(), + vec![Call::ReadySearch, Call::HarnessCheck, Call::NodeCheck], + ), + ] { + assert!(matches!( + run_through(&mut outside, &widgets(), "main"), + Outcome::Audited(Err(_)) + )); + assert_eq!(outside.calls, expected); + } + } + + #[test] + fn a_failed_audit_records_nothing_and_keeps_its_session_log() { + let mut outside = InMemory::default() + .audit_failing("invalid report") + .session_log("audit.jsonl"); + let Outcome::Audited(Err(failed)) = run_through(&mut outside, &widgets(), "main") else { + panic!("audit should fail"); + }; + assert_eq!(failed.log, Some(std::path::PathBuf::from("audit.jsonl"))); + assert!(!outside.calls.contains(&Call::AdvisoryList)); + } + + #[test] + fn only_new_fingerprints_get_a_private_record() { + use crate::github::{DraftAdvisory, Package}; + use serde_json::json; + let draft = |fingerprint: &str| DraftAdvisory { + fingerprint: fingerprint.into(), + summary: "Candidate".into(), + description: format!("Fingerprint: `{fingerprint}`"), + package: Package { + ecosystem: "other".into(), + name: None, + }, + }; + let mut outside = InMemory::default() + .audited(vec![draft("old"), draft("new")]) + .advisories(vec![ + json!({"state":"closed", "description":"Fingerprint: `old`"}), + ]); + let Outcome::Audited(Ok(recorded)) = run_through(&mut outside, &widgets(), "main") else { + panic!("audit should succeed"); + }; + assert_eq!((recorded.created, recorded.existing), (1, 1)); + assert_eq!( + outside + .calls + .iter() + .filter_map(|call| match call { + Call::CreateAdvisory(fingerprint) => Some(fingerprint.as_str()), + _ => None, + }) + .collect::>(), + vec!["new"] + ); + } +} diff --git a/src/security/audit.rs b/src/security/audit.rs new file mode 100644 index 000000000..7a8d4421e --- /dev/null +++ b/src/security/audit.rs @@ -0,0 +1,207 @@ +//! The Security audit session and its validated report artifacts. Kept under +//! the repository's logs, independently of the disposable audit checkout. + +use std::fs; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use anyhow::{Context, Result, bail}; +use serde_json::Value; + +use crate::git::Git; +use crate::github::{DraftAdvisory, Package}; +use crate::harness::Choice; +use crate::issue::Repo; +use crate::logs; +use crate::process::{self, Control, Interruption}; +use crate::progress; +use crate::prompt; +use crate::session::{Logs, Sessions}; +use crate::skills; +use crate::worktree::ReviewWorktree; + +pub struct Audited { + pub findings: Vec, +} + +pub fn check_node() -> Result<()> { + node(&["--version"]) + .context("Node.js is required for the Security audit validators; install node on PATH")?; + Ok(()) +} + +pub fn run( + launch: &Git, + repo: &Repo, + base: &str, + harness: &Choice, +) -> (Result, Option) { + let prepared = (|| -> Result<_> { + let worktree = ReviewWorktree::create(launch, &repo.name, base)?; + let commit = Git::new(worktree.path()).run(&["rev-parse", "HEAD"])?; + let root = logs::root(repo).join("audits"); + fs::create_dir_all(&root) + .with_context(|| format!("could not create {}", root.display()))?; + let output = tempfile::Builder::new() + .prefix(&format!("{}-", logs::stamp())) + .tempdir_in(&root)? + .keep(); + // Case-insensitive filesystems can resolve a candidate with different + // casing. Name the document as it is tracked in the audited checkout. + let tracked = Git::new(worktree.path()).run(&["ls-files", "-z"])?; + let threat_model = [ + "SECURITY.md", + "THREAT_MODEL.md", + "THREAT-MODEL.md", + "docs/threat-model.md", + "docs/THREAT-MODEL.md", + "docs/THREAT_MODEL.md", + ] + .into_iter() + .find(|file| { + tracked.split('\0').any(|tracked| tracked == *file) + && worktree.path().join(file).is_file() + }); + let prompt = prompt::security_audit(base, &commit, &root, &output, threat_model); + Ok((worktree, output, prompt, commit)) + })(); + let (worktree, output, prompt, commit) = match prepared { + Ok(prepared) => prepared, + Err(error) => return (Err(error), None), + }; + let logs = Logs::of_security_run(repo); + Sessions::within(&logs, worktree.path(), harness, |sessions| { + progress::step(format!("starting the Security audit of {base}")); + let message = sessions.run_to_final_message("security-audit", &prompt)?; + let line = message.as_deref().and_then(|message| { + message + .lines() + .map(str::trim) + .rfind(|line| !line.is_empty()) + }); + match line { + Some(prompt::AUDIT_COMPLETE_LINE) => {} + Some(prompt::AUDIT_INCOMPLETE_LINE) => bail!( + "the Security audit ended incomplete; artifacts: {}", + output.display() + ), + _ => bail!("the Security audit ended without the final line its prompt asks for"), + } + let metadata: Value = serde_json::from_slice( + &fs::read(output.join("run-metadata.json")) + .context("could not read the Security audit's run-metadata.json")?, + ) + .context("the Security audit's run-metadata.json is invalid JSON")?; + if metadata["run_status"] != "complete" { + bail!( + "the Security audit's run-metadata.json does not mark the audit complete; artifacts: {}", + output.display() + ); + } + let skill = skills::written_out()?.join("thirdshift-security-audit"); + for (validator, report) in [ + ("validate-findings.cjs", "findings.json"), + ("validate-coverage-ledger.cjs", "coverage-ledger.json"), + ] { + node(&[ + skill + .join(validator) + .to_str() + .context("non-UTF-8 skill path")?, + output + .join(report) + .to_str() + .context("non-UTF-8 report path")?, + ]) + .with_context(|| { + format!("the Security audit report failed the skill's validator {validator}") + })?; + } + let findings: Vec = + serde_json::from_slice(&fs::read(output.join("findings.json"))?)?; + let package = package_in(worktree.path()); + let findings = findings.into_iter().filter(|finding| finding["verdict"] != "rejected").map(|finding| { + let fingerprint = finding["fingerprint"].as_str().context("finding has no fingerprint")?.to_string(); + Ok(DraftAdvisory { + summary: finding["title"].as_str().context("finding has no title")?.to_string(), + description: format!("Found by thirdshift's Security run.\n\nFingerprint: `{fingerprint}`\nAudited commit: `{commit}`\n\n{}\n\n```json\n{}\n```\n", finding["description"].as_str().unwrap_or_default(), serde_json::to_string_pretty(&finding)?), + fingerprint, + package: package.clone(), + }) + }).collect::>()?; + Ok(Audited { findings }) + }) +} + +fn node(args: &[&str]) -> Result<()> { + let output = process::output( + Command::new("node").args(args), + None, + Control { + name: "Node.js", + interruption: Interruption::Ordinary, + stop: &|child| process::stop(child, &[libc::SIGTERM]), + }, + )?; + if !output.status.success() { + // The validators' diagnostics can contain private evidence. Leave + // details in the audit artifacts, out of progress and Command logs. + bail!("Node.js exited {}", output.status); + } + Ok(()) +} + +/// Manifest identity only: no affected or patched versions are inferred. +fn package_in(worktree: &Path) -> Package { + for (file, section, ecosystem) in [ + ("Cargo.toml", "package", "rust"), + ("pyproject.toml", "project", "pip"), + ] { + let name = fs::read_to_string(worktree.join(file)) + .ok() + .and_then(|text| toml::from_str::(&text).ok()) + .and_then(|manifest| { + manifest + .get(section)? + .get("name")? + .as_str() + .map(String::from) + }); + if let Some(name) = name { + return Package { + ecosystem: ecosystem.to_string(), + name: Some(name), + }; + } + } + let npm = fs::read(worktree.join("package.json")) + .ok() + .and_then(|text| serde_json::from_slice::(&text).ok()) + .and_then(|manifest| manifest["name"].as_str().map(String::from)); + if let Some(name) = npm { + return Package { + ecosystem: "npm".to_string(), + name: Some(name), + }; + } + let go = fs::read_to_string(worktree.join("go.mod")) + .ok() + .and_then(|text| { + text.lines().find_map(|line| { + let mut words = line.split_whitespace(); + (words.next()? == "module") + .then(|| words.next().map(|name| name.trim_matches('"').to_string())) + .flatten() + }) + }); + match go { + Some(name) => Package { + ecosystem: "go".to_string(), + name: Some(name), + }, + None => Package { + ecosystem: "other".to_string(), + name: None, + }, + } +} diff --git a/src/session.rs b/src/session.rs index 88430bde0..c4b2d8eb0 100644 --- a/src/session.rs +++ b/src/session.rs @@ -58,6 +58,14 @@ impl Logs { } } + /// The logs of the Security run on `repo`. + pub fn of_security_run(repo: &Repo) -> Self { + Logs { + name: "secure".to_string(), + dir: logs::root(repo).join("sessions"), + } + } + /// Where a session's stream is logged: /// `/sessions/--.jsonl`. pub fn path(&self, kind: &str) -> PathBuf { diff --git a/src/skills.rs b/src/skills.rs index 109c03ea0..a186cbbbf 100644 --- a/src/skills.rs +++ b/src/skills.rs @@ -61,7 +61,7 @@ pub fn remove_written() { /// Where the Command wrote the Factory skills out, writing them now if it /// has not yet. -fn written_out() -> Result { +pub(crate) fn written_out() -> Result { let mut written = WRITTEN.lock().unwrap_or_else(PoisonError::into_inner); if let Some(dir) = &*written { return Ok(dir.path().to_owned()); diff --git a/tests/command_surface.rs b/tests/command_surface.rs index 15d67970a..305b100ff 100644 --- a/tests/command_surface.rs +++ b/tests/command_surface.rs @@ -17,6 +17,8 @@ fn assert_help_text(text: &str) { "thirdshift architect base []", "thirdshift pickup", "thirdshift pickup base ", + "thirdshift secure", + "thirdshift secure base ", "thirdshift email-test [
]", "thirdshift setup", "thirdshift update", diff --git a/tests/fakes/claude.rs b/tests/fakes/claude.rs index 169dafff4..6312c0456 100644 --- a/tests/fakes/claude.rs +++ b/tests/fakes/claude.rs @@ -226,6 +226,7 @@ pub fn main(argv: Vec) { let after_result = format!("{}.after-result.{session}", record_path.display()); let final_message = format!("{}.final-message.{session}", record_path.display()); let status = bash() + .env("FAKE_CLAUDE_PROMPT", argv.last().unwrap()) .env("FAKE_CLAUDE_AFTER_RESULT", &after_result) .env("FAKE_CLAUDE_FINAL_MESSAGE", &final_message) .status() diff --git a/tests/fakes/codex.rs b/tests/fakes/codex.rs index e2d10b019..5df554187 100644 --- a/tests/fakes/codex.rs +++ b/tests/fakes/codex.rs @@ -106,6 +106,7 @@ pub fn main(argv: Vec) { let status = Command::new("bash") .arg("-e") .arg(&script) + .env("FAKE_CLAUDE_PROMPT", argv.last().unwrap()) .env("FAKE_CLAUDE_FINAL_MESSAGE", &final_message) .env("FAKE_CODEX_ERROR", &error) .status() diff --git a/tests/fakes/gh.rs b/tests/fakes/gh.rs index 4afbe8c4c..1b3f89a57 100644 --- a/tests/fakes/gh.rs +++ b/tests/fakes/gh.rs @@ -943,8 +943,84 @@ fn repo_prefix(path: &str) -> Option<(&str, &str)> { /// per run with `--jq` set to `WORKFLOW_RUN_LINES`, and for the PRs one /// ` ` line per PR with `--jq` set to /// `COMMIT_PR_LINES`. +/// Advisory create/list/update endpoints, backed by private JSON state. +fn advisory_api(state: &mut Json, positional: &[String], flags: &Flags) { + use std::io::Read; + let (path, query) = positional[0] + .split_once('?') + .unwrap_or((&positional[0], "")); + let (repo, rest) = repo_prefix(path).unwrap(); + check_repo_is(state, Some(repo)); + let method = flag(flags, "method").unwrap_or("GET"); + if method == "GET" { + let wanted = query + .split('&') + .find_map(|pair| pair.strip_prefix("state=")); + let advisories = state.get("advisories").map(Json::items).unwrap_or_default(); + let listed: Vec = advisories + .iter() + .filter(|record| { + wanted.is_none_or(|wanted| record.at("state").as_str() == Some(wanted)) + }) + .cloned() + .collect(); + if flag(flags, "jq") == Some(".[]") { + for record in listed { + println!("{record}"); + } + } else { + println!("{}", Array(listed)); + } + return; + } + let mut input = String::new(); + let mut body = if flag(flags, "input") == Some("-") { + std::io::stdin().read_to_string(&mut input).unwrap(); + parse_json(&input) + } else { + object([]) + }; + let advisories = state.entry("advisories", Array(Vec::new())).items_mut(); + if method == "POST" && rest == "security-advisories" { + if !body.has("summary") || !body.has("description") || !body.has("vulnerabilities") { + die("HTTP 422: required advisory fields missing", 1); + } + let id = format!("GHSA-test-test-{:04}", advisories.len() + 1); + body.set("ghsa_id", string(&id)); + body.set( + "html_url", + string(format!( + "https://github.com/{repo}/security/advisories/{id}" + )), + ); + body.set("state", string("draft")); + advisories.push(body.clone()); + } else if method == "PATCH" { + let id = rest.strip_prefix("security-advisories/").unwrap(); + let advisory = advisories + .iter_mut() + .find(|advisory| advisory.at("ghsa_id").as_str() == Some(id)) + .unwrap(); + let Json::Object(fields) = body else { + panic!("advisory update must be an object"); + }; + for (key, value) in fields { + advisory.set(&key, value); + } + body = advisory.clone(); + } else { + die("fake gh: unsupported advisory operation", 2); + } + save(state); + println!("{body}"); +} + fn api(state: &mut Json, positional: &[String], flags: &Flags) { let path = &positional[0]; + if path.contains("/security-advisories") { + advisory_api(state, positional, flags); + return; + } if path == "user" && flags.is_empty() { let email = state.get("user_email").cloned().unwrap_or(Null); println!( @@ -1770,7 +1846,13 @@ pub fn main(args: Vec) { release_view(&state, &positional, &flags); } ["api", "graphql", rest @ ..] => graphql(&state, rest), - ["api", "--method", "PATCH", rest @ ..] => pr_patch(&mut state, rest), + ["api", "--method", "PATCH", rest @ ..] + if !rest + .iter() + .any(|word| word.contains("/security-advisories")) => + { + pr_patch(&mut state, rest) + } ["api", "--method", "PUT", rest @ ..] => issue_labels_put(&mut state, rest), ["api", "--method", "DELETE", rest @ ..] => issue_label_delete(&mut state, rest), ["api", path, ..] if generate_notes_repo(path).is_some() => { diff --git a/tests/security_run.rs b/tests/security_run.rs new file mode 100644 index 000000000..42fa9e784 --- /dev/null +++ b/tests/security_run.rs @@ -0,0 +1,434 @@ +mod support; + +use std::fs; + +use serde_json::{Value, json}; +use support::{REPO, Scenario}; + +fn finding(fingerprint: &str) -> Value { + json!({ + "verdict": "needs_validation", "fingerprint": fingerprint, + "title": "Unchecked input size", "description": "Private candidate write-up.", + "claimed_root_cause": "Input reaches storage without a bound.", + "trace": [{"kind":"entrypoint", "file":"README.md", "line":1, "scope":"input", "description":"Private trace."}], + "evidence": [{"file":"README.md", "line":1, "description":"Private evidence."}], + "blockers": ["No sandbox available."], + "validation_plan": {"local":"Exercise a bounded fixture."} + }) +} + +#[test] +fn records_a_finding_privately_as_a_draft_without_severity_or_versions() { + let scenario = Scenario::new(); + scenario.origin_has_commit( + "main", + "Cargo.toml", + "[package]\nname = \"widgets\"\nversion = \"1.2.3\"\n", + "Add manifest", + ); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let state = scenario.gh_state(); + let advisory = &state["advisories"][0]; + assert_eq!(advisory["state"], "draft"); + assert_eq!(advisory["summary"], "Unchecked input size"); + assert_eq!(advisory["severity"], Value::Null); + assert_eq!(advisory["cwe_ids"], json!([])); + assert_eq!( + advisory["vulnerabilities"][0]["package"], + json!({"ecosystem":"rust", "name":"widgets"}) + ); + assert_eq!( + advisory["vulnerabilities"][0]["vulnerable_version_range"], + Value::Null + ); + let description = advisory["description"] + .as_str() + .expect("no private advisory was recorded"); + for required in [ + "Private candidate write-up.", + "Private trace.", + "Private evidence.", + "Exercise a bounded fixture.", + "input-size", + "thirdshift's Security run", + scenario.origin_git(&["rev-parse", "main"]).trim(), + ] { + assert!( + description.contains(required), + "missing {required}: {description}" + ); + } + assert!(!result.stderr.contains("Private candidate write-up.")); + assert!(state["prs"].as_array().unwrap().is_empty()); + assert_eq!(state["issues"].as_object().unwrap().len(), 1); + assert_eq!( + scenario.origin_git(&["log", "--format=%s", "main"]), + "Add manifest\nInitial commit\n" + ); +} + +#[test] +fn reads_the_package_from_an_npm_manifest() { + let scenario = Scenario::new(); + scenario.origin_has_commit( + "main", + "package.json", + r#"{"name":"@acme/widgets","version":"1.0.0"}"#, + "Add npm manifest", + ); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + json!({"ecosystem":"npm", "name":"@acme/widgets"}) + ); +} + +fn audit_script(findings: &str) -> String { + format!( + r#" +printf '%s' "$FAKE_CLAUDE_PROMPT" > prompt.txt +output=$(sed -n 's/^Output directory: `\(.*\)`\.$/\1/p' prompt.txt) +test -n "$output" +printf '%s\n' '{findings}' > "$output/findings.json" +printf '%s\n' '[]' > "$output/coverage-ledger.json" +printf '%s\n' '{{"run_status":"complete"}}' > "$output/run-metadata.json" +printf '%s\n' 'Security audit: complete' > "$FAKE_CLAUDE_FINAL_MESSAGE" +"# + ) +} + +#[test] +fn keeps_fingerprints_in_every_advisory_state_and_creates_each_new_finding_once() { + for state in ["draft", "published", "closed", "triage"] { + let scenario = Scenario::new(); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), finding("storage-bound")]).to_string(), + )); + let first = scenario.run(&["secure"]); + assert_eq!(first.code, Some(0), "{}", first.stderr); + let mut github = scenario.gh_state(); + assert_eq!(github["advisories"].as_array().unwrap().len(), 2); + for advisory in github["advisories"].as_array_mut().unwrap() { + advisory["state"] = json!(state); + } + scenario.write_gh_state(&github); + let second = scenario.run(&["secure"]); + assert_eq!(second.code, Some(0), "{}", second.stderr); + assert!( + second + .stderr + .contains("0 new finding(s), 2 already recorded"), + "{}", + second.stderr + ); + assert_eq!(scenario.gh_state()["advisories"], github["advisories"]); + let api_calls = scenario.gh_calls_of("api", "--method"); + assert_eq!(api_calls.iter().filter(|call| call[2] == "POST").count(), 2); + assert!(api_calls.iter().all(|call| call[2] == "POST")); + assert_eq!( + scenario + .entries("home/.thirdshift/logs/acme/widgets/audits") + .len(), + 2 + ); + } +} + +#[test] +fn an_explicit_base_and_harness_audit_origin_without_pulling_local_work() { + let scenario = Scenario::new(); + scenario.origin_has_branch("stable", "main", &["Stable"]); + scenario.user_config_is("[launch]\npull = true\n"); + scenario.agent_does(&audit_script("[]")); + let head = scenario.launch_git(&["rev-parse", "HEAD"]); + let result = scenario.run(&[ + "secure", "base", "stable", "harness", "codex", "model", "gpt-5.5", "effort", "high", + "merge", "base-fix", "parallel", "2", + ]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(scenario.launch_git(&["rev-parse", "HEAD"]), head); + assert_eq!(scenario.launch_git(&["branch", "--show-current"]), "main\n"); + assert!(scenario.claude_calls().is_empty()); + let calls = scenario.codex_calls(); + assert_eq!(calls.len(), 1); + assert_eq!(calls[0]["branch"], ""); + assert!( + calls[0]["prompt"] + .as_str() + .unwrap() + .contains(scenario.origin_git(&["rev-parse", "stable"]).trim()) + ); + scenario.assert_every_codex_session_found_the_factory_skills(); +} + +#[test] +fn a_ready_issue_skips_before_any_session_and_keeps_only_the_activity_log() { + let scenario = Scenario::new(); + scenario.issue_labelled(7, &["ready-for-agent"]); + scenario.user_config_is("[activity]\nquiet_skips = true\n"); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!(result.stdout, ""); + assert_eq!(result.stderr, ""); + assert!(scenario.claude_calls().is_empty()); + let log = fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!( + log.contains("Security run skipped: Ready issue #7"), + "{log}" + ); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/commands") + .exists() + ); +} + +#[test] +fn another_pass_holds_the_security_run_and_its_skip_is_logged() { + let scenario = Scenario::new(); + let waiting = format!( + "touch \"$HOME/../started\"\nwhile [ ! -f \"$HOME/../release\" ]; do sleep 0.05; done\n{}", + audit_script("[]") + ); + scenario.agent_does(&waiting); + let first = scenario.run_until(&["secure"], &[], "started"); + let second = scenario.run(&["secure"]); + let pickup = scenario.run(&["pickup"]); + fs::write(scenario.path("release"), "").unwrap(); + let first = first.finish(); + assert_eq!(first.code, Some(0), "{}", first.stderr); + for skipped in [second, pickup] { + assert_eq!(skipped.code, Some(0), "{}", skipped.stderr); + assert!( + skipped.stderr.contains("another Pass is already running"), + "{}", + skipped.stderr + ); + } + assert_eq!(scenario.claude_calls().len(), 1); + assert_eq!( + scenario + .entries("home/.thirdshift/logs/acme/widgets/commands/secure") + .len(), + 1 + ); + let log = fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!(log.contains("Security run skipped: another Pass is already running")); +} + +#[test] +fn missing_node_fails_before_any_session_and_names_the_prerequisite() { + let scenario = Scenario::new(); + let git = std::process::Command::new("which") + .arg("git") + .output() + .unwrap(); + let git = String::from_utf8(git.stdout).unwrap(); + std::os::unix::fs::symlink(git.trim(), scenario.path("bin/git")).unwrap(); + let path = scenario.path("bin"); + let result = scenario.run_with_env(&["secure"], &[("PATH", path.to_str().unwrap())]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("Node.js is required"), + "{}", + result.stderr + ); + assert!(scenario.claude_calls().is_empty()); + assert_eq!(scenario.entries("work"), vec![REPO]); + assert!( + !scenario + .path("home/.thirdshift/logs/acme/widgets/commands") + .exists() + ); +} + +#[test] +fn missing_final_line_or_invalid_reports_fail_without_recording() { + for (script, cause) in [ + ( + audit_script("[]").replace("Security audit: complete", "No protocol line"), + "without the final line", + ), + ( + audit_script("[]").replace("Security audit: complete", "Security audit: incomplete"), + "ended incomplete", + ), + (audit_script("[{}]"), "validator validate-findings.cjs"), + ( + audit_script("[]").replace( + "'[]' > \"$output/coverage-ledger.json\"", + "'[{}]' > \"$output/coverage-ledger.json\"", + ), + "validator validate-coverage-ledger.cjs", + ), + ] { + let scenario = Scenario::new(); + scenario.agent_does(&script); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains(cause), + "expected {cause}: {}", + result.stderr + ); + assert!(result.stderr.contains("session log:")); + assert!(scenario.gh_state()["advisories"].is_null()); + assert_eq!(scenario.entries("work"), vec![REPO]); + let log = + fs::read_to_string(scenario.path("home/.thirdshift/logs/acme/widgets/activity.log")) + .unwrap(); + assert!(log.contains("Security run started")); + assert!(log.contains("Security run ended: failed:"), "{log}"); + let logs = scenario.entries("home/.thirdshift/logs/acme/widgets/commands/secure"); + let command_log = fs::read_to_string(scenario.path(&format!( + "home/.thirdshift/logs/acme/widgets/commands/secure/{}", + logs[0] + ))) + .unwrap(); + assert!(command_log.contains(cause)); + } +} + +#[test] +fn rejected_candidates_stay_in_the_report_without_a_private_advisory() { + let scenario = Scenario::new(); + let mut rejected = finding("refuted-claim"); + rejected.as_object_mut().unwrap().remove("blockers"); + rejected.as_object_mut().unwrap().remove("validation_plan"); + rejected["verdict"] = json!("rejected"); + rejected["reason"] = json!("The source already checks the bound."); + scenario.agent_does(&audit_script( + &json!([finding("input-size"), rejected]).to_string(), + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"].as_array().unwrap().len(), + 1 + ); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + json!({"ecosystem":"other", "name":null}) + ); +} + +#[test] +fn an_incomplete_run_record_cannot_be_overridden_by_a_complete_final_line() { + let scenario = Scenario::new(); + scenario.agent_does(&audit_script("[]").replace( + "\"run_status\":\"complete\"", + "\"run_status\":\"incomplete\"", + )); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(1), "{}", result.stderr); + assert!( + result.stderr.contains("run-metadata.json"), + "{}", + result.stderr + ); + assert!(scenario.gh_state()["advisories"].is_null()); +} + +#[test] +fn audits_origins_base_without_changing_the_launch_directory() { + let scenario = Scenario::new(); + scenario.origin_has_commit("main", "SECURITY.md", "Trust model", "Document trust model"); + scenario.agent_does(&format!( + "{}\ntest -z \"$(git branch --show-current)\"\ntest -f SECURITY.md\n", + audit_script("[]") + )); + let head = scenario.launch_git(&["rev-parse", "HEAD"]); + fs::write(scenario.launch_dir().join("local.txt"), "local work").unwrap(); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert!( + result + .stderr + .contains("Security audit recorded 0 new finding(s)"), + "{}", + result.stderr + ); + assert_eq!(scenario.launch_git(&["rev-parse", "HEAD"]), head); + assert_eq!( + fs::read_to_string(scenario.launch_dir().join("local.txt")).unwrap(), + "local work" + ); + assert_eq!(scenario.entries("work"), vec![REPO]); + assert_eq!( + scenario.origin_git(&["log", "--format=%s", "main"]), + "Document trust model\nInitial commit\n" + ); + assert!(scenario.gh_state()["prs"].as_array().unwrap().is_empty()); + let prompt = scenario.first_prompt(); + for required in [ + "thirdshift-security-audit", + "full audit mode", + "quick", + "vendored", + "third-party", + "SECURITY.md", + "earlier runs", + "incomplete", + "Security audit: complete", + "You run headless", + "commits, pushes, opens and publishes nothing", + ] { + assert!(prompt.contains(required), "missing {required}: {prompt}"); + } + scenario.assert_every_session_found_the_factory_skills(); +} + +#[test] +fn security_advisories_preserve_python_and_go_manifest_identity() { + for (manifest, contents, package) in [ + ( + "pyproject.toml", + "[project]\nname = \"widgets\"\nversion = \"1.0.0\"\n", + json!({"ecosystem":"pip", "name":"widgets"}), + ), + ( + "go.mod", + "module example.com/acme/widgets\n\ngo 1.24\n", + json!({"ecosystem":"go", "name":"example.com/acme/widgets"}), + ), + ] { + let scenario = Scenario::new(); + scenario.origin_has_commit("main", manifest, contents, "Add manifest"); + scenario.agent_does(&audit_script(&json!([finding("input-size")]).to_string())); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + assert_eq!( + scenario.gh_state()["advisories"][0]["vulnerabilities"][0]["package"], + package, + "package identity was lost for {manifest}" + ); + } +} + +#[test] +fn security_audit_names_the_threat_model_under_docs() { + let scenario = Scenario::new(); + fs::create_dir_all(scenario.launch_dir().join("docs")).unwrap(); + fs::write( + scenario.launch_dir().join("docs/THREAT-MODEL.md"), + "# Threat model\nAll callers are authenticated; tenant isolation is the boundary.\n", + ) + .unwrap(); + scenario.launch_git(&["add", "docs/THREAT-MODEL.md"]); + scenario.launch_git(&["commit", "-m", "Document threat model"]); + scenario.launch_git(&["push", "origin", "main"]); + scenario.agent_does(&audit_script("[]")); + let result = scenario.run(&["secure"]); + assert_eq!(result.code, Some(0), "{}", result.stderr); + let prompt = scenario.first_prompt(); + assert!( + prompt.contains("Read the repository's threat-model document `docs/THREAT-MODEL.md`."), + "the existing threat-model document was not named: {prompt}" + ); +}