Skip to content

fix(workflows): give plan validation adversarial teeth - #1870

Merged
bradygaster merged 3 commits into
devfrom
bradygaster-1757-adversarial-validation
Aug 25, 2026
Merged

bradygaster merged 3 commits into
devfrom
bradygaster-1757-adversarial-validation

Conversation

@bradygaster

Copy link
Copy Markdown
Collaborator

Summary

  • add a bounded Fact Checker Devil's Advocate sub-agent that produces the established five-part advisory brief
  • extend /squad plan validate from ten structural checks to sixteen structural, adversarial, and verdict-ownership checks
  • fail closed on unavailable/incomplete evidence, forwarded verdicts, or missing validator synthesis
  • preserve the existing implementation -> validate -> accept scope -> accept implementation -> activate order
  • add behavioral lifecycle tests that reject a neatly formatted structural-only PASS

Closes #1757

Validation

  • strict gh aw compile passed in an isolated .github/workflows fixture
  • compiled lock retained the Squad runtime prompt import
  • gh-aw-equivalent extraction discarded 0 bytes and delivered the Fact Checker plus all five evidence sections
  • TypeScript syntax check passed
  • five mutations went red: removed pre-mortem check, bypassed fail-closed verdict, removed verdict ownership, removed the skill/agent boundary, and removed alternative evidence
  • focused Vitest, targeted ESLint, and the full build were attempted but could not execute because the required package proxy lacks lockfile-pinned vitest@4.1.11; the failed restore left @typescript-eslint/parser and tsc unavailable. No manifest or lockfile workaround was made.

Security review

Prompt-body and test changes only. No frontmatter, permissions, secrets, actions, network access, or safe-output configuration changed. The isolated compiler's first-run safe-update warning named the workflow's existing SQUAD_GITHUB_APP_PRIVATE_KEY and SQUAD_GITHUB_TOKEN; this PR does not add or modify either secret.

Closes #1757

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 25, 2026 04:04
@github-actions

Copy link
Copy Markdown
Contributor

🟡 Impact Analysis — PR #1870

Risk tier: 🟡 MEDIUM

📊 Summary

Metric Count
Files changed 2
Files added 0
Files modified 2
Files deleted 0
Modules touched 2

🎯 Risk Factors

  • 2 files changed (≤5 → LOW)
  • 2 modules touched (2-4 → MEDIUM)

📦 Modules Affected

root (1 file)
  • workflows/squad.md
tests (1 file)
  • test/gh-aw-plan-lifecycle.test.ts

This report is generated automatically for every PR. See #733 for details.

@github-actions

github-actions Bot commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit 2505e6c

PR Scope: 🔧 Infrastructure

⚠️ 3 item(s) to address before review

Status Check Details
❌ Single commit 3 commits — consider squashing before review
✅ Not in draft Ready for review
❌ Branch up to date dev is 1 commit(s) ahead — rebase recommended
❌ Copilot review No Copilot review yet — it may still be processing
✅ Changeset present No source files changed — changeset not required
✅ Scope clean No .squad/ or docs/proposals/ files
✅ No merge conflicts No merge conflicts
✅ Copilot threads resolved 0 active Copilot thread(s) resolved (2 outdated skipped)
✅ CI passing All checks passing

Files Changed (2 files, +267 −8)

File +/−
test/gh-aw-plan-lifecycle.test.ts +162 −5
workflows/squad.md +105 −3

Total: +267 −8


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The PR introduces contract vocabulary/parsing inconsistencies (risk-acceptance heading) and a correctness issue in the updated skillBlock() helper that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR strengthens the long-path planning lifecycle by turning /squad plan validate into a true gate: it now requires adversarial Devil’s Advocate evidence and validator-owned synthesis (PASS/FAIL), rather than allowing a structurally neat but substantively weak plan to pass.

Changes:

  • Expands squad-plan-validate from 10 structural checks to 16 checks including adversarial evidence requirements and explicit verdict-ownership synthesis.
  • Introduces a bounded fact-checker sub-agent definition intended to supply the five-part Devil’s Advocate brief as advisory input (not a verdict).
  • Adds lifecycle contract tests to prevent regressions and to ensure structural-only validation artifacts are rejected.
File summaries
File Description
workflows/squad.md Extends plan validation contract with adversarial checks and adds a Fact Checker Devil’s Advocate sub-agent definition.
test/gh-aw-plan-lifecycle.test.ts Adds behavioral contract tests for adversarial validation and updates parsing helpers to recognize explicit end-skill markers and the new agent block.
Review details

Suppressed comments (1)

test/gh-aw-plan-lifecycle.test.ts:440

  • This test expects the Fact Checker to emit a "##### Risk acceptance" section, but the validation gate names the check/section "Remaining risk acceptance". Update the expected heading to match the workflow contract so the test enforces the same vocabulary the prompt uses.
      '##### Risk acceptance',
  • Files reviewed: 2/2 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/gh-aw-plan-lifecycle.test.ts Resolved
Comment thread workflows/squad.md Resolved
brady gaster added 2 commits August 24, 2026 21:49
Closes #1757

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 77d7fed5-63d6-4767-8fd5-7969bc476f0d
Closes #1757

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 77d7fed5-63d6-4767-8fd5-7969bc476f0d
@bradygaster
bradygaster merged commit 63426de into dev Aug 25, 2026
17 checks passed
@bradygaster
bradygaster deleted the bradygaster-1757-adversarial-validation branch September 9, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Give /squad plan validate real adversarial teeth

2 participants