Skip to content

Certify roster set via emitting Team Guard Step TG-2 (Closes #1812) - #1837

Merged
bradygaster merged 2 commits into
devfrom
bradygaster-activate-roster-binding-1812
Aug 23, 2026
Merged

bradygaster merged 2 commits into
devfrom
bradygaster-activate-roster-binding-1812

Conversation

@bradygaster

Copy link
Copy Markdown
Collaborator

Closes #1812

Working as Procedures (Prompt Engineer).

The defect (measured)

On fixture aspiregregator-squad-e2e (run 32471509974), plan activate printed a provenance sentence claiming it read the roster from .squad/team.md ## Members -> Name column, reporting lead, reviewer, devrel, security, docs. The fixture's real cast is Keaton, McManus, Fenster, Hockney, Kint. The five reported names are exactly the squad init --preset default scaffold. Two defects: (1) activate bound the hardcoded preset roster, not the committed cast; (2) it claimed a provenance it did not have. Downstream, activate matched the planner's real names against the hardcoded list, found none, and applied no squad:{agent} label — so #1784 Condition 2 "passed" only because activate refused everything. Refusal and correct binding are indistinguishable from the outside.

The fix (structural, not prose)

Add Team Guard Step TG-2 — Certify the Roster Set, modeled on the existing TG-1 TEAM_PRESENT/TEAM_ABSENT guard. It:

  • reads committed HEAD .squad/team.md (git show HEAD: — working-tree preset scaffolds are invisible),
  • finds the ## Members Name column by header (not position — a | Role | Name | table would otherwise leak the Role column with a true-looking citation),
  • emits one lowercased ROSTER_MEMBER: {name} per data row, or a single ROSTER_UNREADABLE: {reason}.

Every minting/binding site (accept squad:{owner}, activate label pre-flight squad:{agent}, plan/impl binders, Check 10) binds only to TG-2's stdout. The summary can only reproduce names the command produced, so provenance is true by construction. ROSTER_UNREADABLE: halts binding with the named reason — never a preset fallback, never a provenance sentence for a read that did not happen.

Prose was chosen against deliberately: the file was already hardened with bold, repeated roster prose and the defect happened anyway. TG-2's stdout is an observable artifact a test can assert; a directive's compliance cannot be observed.

What I measured

  • New test test/gh-aw-activate-roster-binding.test.ts (10 tests) extracts the TG-2 bash block and runs it against real committed-HEAD git repos. All green. Full gh-aw family: 172 passed / 13 skipped across 5 suites.
  • Mutation evidence (each mutation applied, suite run, offending case confirmed red naming the input, then restored):
    • M1 drop tolower -> lowercase cases red, naming ROSTER_MEMBER: Keaton.
    • M2 pin column to position 2 -> header-order case red, naming leaked ROSTER_MEMBER: lead/reviewer (the exact plan activate reads a hardcoded roster, not .squad/team.md — and reports a false provenance (re-diagnoses #1784) #1812 anti-pattern).
    • M3 drop __NOCOL__ sentinel -> no-Name-column case red (misdiagnosed reason).
    • M4 git show HEAD: -> cat -> preset-leak case red, naming the working-tree ROSTER_MEMBER: lead/reviewer/devrel/security/docs; also reddens the static git show HEAD guard.
    • M5/M6/M7 replace absent / no-section / no-rows ROSTER_UNREADABLE with a spurious ROSTER_MEMBER: ghost -> each respective case red naming ghost.
  • Caller enumeration: every squad:{...} and roster site across workflows/ grepped. All minting/binding sites bind to TG-2. The two other roster consumers — squad-implement-worker.md (routes by an already-certified label) and shared/squad.md (gh-aw lifecycle overwrites a merged cast with the default preset on later runs #1657 squad init existence guard) — derive no roster and do not share the defect. No compiled .lock.yml artifacts are committed, so nothing needs regeneration.
  • Prose removal: every compressed block checked against test/gh-aw-*; all pinned substrings survive (verified by the green plan-lifecycle + quality suites). No load-bearing prose dropped.

What I could NOT measure (named boundaries)

  • CR-strip (M8): removing the extraction's sub(/\r$/,"") could not be reddened on the Windows git-bash test host — git show emits LF there (CR normalized before awk), and a trailing pipe quarantines any CR into an unread field. The strip is retained as defense-in-depth (load-bearing on Linux runners for irregular tables); its behavior is reasoned for Linux, not measured on Windows. The CRLF test was rewritten to assert the end-to-end parse invariant (reddenable by M1/M4) rather than claim to prove the strip.
  • Machine-checked postcondition applied_labels ⊆ emitted_roster_members: not reachable under gh-aw safe-outputs — labels are applied by a separate executor job, and TG-2's stdout is not promotable to a cross-job artifact. Emission is shell-enforced/assertable; consumption stays a prompt directive (the good kind of unenforced — it labels itself). The ≥1 label applied completeness rule remains the guard that stops "refused everything" from masquerading as "bound correctly".

Files

  • workflows/squad.md — TG-2 block + rewired binders (net +80/-61; byte budget passes, 87B margin above floor).
  • test/gh-aw-activate-roster-binding.test.ts — new suite (10 tests).
  • .squad/decisions/inbox/procedures-1812-activate-roster-binding.md — decision record (Scribe merges).

No packages/*/src/ touched — no changeset required.

Activate bound `squad:{agent}` labels to the hardcoded preset roster
(lead/reviewer/devrel/security/docs) instead of the repo's committed cast,
and printed a false provenance sentence claiming it read team.md. Prose
hardening had failed repeatedly.

Add Team Guard Step TG-2: a bash block (modeled on the TG-1 TEAM_PRESENT
guard) that reads committed-HEAD team.md, finds the `## Members` `Name`
column by header, and emits `ROSTER_MEMBER: {name}` per row (lowercased) or
`ROSTER_UNREADABLE: {reason}`. Every minting/binding site (accept, activate
gate, plan, impl, Check 10) binds only to TG-2 stdout, so the provenance
claim is true by construction. Unreadable rosters halt with the named
reason -- never a preset fallback, never a provenance sentence for a read
that did not happen.

Add test/gh-aw-activate-roster-binding.test.ts: extracts the TG-2 block and
runs it against real committed-HEAD git repos (10 tests). Mutation-proved
M1-M7 (each reddens naming the offending input); M8 (CR-strip) could not be
reddened on Windows git-bash and is documented as a named boundary.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 23, 2026 03:53
@github-actions

Copy link
Copy Markdown
Contributor

🟡 Impact Analysis — PR #1837

Risk tier: 🟡 MEDIUM

📊 Summary

Metric Count
Files changed 3
Files added 2
Files modified 1
Files deleted 0
Modules touched 3

🎯 Risk Factors

  • 3 files changed (≤5 → LOW)
  • 3 modules touched (2-4 → MEDIUM)

📦 Modules Affected

root (1 file)
  • workflows/squad.md
squad-state (1 file)
  • .squad/decisions/inbox/procedures-1812-activate-roster-binding.md
tests (1 file)
  • test/gh-aw-activate-roster-binding.test.ts

This report is generated automatically for every PR. See #733 for details.

@github-actions

github-actions Bot commented Aug 23, 2026 •

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit ab8649e

PR Scope: 🔧 Infrastructure

⚠️ 3 item(s) to address before review

Status Check Details
❌ Single commit 2 commits — consider squashing before review
✅ Not in draft Ready for review
✅ Branch up to date Up to date with dev
❌ Copilot review No Copilot review yet — it may still be processing
✅ Changeset present No source files changed — changeset not required
✅ Scope clean ⚠️ PR includes 1 .squad/ file(s) — ensure these are intentional
✅ No merge conflicts No merge conflicts
❌ Copilot threads resolved 5 unresolved Copilot thread(s) — fix and resolve before merging
✅ CI passing All checks passing

Files Changed (3 files, +532 −62)

File +/−
.squad/decisions/inbox/procedures-1812-activate-roster-binding.md +146 −0
test/gh-aw-activate-roster-binding.test.ts +305 −0
workflows/squad.md +81 −62

Total: +532 −62


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Squad File Leakage Detected

The following .squad/ files were modified in this PR:

  • .squad/decisions/inbox/procedures-1812-activate-roster-binding.md

These files affect team routing, agent charters, and decisions.
If intentional, ensure approval from the team lead.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical and moderate findings remain in TG-2 enforcement, parsing, roster matching, and label handling.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds TG-2 roster certification from committed .squad/team.md to prevent preset roster leakage during planning and activation.

Changes:

  • Adds committed-HEAD roster parsing and validation.
  • Rewires roster binding guidance.
  • Adds behavioral tests and a decision record.
File summaries
File Description
workflows/squad.md Adds TG-2 and updates roster-binding rules.
test/gh-aw-activate-roster-binding.test.ts Tests roster extraction and edge cases.
.squad/decisions/inbox/procedures-1812-activate-roster-binding.md Documents the design and limitations.
Review details

Suppressed comments (9)

test/gh-aw-activate-roster-binding.test.ts:29

  • The workflow text is read without normalizing CRLF, and only the fence regex is CRLF-aware; the extracted shell block therefore retains \r at the end of each command. On the Windows path this suite explicitly supports, Git Bash can parse tokens such as then\r incorrectly and fail before the behavioral cases execute. Normalize the workflow text to LF as the other gh-aw suites do.
const workflow = readFileSync(SQUAD_WORKFLOW, 'utf8');

test/gh-aw-activate-roster-binding.test.ts:186

  • These tests execute only the TG-2 producer. They do not assert that the five downstream binders (plan, accept, implementation, Check 10, and activate) actually consume TG-2 output, so reverting any one of those sites to an independent or preset roster would leave all ten cases green. Add structural assertions enumerating those sections and requiring the TG-2 certified-set contract, even though the applied-label subset is unavailable cross-job.
    expect(TG2_BLOCK.length, 'TG-2 bash block extracted empty from workflows/squad.md').toBeGreaterThan(0);
    expect(TG2_BLOCK, 'TG-2 block must emit ROSTER_MEMBER: lines').toContain('ROSTER_MEMBER:');
    expect(TG2_BLOCK, 'TG-2 block must emit ROSTER_UNREADABLE: on failure').toContain('ROSTER_UNREADABLE:');
    // It must read the *committed* revision, not the working tree — the entire point.
    expect(TG2_BLOCK, 'TG-2 must read the committed HEAD revision of team.md').toMatch(/git show HEAD:\.squad\/team\.md/);

workflows/squad.md:1319

  • TG-2's stdout is not connected to the create-issue payload or to the executor: create-issue still accepts a model-generated arbitrary label list. These lines are therefore prompt-only; a model can still emit squad:lead or omit all roster labels while the TG-2 tests pass. An enforceable check in the label-application path (or a safe-output/schema constraint) is needed before this guarantees the #1812 binding invariant.
4. For every `Agent` value, mint `squad:{agent}` only when its lowercased form matches
   a certified `ROSTER_MEMBER:` name, or the value is exactly `@copilot`.
5. A value matching no certified name and not `@copilot` MUST NOT become a
   `squad:{agent}` label: apply only `squad` for that issue and record the value under a
   `Non-roster agent values` heading, naming the certified set it should come from.

workflows/squad.md:473

  • Valid Markdown tables may omit the final pipe. For a separator such as |------|------, this regex does not match, so the separator's second cell is emitted as ROSTER_MEMBER: ------ before the real names, creating a bogus roster label. Allow the final delimiter to be optional and add a regression case.
      if($0 ~ /^\|[-: |]*\|$/)next

workflows/squad.md:477

  • __NOCOL__ is an in-band marker and is detected with substring matching. A legitimate cast name containing that token (including the exact name) makes a readable table report ROSTER_UNREADABLE, dropping valid bindings. Use awk's exit status or a separately scoped status channel instead of mixing a marker into member data.
  if printf '%s\n' "$ROSTER" | grep -q "__NOCOL__"; then

workflows/squad.md:868

  • This step still tells the model to read the working-tree .squad/team.md directly, contradicting the invariant above that binders use only TG-2 stdout. In the targeted failure mode that copy can be the uncommitted preset while TG-2 intentionally reads committed HEAD, so the planner can repopulate Owner/Agent from the wrong roster. Remove the independent roster read and use only the certified output for binding.
3. Read `.squad/team.md` if it exists. **Owner/Agent binding rule:** permitted
   `Owner`/`Agent` values are Team Guard Step TG-2's certified roster set — the
   `Name` column of `## Members` in **this repository's** `.squad/team.md` — plus
   `@copilot`. Map each item's domain to a member via `.squad/routing.md` and emit
   that member's exact `Name` cell — never a recalled name, never another column

workflows/squad.md:1316

  • This condition permits activation to mint squad:@copilot, but TG-2 explicitly says that @copilot must never become a squad:{name} label; the accept path also applies only squad for that fallback. In addition, the check is case-insensitive while {agent} is left raw, so an exact Name such as Keaton can produce squad:Keaton instead of the lowercased label. Make the label token the lowercased certified name and exclude @copilot from agent-label creation.
4. For every `Agent` value, mint `squad:{agent}` only when its lowercased form matches
   a certified `ROSTER_MEMBER:` name, or the value is exactly `@copilot`.

workflows/squad.md:1125

  • The structural validation immediately above still defines agent validity by reading .squad/team.md directly (line 1117), not by checking TG-2's ROSTER_MEMBER: output. A working-tree/default-preset value can therefore pass this validation even when it is absent from the certified stdout and be posted in the implementation artifact. Update this validation to use only TG-2's emitted set, matching the new binder.
Re-check every `Agent` against the Step 2 binding rule before posting.

workflows/squad.md:462

  • A committed empty (or newline-only) team.md makes git show succeed but leaves TEAM_MD empty, so this branch reports .squad/team.md absent from HEAD, which is false and hides the distinct empty/no-section failure. Capture the git show exit status separately from the content check and report the actual condition.
TEAM_MD="$(git show HEAD:.squad/team.md 2>/dev/null)"
if [ -z "$TEAM_MD" ]; then
  • Files reviewed: 3/3 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread workflows/squad.md
if(col==0){for(i=1;i<=NF;i++){h=$i;gsub(/^[ \t]+|[ \t]+$/,"",h);if(h=="Name")col=i}next}
if($0 ~ /^\|[-: |]*\|$/)next
if(col==0)next
n=$col;gsub(/^[ \t]+|[ \t]+$/,"",n);if(n!="")print tolower(n)}
Comment thread workflows/squad.md
Comment on lines +1320 to +1323
6. Completeness: when the plan names at least one roster `Agent`, at least one
`squad:{agent}` label MUST be applied across the created issues. Zero labels on a
plan with roster owners is a binding failure, not a pass — report it, don't proceed
silently.
Comment thread workflows/squad.md
Comment on lines +458 to +460
Every step that mints a `squad:{name}` label or binds an `Owner`/`Agent` value binds **only** to this command's stdout. Run once.

```bash
Comment thread workflows/squad.md
Comment on lines +464 to +469
elif ! printf '%s\n' "$TEAM_MD" | awk '{sub(/\r$/,"")} /^## Members/{f=1} END{exit !f}'; then
echo "ROSTER_UNREADABLE: no ## Members section in .squad/team.md"
else
ROSTER="$(printf '%s\n' "$TEAM_MD" | awk -F'|' '
{sub(/\r$/,"")}
/^## Members/{f=1;next}
Comment thread workflows/squad.md
/^## Members/{f=1;next}
f&&/^#/{f=0}
f&&/^\|/{
if(col==0){for(i=1;i<=NF;i++){h=$i;gsub(/^[ \t]+|[ \t]+$/,"",h);if(h=="Name")col=i}next}
@bradygaster

Copy link
Copy Markdown
Collaborator Author

FIDO adversarial review — VERDICT: APPROVE WITH NITS

Measured in a branch worktree with node_modules (HEAD 41d3058b). Bar as always: every mutation must go red AND name the offending input — a status-only assertion passes a truncating parser.

Standing asks — all MET

Per-case mutations (7), all red + naming input:

Mut Target Result
M1 drop tolower lowercasing RED — got: ROSTER_MEMBER: Kint
M2 pin col=2 #1812 anti-pattern RED — leaks ROSTER_MEMBER: lead/reviewer on Role-first table
M3 drop __NOCOL__ reason specificity RED — names which reason, not status-only
M4 git show HEAD:→cat committed-HEAD read RED — extraction guard + working-tree leak both name input
M5–M7 + 4th: each ROSTER_UNREADABLE:→ROSTER_MEMBER: ghost 4 named reasons RED (5 failed) — each names the ghost leak

Caller enumeration (incl. outside workflows/): repo-wide grep of packages//scripts/ — every squad:{member} hit reacts to already-applied labels or produces team.md; none reads the Name column to mint. Only minting path is workflows/squad.md, now TG-2-bound. No third caller — the #1832 caller-blindness trap is clean.

Honesty claims — both VERIFIED by measurement

  1. M8 CR-strip: removed BOTH sub(/\r$/,"") strips → CRLF test still passes. Confirms the strip is not reddenable on this substrate (trailing table-pipe quarantines the CR into an unread field). Retained as reasoned-for-Linux defense-in-depth; the CRLF test asserts only the end-to-end invariant (reddenable by M1/M4), not the strip. Honest. Does not compound gh-aw quality suite silently skips 13 tests (28 assertions) on Windows via existsSync('/bin/sh') probe #1833 — new suite ran 10/10, 0 skips (Git-bash fallback + hard-fail first test).
  2. Postcondition unreachable: measured against frontmatter — agent job has issues: read, not write; labels flow through the create-issue safe-output executed by a separate executor job. TG-2's bash-sandbox stdout genuinely can't cross jobs, so applied_labels ⊆ emitted_roster_members is not shell-assertable in-run. Claim correct; consumption-as-directive is the acceptable self-labeling form (like RETRO hop-1, Enforce the gh-aw shell input security contract with a gate over compiled workflow output #1834).

#1784 completeness rule — CONFIRMED

Activation step 6: "Zero labels on a plan with roster owners is a binding failure, not a pass." Prevents "refused everything" masquerading as "bound correctly" — the exact #1784 false pass.

Prose compression — nothing load-bearing dropped

(Reconciled against gh pr diff; local dev was stale.) All −61 is the old file-based "read team.md, trust recall" language, replaced by TG-2 binding. Every constraint survived: Role-column exclusion ✅, "no roster → ❌ Critical → stop" ✅, "name the offending value" ✅, lowercasing ✅.

The finding (main ask) — L1117: REAL, NON-BLOCKING

Three validation sites; two rewired, one missed:

  • L1111 (Step 2 binding) → TG-2 ✅
  • L1165–1179 (Check 10) → ROSTER_MEMBER: ✅
  • L1117 (Step 3 "Validate Structure") still names the file: "every Agent value appears verbatim in the Name column … in .squad/team.md" ❗ — unchanged by the PR; absent from Procedures' own caller enumeration.

Non-blocking because the authoritative minting site and the authoritative gate (Check 10) are both TG-2-bound; L1117 is an inline sanity pre-check re-validating values Step 2 already sourced from TG-2, and any miss is caught by TG-2-bound Check 10 downstream. But a real should-fix — it reintroduces the untrusted source at a validation gate, the exact residual that let #1812 recur 5×. One-line fix: rebind L1117 to "TG-2's certified roster set (ROSTER_MEMBER: lines), or @copilot". (Reasoned, not shell-measured — it's LLM prose.)

CI: test = pass (4m41s), all gates green (2 expected path-filter skips). Local suite 10/10, tree clean.


Recommendation: merge as-is; track L1117 as a one-line follow-up. Procedures is locked out of its own revision under reviewer protocol — if fixing pre-merge, route the L1117 one-liner to EECOM.

(Lands as COMMENTED — GitHub refuses a formal Approve from the account that owns the PR. Verdict stated in text: APPROVE WITH NITS.)

FIDO nit on #1837: Step 3 "Validate Structure" (L1117) still derived
agent validity from the untrusted file (.squad/team.md) while the binder
(L1111) and Check 10 (L1165) bind to Team Guard Step TG-2's certified
ROSTER_MEMBER: set. A validator that derives membership differently from
the binder can pass work the binder would reject. Rewire the validator to
the certified set; retain the verbatim `Name`-column phrase as provenance.

Extends #1793: the criterion a check applies is part of the check.

Closes #1812

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9cace6b6-8abc-4aa4-ac36-db3536dbb3bb
@bradygaster
bradygaster merged commit 4b32f7b into dev Aug 23, 2026
17 checks passed
@bradygaster
bradygaster deleted the bradygaster-activate-roster-binding-1812 branch September 9, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

plan activate reads a hardcoded roster, not .squad/team.md — and reports a false provenance (re-diagnoses #1784)

2 participants