From adb718b23dc45dcc0950b455f1b13957d8e956ac Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 31 Aug 2026 14:59:21 -0700 Subject: [PATCH 1/5] fix(gh-aw): prevent silent truncation of activation label operations Working as FIDO (Quality Owner). A 50-issue activation could lose `add_labels` operations while the run still reported success. Two causes: 1. `add-labels: max: 80` was below the worst case under the reading gh-aw's own injected prose invites. The compiler emits "Maximum {max} label(s) can be added", which reads as a budget of label NAMES; a full 50-issue activation applying `squad` + `squad:{agent}` needs 100. An agent taking that phrasing literally could conclude it had overrun and stop labeling early or batch issues together. 2. gh-aw v0.87.10 drops an over-limit item rather than failing. Its collector (`collect_ndjson_output.cjs`) rejects the item and `continue`s, pushing a string into `errors`; those are emitted with `core.warning`, never `core.setFailed`. The run finishes green with label operations missing and nothing announces it. Verified against the pinned runtime rather than inferred: `max` caps safe-output ITEMS (tool calls) per type, not label names inside a call. A two-label call costs one item. Capacity: largest supported activation is 50 issues (`enterprise` profile `max_issues: 50`, the highest documented profile limit, and the same threshold `squad-plan-activate` uses to force phased activation). Worst case at that size is 50 `create-issue` items, 50 `add_labels` calls, <=2 labels per call, 100 label names across the run. Changes: - `add-labels: max` 80 -> 110, sized to cover the worst case under BOTH readings (50 calls, 100 names) so no interpretation of the cap can justify dropping a label operation. `create-issue` stays at 75. - Record the derivation and the item-vs-label-name semantics in the activation skill, plus the fact that an over-limit item is dropped, not failed. - New Step 2e Label-Operation Reconciliation: track `activated` vs `labeled`, count a never-made/rejected/errored call as unlabeled, and on shortfall call `report_incomplete` naming the affected work items. gh-aw treats that as a failure signal even on successful exit, so a truncated activation can no longer be recorded as clean. Cap exhaustion becomes a nameable cause when observed; #1683's rule against guessing at caps is preserved and narrowed. Tests: `test/gh-aw-activation-capacity.test.ts` (27) covers the derived maximum, capacity under both readings, at-maximum and one-over boundaries, and the reconciliation contract. Compiled-artifact assertions prefer runtime evidence over prose: the declared cap reaching `GH_AW_SAFE_OUTPUTS_CONFIG`, the injected constraint carrying the same number, `report_incomplete` being present (and so callable), and agent-job permissions staying `issues: read`. Fails closed if `gh aw` is absent, per #1834. Mutation-checked: reverting the cap to 80 fails 4. No changes to temporary-ID linkage (#1962), fast-path parity (#1959), label result reporting (#1963), broad contract coverage (#1960), or E4 (#1958). Step 2e is deliberately neutral about how `add_labels` identifies its target so it does not conflict with #1962. Closes #1961 Parent #1957 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- test/gh-aw-activation-capacity.test.ts | 466 +++++++++++++++++++++++++ workflows/squad.md | 49 ++- 2 files changed, 513 insertions(+), 2 deletions(-) create mode 100644 test/gh-aw-activation-capacity.test.ts diff --git a/test/gh-aw-activation-capacity.test.ts b/test/gh-aw-activation-capacity.test.ts new file mode 100644 index 000000000..a0559c1ed --- /dev/null +++ b/test/gh-aw-activation-capacity.test.ts @@ -0,0 +1,466 @@ +/** + * Activation label-operation capacity and silent-truncation safeguards (#1961, parent #1957). + * + * ## The defect + * + * `workflows/squad.md` promises a `squad` (and where certified, `squad:{agent}`) label on + * every issue an activation creates. Those labels are applied by the `add-labels` safe + * output, which is governed by a `max`. Two independent problems made a large activation + * able to lose label operations *while still reporting a clean success*: + * + * 1. **`max` is ambiguous in gh-aw's own surface area.** The compiler injects the tool + * constraint as "Maximum {N} label(s) can be added", which reads as a budget of label + * *names*. The runtime enforces something different — see below. An agent that believed + * the label reading would conclude a 50-issue activation (up to 100 label names) had + * overrun its budget and could stop labeling early or batch issues together. + * 2. **An over-limit item is dropped, not failed.** gh-aw v0.87.10's collector + * (`actions/setup/js/collect_ndjson_output.cjs`) rejects the offending item and + * `continue`s, pushing a string into `errors`. Those errors are emitted with + * `core.warning(...)`, never `core.setFailed(...)`. The run finishes green with label + * operations missing. Nothing in the run announces the truncation. + * + * ## Runtime semantics this suite is written against (gh-aw v0.87.10, the CI pin) + * + * `collect_ndjson_output.cjs`: + * + * ```js + * const typeCount = parsedItems.filter(existing => existing.type === itemType).length; + * const maxAllowed = getMaxAllowedForType(itemType, expectedOutputTypes); + * if (typeCount >= maxAllowed) { + * errors.push(`Line ${i + 1}: Too many items of type '${itemType}'. Maximum allowed: ${maxAllowed}.`); + * continue; // <- the item is DROPPED + * } + * ``` + * + * `max` therefore caps **safe-output items (tool calls) of that type**, not label names + * inside one call. One `add_labels` call carrying two labels costs one item, not two. + * + * ## The capacity calculation this suite locks in + * + * Largest supported activation = **50 issues** — `enterprise`'s `max_issues: 50`, the + * highest documented profile limit in `workflows/shared/squad-planning-policy.md`, and the + * same threshold `squad-plan-activate`'s Output Budget Awareness uses to force phased + * activation. At that size, worst case: + * + * | Safe output | Worst case | Configured | + * |--------------------------------|-----------:|-----------:| + * | `create-issue` items | 50 | 75 | + * | `add_labels` items (calls) | 50 | 110 | + * | labels within one call | 2 | n/a | + * | label names across the run | 100 | 110 | + * + * `add-labels: max` is sized to satisfy the worst case under **both** readings (50 calls + * and 100 label names) so that no interpretation of the cap can justify dropping a label + * operation. + * + * ## Out of scope (sibling issues — do not broaden these assertions here) + * + * - #1962 temporary-ID linkage between `create_issue` and `add_labels`. This suite is + * deliberately neutral about *how* an `add_labels` call identifies its target, so it + * does not conflict with that change. + * - #1959 `/squad activate` fast-path parity. + * - #1963 label-result reporting beyond the explicit overflow/incomplete signal. + * - #1960 broad safe-output contract coverage; #1958 E4 evidence. + */ + +import { afterAll, describe, expect, it } from 'vitest'; +import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { execFileSync, spawnSync } from 'node:child_process'; +import { join } from 'node:path'; +import { extractSafeOutputsConfigJson } from './helpers/gh-aw-lock'; + +const WORKFLOWS_DIR = join(process.cwd(), 'workflows'); +const SQUAD_WORKFLOW = join(WORKFLOWS_DIR, 'squad.md'); +const PLANNING_POLICY = join(WORKFLOWS_DIR, 'shared', 'squad-planning-policy.md'); +const TEST_WORKSPACES_DIR = join(process.cwd(), '.test-workspaces-activation-capacity'); + +afterAll(() => { + rmSync(TEST_WORKSPACES_DIR, { recursive: true, force: true }); +}); + +/** Read a text file with line endings normalized to LF (Windows checkouts materialize CRLF). */ +function readText(filePath: string): string { + return readFileSync(filePath, 'utf8').replace(/\r\n/g, '\n'); +} + +const workflow = readText(SQUAD_WORKFLOW); +const policy = readText(PLANNING_POLICY); + +const ACTIVATE_START = workflow.indexOf('## skill: `squad-plan-activate`'); +const ACTIVATE_END = workflow.indexOf('## end skill: `squad-plan-activate`'); +expect(ACTIVATE_START, '"## skill: `squad-plan-activate`" is missing').toBeGreaterThan(-1); +expect(ACTIVATE_END, '"## end skill: `squad-plan-activate`" is missing').toBeGreaterThan(ACTIVATE_START); +const activateSkill = workflow.slice(ACTIVATE_START, ACTIVATE_END); +const activateProse = activateSkill.replace(/\s+/g, ' '); + +// --------------------------------------------------------------------------- +// Derived capacity model — every input is read from a source of truth, so drift +// in the policy file or the frontmatter fails here instead of silently changing +// the safe number. +// --------------------------------------------------------------------------- + +/** + * The largest documented activation, derived from the planning policy rather than + * hardcoded: the maximum `max_issues` across all documented profiles. + */ +function documentedMaxIssues(): number { + const values = [...policy.matchAll(/max_issues[`|:\s]+(\d+)/g)].map(m => Number(m[1])); + expect(values.length, 'squad-planning-policy.md must document max_issues values').toBeGreaterThan(0); + return Math.max(...values); +} + +/** Read `max` for a safe output straight out of the workflow frontmatter block. */ +function frontmatterMax(safeOutput: string): number { + const frontmatter = workflow.match(/^---\n([\s\S]*?)\n---/)?.[1] ?? ''; + expect(frontmatter, 'workflows/squad.md must have frontmatter').not.toBe(''); + const block = frontmatter.match(new RegExp(`\\n ${safeOutput}:\\n([\\s\\S]*?)(?=\\n [\\w-]+:\\n|$)`))?.[1]; + expect(block, `frontmatter safe-outputs.${safeOutput} block must exist`).toBeDefined(); + const max = block?.match(/^\s*max:\s*(\d+)\s*$/m)?.[1]; + expect(max, `safe-outputs.${safeOutput} must declare an explicit max`).toBeDefined(); + return Number(max); +} + +/** Worst-case labels applied to a single created issue: `squad` + one `squad:{agent}`. */ +const MAX_LABELS_PER_ISSUE = 2; + +describe('gh-aw: activation capacity is calculated, recorded, and bounded (#1961)', () => { + it('derives the documented maximum activation size from the planning policy', () => { + // enterprise is the highest documented profile; if a profile ever raises + // max_issues above it, this recomputes and the capacity assertions below + // re-evaluate against the new number rather than silently going stale. + expect(documentedMaxIssues()).toBe(50); + }); + + it('the activation skill enforces the same threshold it is sized for', () => { + // The capacity math is only meaningful if the workflow actually refuses to + // attempt a larger single run. > 50 must route to phased activation. + expect( + /total > 50/.test(activateSkill) && /phased activation/i.test(activateSkill), + 'Output Budget Awareness must force phased activation above the documented maximum.', + ).toBe(true); + }); + + it('records the capacity calculation as workflow rationale, not folklore', () => { + expect( + /Activation capacity budget/i.test(activateSkill), + 'The activation skill must record the capacity calculation so the configured caps ' + + 'are explainable and cannot be lowered by accident.', + ).toBe(true); + // The rationale must name the worst-case operand counts, not just the caps. + expect(activateProse).toMatch(/one per epic\/task/i); + expect(activateProse).toMatch(/one per created issue/i); + }); + + it('distinguishes limits on output items/calls from labels per call', () => { + // This is the specific ambiguity #1961 exists to close. The prose must state + // that `max` counts items/tool calls, and that labels inside one call are not + // individually charged against it. + expect( + /limits safe-output items \(tool calls\), not label names inside a call/i.test(activateProse), + 'The workflow must state explicitly that `max` counts safe-output items (tool ' + + 'calls) rather than label names within a call.', + ).toBe(true); + expect( + /consumes \*\*one\*\* item, not two/i.test(activateProse) || /one\*\* item, not two/i.test(activateProse), + 'The workflow must show that a two-label call costs one item, so the agent cannot ' + + 'conclude it must ration labels.', + ).toBe(true); + }); + + it('warns that an over-limit item is dropped with a warning rather than failing the run', () => { + // gh-aw's collector pushes a string into `errors` and the job emits it via + // core.warning — never core.setFailed. Truncation is invisible unless the + // workflow itself detects it. + expect( + /dropped, not failed/i.test(activateProse), + 'The workflow must state that an over-limit safe-output item is dropped rather ' + + 'than failing the run — this is why self-validation is mandatory.', + ).toBe(true); + expect(activateProse).toMatch(/finish green|still succeeds/i); + }); +}); + +describe('gh-aw: configured caps cover the worst case under both readings of max (#1961)', () => { + const maxIssues = documentedMaxIssues(); + const createIssueMax = frontmatterMax('create-issue'); + const addLabelsMax = frontmatterMax('add-labels'); + + it('create-issue covers one item per activated issue with margin', () => { + expect(createIssueMax).toBeGreaterThanOrEqual(maxIssues); + // Bounded, not unbounded: an accidental extra zero should fail review here. + expect(createIssueMax).toBeLessThanOrEqual(maxIssues * 3); + }); + + it('add-labels covers one CALL per activated issue (the runtime reading)', () => { + // Runtime semantics: max caps NDJSON items of type add_labels. + expect(addLabelsMax).toBeGreaterThanOrEqual(maxIssues); + }); + + it('add-labels also covers every LABEL NAME in the run (the prompt-prose reading)', () => { + // gh-aw injects "Maximum {max} label(s) can be added". Even if the agent takes + // that literally, a full 50-issue activation applying squad + squad:{agent} to + // every issue (100 names) must still fit, or the agent can rationalize dropping + // label operations. This is the assertion that would have failed at max: 80. + const worstCaseLabelNames = maxIssues * MAX_LABELS_PER_ISSUE; + expect(worstCaseLabelNames).toBe(100); + expect( + addLabelsMax, + `add-labels max (${addLabelsMax}) must cover the worst-case ${worstCaseLabelNames} ` + + 'label names so the "Maximum N label(s)" constraint gh-aw injects cannot be read ' + + 'as a reason to stop labeling early.', + ).toBeGreaterThanOrEqual(worstCaseLabelNames); + }); + + it('keeps the margin explicit and bounded, not open-ended', () => { + const worstCase = Math.max(maxIssues, maxIssues * MAX_LABELS_PER_ISSUE); + expect(addLabelsMax).toBeGreaterThan(worstCase); + expect( + addLabelsMax, + 'The margin above the worst case must stay bounded — a very large cap trades a ' + + 'silent-truncation bug for an unbounded-write one.', + ).toBeLessThanOrEqual(worstCase * 2); + }); + + it('never lets add-labels fall below create-issue (one call per created issue)', () => { + expect(addLabelsMax).toBeGreaterThanOrEqual(createIssueMax); + }); +}); + +// --------------------------------------------------------------------------- +// Boundary behaviour: exactly at the maximum, and one over it. +// --------------------------------------------------------------------------- + +describe('gh-aw: activation size boundaries (#1961)', () => { + const maxIssues = documentedMaxIssues(); + const createIssueMax = frontmatterMax('create-issue'); + const addLabelsMax = frontmatterMax('add-labels'); + + it('AT the maximum supported size, every operation fits without truncation', () => { + const issues = maxIssues; // 50 + expect(issues, 'create-issue items at max size must fit').toBeLessThanOrEqual(createIssueMax); + expect(issues, 'add_labels calls at max size must fit').toBeLessThanOrEqual(addLabelsMax); + expect( + issues * MAX_LABELS_PER_ISSUE, + 'label names at max size must fit even under the prose reading of max', + ).toBeLessThanOrEqual(addLabelsMax); + }); + + it('ONE OVER the maximum supported size is routed to phased activation, not attempted whole', () => { + const issues = maxIssues + 1; // 51 + expect(issues).toBeGreaterThan(maxIssues); + // The workflow must not silently attempt an oversized run. The guard is a + // documented threshold on the plan total, evaluated before any create-issue call. + expect( + /Count expected issues before starting/i.test(activateProse), + 'The over-limit guard must run before issue creation begins, not after.', + ).toBe(true); + expect( + /If total > 50: recommend phased activation/i.test(activateProse), + 'One over the documented maximum must recommend phased activation and proceed ' + + 'with the current phase only.', + ).toBe(true); + }); + + it('ONE OVER a safe-output cap must surface as an explicit incomplete result', () => { + // The cap itself cannot report this (dropped item + warning only), so the + // workflow must name cap exhaustion as a reportable cause with affected items. + expect( + /Cap exhaustion is a reportable/i.test(activateProse), + 'Reaching a safe-output cap must be an explicitly reportable cause of an ' + + 'incomplete activation.', + ).toBe(true); + expect( + /list the work items that did not fit/i.test(activateProse), + 'An overflow report must identify the affected work items, not just state that ' + + 'overflow happened.', + ).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// Self-validation: expected activated items vs targeted label operations. +// --------------------------------------------------------------------------- + +describe('gh-aw: self-validation reconciles activated items with label operations (#1961)', () => { + it('declares a label-operation reconciliation step', () => { + expect( + /\*\*2e\. Label-Operation Reconciliation/i.test(activateSkill), + 'Step 2e must exist: counting created issues alone cannot detect a lost label ' + + 'operation.', + ).toBe(true); + }); + + it('compares expected activated items against successful label operations', () => { + expect(activateProse).toMatch(/`activated`.*issues created or recognized/i); + expect(activateProse).toMatch(/`labeled`.*add_labels.*returned successfully/i); + expect( + /labeled < activated/.test(activateSkill), + 'The reconciliation must compare the two counts explicitly.', + ).toBe(true); + }); + + it('counts a missing, rejected, or errored label call as unlabeled', () => { + // A dropped over-limit item produces no error the agent can see, so "no + // successful result" — not "an observed error" — must be the failing condition. + expect( + /never made, was rejected, or returned an error counts as \*\*unlabeled\*\*/i.test(activateProse), + 'A label call that was never made must count as unlabeled, otherwise a dropped ' + + 'item is invisible to the reconciliation.', + ).toBe(true); + }); + + it('escalates a shortfall to report_incomplete identifying every affected work item', () => { + const reconciliation = activateSkill.slice(activateSkill.indexOf('**2e. Label-Operation Reconciliation')); + expect(reconciliation).toContain('report_incomplete'); + expect( + /issue number, its title, and the label set it should have received/i.test( + reconciliation.replace(/\s+/g, ' '), + ), + 'The incomplete report must identify affected work items specifically enough to ' + + 'act on.', + ).toBe(true); + }); + + it('states why report_incomplete is the mechanism that prevents a green truncated run', () => { + // gh-aw's report_incomplete is a first-class failure signal: handle_agent_failure + // activates failure handling even when the agent exits 0. That property is the + // whole reason this is the right primitive. + expect( + /failure signal even when the agent exits successfully/i.test(activateProse), + 'The workflow must record why report_incomplete (not noop, not a comment) is the ' + + 'primitive that stops a truncated run being recorded as clean.', + ).toBe(true); + }); + + it('forbids emitting a clean activation artifact when labels are missing', () => { + expect( + /false success report/i.test(activateProse), + 'Emitting an activated/phases-activated artifact while labeled < activated must be ' + + 'named as a false success report.', + ).toBe(true); + expect(activateProse).toMatch(/phases-activated.*artifact.*while.*labeled < activated|activated.*artifact/i); + }); + + it('still forbids blaming a cap that was never actually reached', () => { + // Guard against over-correction: #1683's rule (do not excuse a partial run by + // pointing at a cap) must survive, now narrowed to *guessed* causes. + expect( + /Never surface the `create-issue` or `add-comment` safe-output caps as a guessed reason/i.test( + activateProse, + ), + 'A cap may be named only when observed (Step 2e), never as a guess.', + ).toBe(true); + }); +}); + +// --------------------------------------------------------------------------- +// Compiled-artifact verification — runtime contract evidence, not prose. +// +// Fails closed (never skips) per this repo's #1833/#1834 convention: an +// unmeasured contract is indistinguishable from a violated one. +// --------------------------------------------------------------------------- + +const GH_AW_INSTALL_HINT = + '`gh aw` is required to compile the workflow this gate inspects. Install it with ' + + '`gh extension install --pin v0.87.10 github/gh-aw` (matches .github/workflows/squad-ci.yml). ' + + 'This gate fails closed rather than skipping: an unmeasured contract is ' + + 'indistinguishable from a violated one (#1834).'; + +describe('gh-aw: compiled runtime proves the capacity contract (#1961)', () => { + let compiledLock: string | null = null; + + function lockText(): string { + if (compiledLock !== null) return compiledLock; + + const versionProbe = spawnSync('gh', ['aw', '--version'], { encoding: 'utf8' }); + if (versionProbe.status !== 0) { + throw new Error(`gh aw --version failed. ${GH_AW_INSTALL_HINT}`); + } + + mkdirSync(TEST_WORKSPACES_DIR, { recursive: true }); + const workspace = mkdtempSync(join(TEST_WORKSPACES_DIR, 'capacity-')); + execFileSync('git', ['init', '--quiet'], { cwd: workspace }); + // Mirror the real deployment layout: this repo ships gh-aw sources from a + // top-level `workflows/` dir; consumers install them into `.github/workflows/`. + cpSync(WORKFLOWS_DIR, join(workspace, '.github', 'workflows'), { recursive: true }); + execFileSync('gh', ['aw', 'compile', '.github/workflows/squad.md', '--strict', '--approve', '--no-check-update'], { + cwd: workspace, + encoding: 'utf8', + stdio: 'pipe', + }); + + const lockPath = join(workspace, '.github', 'workflows', 'squad.lock.yml'); + if (!existsSync(lockPath)) { + throw new Error( + `gh aw compile produced no squad.lock.yml — the contract is unmeasured. ${GH_AW_INSTALL_HINT}`, + ); + } + compiledLock = readText(lockPath); + return compiledLock; + } + + function safeOutputsConfig(): Record> { + const json = extractSafeOutputsConfigJson(lockText()); + expect(json, 'compiled lock must carry the safe-outputs config').toBeDefined(); + return JSON.parse(json as string); + } + + it('strict-compiles with the capacity configuration in place', () => { + expect(lockText().length).toBeGreaterThan(0); + }); + + it('compiles the declared add_labels max into the runtime config (not just frontmatter prose)', () => { + const config = safeOutputsConfig(); + expect(config['add_labels']).toBeDefined(); + expect(config['add_labels']['max']).toBe(frontmatterMax('add-labels')); + // The number that actually reaches the runtime must cover the worst case. + expect(config['add_labels']['max'] as number).toBeGreaterThanOrEqual( + documentedMaxIssues() * MAX_LABELS_PER_ISSUE, + ); + }); + + it('keeps create_issue capacity unchanged at its established value', () => { + const config = safeOutputsConfig(); + expect(config['create_issue']['max']).toBe(75); + }); + + it("injects a tool constraint whose number matches the configured cap", () => { + // gh-aw phrases this as "Maximum {max} label(s) can be added". The workflow's + // rationale quotes that phrasing; if gh-aw's number and the configured cap ever + // diverge, the rationale would be describing a budget that does not exist. + const compiled = lockText().replace(/\\"/g, '"'); + const declared = frontmatterMax('add-labels'); + expect(compiled).toContain(`CONSTRAINTS: Maximum ${declared} label(s) can be added`); + }); + + it('enables report_incomplete so the overflow signal is actually callable', () => { + // Step 2e is unenforceable if this safe output is not wired in: gh-aw builds the + // agent's tool list by intersecting the config keys with its known tools, so the + // key's presence here is what makes report_incomplete available at runtime. + const config = safeOutputsConfig(); + expect( + Object.prototype.hasOwnProperty.call(config, 'report_incomplete'), + 'report_incomplete must be present in the compiled safe-outputs config — Step 2e ' + + 'depends on it to convert a truncated activation into a failure signal.', + ).toBe(true); + }); + + it('exposes add_labels and create_issue as tools the agent can call', () => { + const compiled = lockText(); + expect(compiled).toMatch(/"tools":\[[^\]]*"add_labels"[^\]]*\]/); + expect(compiled).toMatch(/"tools":\[[^\]]*"create_issue"[^\]]*\]/); + }); + + it('does not broaden the agent job permissions — writes stay in the safe-output job', () => { + // Raising a capacity cap must not become a permissions change. + const compiled = lockText(); + const agentJobAt = compiled.indexOf('\n agent:\n'); + const safeOutputsJobAt = compiled.indexOf('\n safe_outputs:\n'); + expect(agentJobAt, '"agent:" job missing from compiled lock').toBeGreaterThan(-1); + expect(safeOutputsJobAt, '"safe_outputs:" job missing from compiled lock').toBeGreaterThan(agentJobAt); + + const permsBlock = compiled.slice(agentJobAt, safeOutputsJobAt).match(/permissions:\n((?:\s{6}\S.*\n)+)/)?.[1] ?? ''; + expect(permsBlock).toContain('issues: read'); + expect(permsBlock).not.toContain('issues: write'); + }); +}); diff --git a/workflows/squad.md b/workflows/squad.md index 407c48611..fade42f74 100644 --- a/workflows/squad.md +++ b/workflows/squad.md @@ -102,8 +102,13 @@ safe-outputs: protected-files: allowed max-patch-files: 500 expires: 14d + # Capacity: largest supported activation is 50 issues. `max` counts safe-output + # ITEMS (tool calls), not label names within a call, and an over-limit item is + # dropped with only a warning. Derivation: `squad-plan-activate` > + # "Activation capacity budget". create-issue: labels: [squad] + # 50 worst-case issues + 25 bounded margin. max: 75 add-labels: allowed: [squad, "squad:*"] @@ -111,7 +116,9 @@ safe-outputs: issues: true pull-requests: false target: "*" - max: 80 + # 50 calls worst case (one per issue); 100 label names worst case (2 each). + # 110 covers both readings of `max`. Do not reduce below create-issue max. + max: 110 add-comment: max: 20 target: "*" @@ -1742,6 +1749,33 @@ After EVERY `create-issue` call: verify returned issue number, stop on failure, Count expected issues before starting. If total > 50: recommend phased activation (`/squad plan activate phase {N}`) and proceed with the current phase only. If total > 30: use compact issue bodies (scope + acceptance criteria only; omit elaboration). +**Activation capacity budget.** The largest activation supported in a single run is +**50 issues** — the `enterprise` profile's `max_issues: 50` (the highest documented +profile limit) and the same threshold the phased-activation rule above enforces. Worst +case for an activation of that size: + +| Safe output | Worst case at 50 issues | Configured `max` | Bounded margin | +|---|---|---|---| +| `create-issue` | 50 items — one per epic/task | 75 | 25 | +| `add_labels` | 50 items — one per created issue | 110 | 60 | +| Labels *within* one `add_labels` call | 2 — `squad` + one `squad:{agent}` | not governed by `max` | — | +| Label names across the whole run | 100 — 50 × 2 | 110 | 10 | + +**`max` limits safe-output items (tool calls), not label names inside a call.** gh-aw's +collector counts NDJSON items per type and rejects the item that would exceed the cap +(`collect_ndjson_output.cjs`: `Too many items of type '{type}'. Maximum allowed: {max}.`). +One `add_labels` call carrying two labels therefore consumes **one** item, not two. +gh-aw's injected tool constraint phrases the same number as "Maximum 110 label(s) can be +added"; `max: 110` is deliberately sized to cover the worst case under **both** readings — +50 calls and 100 label names — so no interpretation of the cap can justify skipping a +label operation. Never batch several issues' labels into one call to conserve budget +(it breaks the per-issue correspondence rule), and never stop labeling early because the +budget looks tight: it is sized for the full documented maximum. + +**An over-limit item is dropped, not failed.** gh-aw logs a rejected item as a warning and +continues, so a run can finish green with label operations missing. Cap overflow does not +announce itself — Step 2e is the only thing that detects it. + ##### Label Pre-flight **Roster binding gate — run this before any `create-issue` call.** @@ -1835,7 +1869,18 @@ Root → Epics → Tasks. Phase-specific: filter to matching phase heading. - Size: Project field if available, else body line - Label application: same as epics — call `add_labels` on the verified task issue number with this task's computed label set (see Label Pre-flight). `create-if-missing` provisions `squad`/`squad:{agent}` on a fresh repository automatically. -**2d. Self-Validation:** Compare created/recognized task count vs expected (use the plan's declared total — not the safe-output cap). If created count is below expected: call `report_incomplete` immediately with `created={N}`, `expected={M}`, and the last verified issue number — never noop. Post: `N of M issues created so far — rerun the identical activation command to continue.` Re-runs are idempotent via title match. Never surface the `create-issue` or `add-comment` safe-output caps as the reason for a partial run. +**2d. Self-Validation:** Compare created/recognized task count vs expected (use the plan's declared total — not the safe-output cap). If created count is below expected: call `report_incomplete` immediately with `created={N}`, `expected={M}`, and the last verified issue number — never noop. Post: `N of M issues created so far — rerun the identical activation command to continue.` Re-runs are idempotent via title match. Never surface the `create-issue` or `add-comment` safe-output caps as a guessed reason for a partial run — name a cap only when Step 2e observed one actually being reached. + +**2e. Label-Operation Reconciliation — an activation is not complete until every activated item's labels are accounted for.** + +While Steps 2b/2c run, keep two counts: `activated` (issues created or recognized this run) and `labeled` (issues whose `add_labels` call returned successfully). An `add_labels` call that was never made, was rejected, or returned an error counts as **unlabeled**. At the end of Step 2, compare them: + +1. `labeled == activated` → the activation is complete; proceed to Step 3. +2. `labeled < activated` → **this is an incomplete activation, not a successful one.** Call `report_incomplete` with a `reason` naming the shortfall (`{labeled} of {activated} activated issues received their labels`) and `details` listing **every affected work item — its issue number, its title, and the label set it should have received**. gh-aw treats `report_incomplete` as a failure signal even when the agent exits successfully, so this is what stops a truncated run from being recorded as a clean one. + +**Cap exhaustion is a reportable, nameable cause.** A safe-output item rejected for exceeding `max` is dropped with only a warning — it never fails the run on its own and it never appears as an error to the agent. If the Step 2e shortfall is because a cap was reached, say so explicitly in the `report_incomplete` `reason`, name which cap (`create-issue` 75 or `add_labels` 110) and list the work items that did not fit, and recommend `/squad plan activate phase {N}`. This is the one case where a cap may be named as the cause — because it was observed here, not guessed. + +**Never report a clean activation you did not perform.** Emitting an `activated` or `phases-activated` artifact that lists every item as activated while `labeled < activated`, or that omits the shortfall entirely, is a false success report — precisely the silent truncation this reconciliation exists to prevent. Labels must have descriptions and intentional colors when they already exist in the repository. A label auto-provisioned by `add-labels`'s `create-if-missing` on a fresh From ef5f2de22857a26d00d1865f28a1fe07be92956c Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 31 Aug 2026 15:20:11 -0700 Subject: [PATCH 2/5] fix(gh-aw): correct runtime claims in activation capacity safeguards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independent review found three factual defects in the prose and tests shipped by the first commit. All three are corrected against the pinned gh-aw v0.87.10 sources rather than gh-aw's own injected descriptions. 1. `report_incomplete` does NOT make the run conclude non-successfully. `report_incomplete_handler.cjs` emits `core.warning` only, and `handle_agent_failure.cjs` contains zero `setFailed`/`process.exit` calls in 4453 lines. What it actually does is open or update a durable `[aw] {workflow} reported incomplete result` tracking issue. Step 2e now states that, says the run still reports success, and tells the agent never to rely on a red run to carry the signal. gh-aw's own tool description ("treated as a failure signal even when the agent exits successfully") is misleading; trusting it was exactly the injected-prose mistake #1961 exists to prevent. Limitation, stated honestly: no narrow mechanism in the pinned runtime makes an incomplete activation conclude red. Forcing one would require a custom safe-job, which is outside this issue's scope. The durable tracking record plus the "never report a clean activation you did not perform" rule are what carry the signal. 2. Cap enforcement is dual, not collection-only. Per Safe Outputs MCE4, `enforcePerTypeMax` in `safe_outputs_handlers.cjs` throws JSON-RPC `E002: {type} limit reached` at invocation time, which the agent DOES see; the collector then drops surplus items with a warning. Removed the claim that an over-limit item "never appears as an error to the agent". Reconciliation still keys on absence-of-success, not presence-of-error, because a call can also simply never be made. 3. The old `max: 80` did not cause runtime truncation — 50 calls never reached it. The real hazard is gh-aw's injected "Maximum 80 label(s) can be added" wording against a 100-label worst case, which invites agent self-truncation. 110 is justified by that hazard plus bounded margin, and the docblock now explicitly records what is NOT claimed. Also, per the PR #1965 review: Step 2e no longer demands a real issue number for items created during the run. Issue creation is deferred to the post-agent job, so it names the stable temporary ID plus title and intended labels, and quotes a real number only for a reused issue — avoiding reintroduction of #1962's invalid-number assumption. Tests: renamed the two overclaiming tests and gave each a negative guard that fails if the retracted wording returns; added coverage for the temporary-ID rule. Trimmed prose to stay under the existing 160 KB source-growth guard rather than raising another test's budget. Verified: 221/221 across the three gh-aw suites; mutation-checked (cap -> 80 fails 5 tests incl. the compiled-runtime one, reintroducing either retracted claim fails its guard); strict compile clean on all four workflows; npm run build passes. Closes #1961 Parent #1957 Working as FIDO (Quality Owner) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2471face-3b27-4419-bd48-0becd2642d7f --- test/gh-aw-activation-capacity.test.ts | 175 ++++++++++++++++++------- workflows/squad.md | 62 ++++----- 2 files changed, 157 insertions(+), 80 deletions(-) diff --git a/test/gh-aw-activation-capacity.test.ts b/test/gh-aw-activation-capacity.test.ts index a0559c1ed..264b32a56 100644 --- a/test/gh-aw-activation-capacity.test.ts +++ b/test/gh-aw-activation-capacity.test.ts @@ -5,35 +5,53 @@ * * `workflows/squad.md` promises a `squad` (and where certified, `squad:{agent}`) label on * every issue an activation creates. Those labels are applied by the `add-labels` safe - * output, which is governed by a `max`. Two independent problems made a large activation - * able to lose label operations *while still reporting a clean success*: + * output, which is governed by a `max`. Two problems let a large activation lose label + * operations *while the run still reported success*: * * 1. **`max` is ambiguous in gh-aw's own surface area.** The compiler injects the tool * constraint as "Maximum {N} label(s) can be added", which reads as a budget of label - * *names*. The runtime enforces something different — see below. An agent that believed - * the label reading would conclude a 50-issue activation (up to 100 label names) had - * overrun its budget and could stop labeling early or batch issues together. - * 2. **An over-limit item is dropped, not failed.** gh-aw v0.87.10's collector - * (`actions/setup/js/collect_ndjson_output.cjs`) rejects the offending item and - * `continue`s, pushing a string into `errors`. Those errors are emitted with - * `core.warning(...)`, never `core.setFailed(...)`. The run finishes green with label - * operations missing. Nothing in the run announces the truncation. + * *names*. The runtime counts operations instead. At the previous `max: 80`, an agent + * that believed the label reading would conclude a 50-issue activation (up to 100 label + * names) had overrun its budget, and could stop labeling early or batch issues together. + * This is agent self-truncation, and it was reachable at 80. + * 2. **Neither enforcement layer fails the run.** Even when a cap is genuinely hit, the + * workflow run still concludes successfully, so nothing forces the truncation to + * surface. See below. + * + * Note what is *not* claimed: at `max: 80`, 50 `add_labels` calls did **not** overflow the + * operation cap. Runtime truncation was not reachable at the documented maximum. The cap + * moves to 110 to defeat the misleading injected wording and to hold a bounded margin — + * not to fix a proven 80-item overflow. * * ## Runtime semantics this suite is written against (gh-aw v0.87.10, the CI pin) * - * `collect_ndjson_output.cjs`: + * **Cap enforcement is dual (Safe Outputs Specification MCE4) and neither half is fatal.** + * Invocation time — `safe_outputs_handlers.cjs`, `enforcePerTypeMax` via + * `appendSafeOutputCounted`, applied whenever `max` is explicitly configured (it is here): * * ```js - * const typeCount = parsedItems.filter(existing => existing.type === itemType).length; - * const maxAllowed = getMaxAllowedForType(itemType, expectedOutputTypes); - * if (typeCount >= maxAllowed) { - * errors.push(`Line ${i + 1}: Too many items of type '${itemType}'. Maximum allowed: ${maxAllowed}.`); - * continue; // <- the item is DROPPED + * if (current >= maxAllowed) { + * throw { code: -32602, message: `E002: ${type} limit reached — ${current} of ${maxAllowed} already used this run`, ... }; * } * ``` * - * `max` therefore caps **safe-output items (tool calls) of that type**, not label names - * inside one call. One `add_labels` call carrying two labels costs one item, not two. + * That is a JSON-RPC error the agent **does** observe. Collection time — + * `collect_ndjson_output.cjs` — is the second half: a surplus item is dropped with + * `continue` and reported via `core.warning`. Neither path calls `core.setFailed`, so the + * run concludes successfully with label operations missing. + * + * `max` therefore caps **operations of that type**, not label names inside one call. One + * `add_labels` call carrying two labels costs one unit of budget, not two. + * + * **`report_incomplete` does not turn the run red.** gh-aw's own tool description claims it + * is "treated as a failure signal even when the agent exits successfully" — misleading in + * precisely the way this issue is about. In the pinned runtime, + * `report_incomplete_handler.cjs` emits `core.warning` only, and `handle_agent_failure.cjs` + * contains no `core.setFailed` or `process.exit`: its "failure handling" opens or updates an + * `[aw] {workflow} reported incomplete result` tracking issue/comment. That is a durable, + * human-actionable record — it satisfies #1961's "explicit incomplete result" — but the run + * conclusion stays green. No narrower supported mechanism in v0.87.10 makes it red, so the + * workflow states that limitation rather than implying a failure it cannot produce. * * ## The capacity calculation this suite locks in * @@ -55,9 +73,11 @@ * * ## Out of scope (sibling issues — do not broaden these assertions here) * - * - #1962 temporary-ID linkage between `create_issue` and `add_labels`. This suite is - * deliberately neutral about *how* an `add_labels` call identifies its target, so it - * does not conflict with that change. + * - #1962 / PR #1965 temporary-ID linkage between `create_issue` and `add_labels`. This + * suite stays neutral about *how* an `add_labels` call identifies its target. It does + * assert that Step 2e's incomplete report names a temporary ID rather than a real issue + * number for items created during the run, because demanding a number that does not yet + * exist would reintroduce the very assumption #1962 removes. * - #1959 `/squad activate` fast-path parity. * - #1963 label-result reporting beyond the explicit overflow/incomplete signal. * - #1960 broad safe-output contract coverage; #1958 E4 evidence. @@ -161,21 +181,40 @@ describe('gh-aw: activation capacity is calculated, recorded, and bounded (#1961 'calls) rather than label names within a call.', ).toBe(true); expect( - /consumes \*\*one\*\* item, not two/i.test(activateProse) || /one\*\* item, not two/i.test(activateProse), - 'The workflow must show that a two-label call costs one item, so the agent cannot ' + - 'conclude it must ration labels.', + /consumes \*\*one\*\* unit\s*of budget, not two/i.test(activateProse), + 'The workflow must show that a two-label call costs one unit of budget, so the ' + + 'agent cannot conclude it must ration labels.', ).toBe(true); }); - it('warns that an over-limit item is dropped with a warning rather than failing the run', () => { - // gh-aw's collector pushes a string into `errors` and the job emits it via - // core.warning — never core.setFailed. Truncation is invisible unless the - // workflow itself detects it. + it('describes cap enforcement as dual and non-fatal, without overstating either half', () => { + // Pinned-runtime semantics, both halves: + // invocation time — safe_outputs_handlers.cjs enforcePerTypeMax/appendSafeOutputCounted + // throw JSON-RPC -32602 "E002: {type} limit reached", which the + // agent DOES observe; + // collection time — collect_ndjson_output.cjs drops the surplus item via core.warning. + // Neither calls core.setFailed, so the run still concludes successfully. + expect( + /enforced twice/i.test(activateProse), + 'The workflow must describe both halves of cap enforcement rather than only the ' + + 'collector drop.', + ).toBe(true); + expect( + /E002/.test(activateSkill), + 'The invocation-time rejection must be named by its actual error, since the agent ' + + 'can observe it.', + ).toBe(true); expect( - /dropped, not failed/i.test(activateProse), - 'The workflow must state that an over-limit safe-output item is dropped rather ' + - 'than failing the run — this is why self-validation is mandatory.', + /neither layer fails the run/i.test(activateProse), + 'The workflow must state that neither enforcement layer fails the run — that is ' + + 'why self-validation is mandatory.', ).toBe(true); + // Guard against the inverse overclaim: the agent is NOT blind to cap rejections. + expect( + /never appears as an error to the agent/i.test(activateProse), + 'The workflow must not claim the agent never sees a cap error — invocation-time ' + + 'enforcement surfaces E002 to the agent.', + ).toBe(false); expect(activateProse).toMatch(/finish green|still succeeds/i); }); }); @@ -192,7 +231,7 @@ describe('gh-aw: configured caps cover the worst case under both readings of max }); it('add-labels covers one CALL per activated issue (the runtime reading)', () => { - // Runtime semantics: max caps NDJSON items of type add_labels. + // Runtime semantics: max caps add_labels operations, not label names. expect(addLabelsMax).toBeGreaterThanOrEqual(maxIssues); }); @@ -300,36 +339,78 @@ describe('gh-aw: self-validation reconciles activated items with label operation }); it('counts a missing, rejected, or errored label call as unlabeled', () => { - // A dropped over-limit item produces no error the agent can see, so "no - // successful result" — not "an observed error" — must be the failing condition. + // Absence-of-success, not presence-of-error, must be the failing condition. The agent + // does see an E002 rejection at invocation time, but a call can also simply never be + // made, and a surplus item can be dropped downstream with only a warning. Reconciling + // on counts covers all three; reconciling on observed errors does not. expect( /never made, was rejected, or returned an error counts as \*\*unlabeled\*\*/i.test(activateProse), - 'A label call that was never made must count as unlabeled, otherwise a dropped ' + - 'item is invisible to the reconciliation.', + 'A label call that was never made must count as unlabeled, otherwise a lost ' + + 'operation is invisible to the reconciliation.', ).toBe(true); }); it('escalates a shortfall to report_incomplete identifying every affected work item', () => { const reconciliation = activateSkill.slice(activateSkill.indexOf('**2e. Label-Operation Reconciliation')); + const reconciliationProse = reconciliation.replace(/\s+/g, ' '); expect(reconciliation).toContain('report_incomplete'); expect( - /issue number, its title, and the label set it should have received/i.test( - reconciliation.replace(/\s+/g, ' '), + /the identifier you used to target its `add_labels` call, its title, and the label set it should have received/i.test( + reconciliationProse, ), - 'The incomplete report must identify affected work items specifically enough to ' + - 'act on.', + 'The incomplete report must identify affected work items specifically enough to act on.', ).toBe(true); }); - it('states why report_incomplete is the mechanism that prevents a green truncated run', () => { - // gh-aw's report_incomplete is a first-class failure signal: handle_agent_failure - // activates failure handling even when the agent exits 0. That property is the - // whole reason this is the right primitive. + it('does not demand a real issue number for items created during the run', () => { + // gh-aw defers issue creation to the post-agent safe-output job, so no real number + // exists during the agent turn. Requiring one here would reintroduce exactly the + // invalid-number assumption #1962 exists to remove. + const reconciliation = activateSkill.slice(activateSkill.indexOf('**2e. Label-Operation Reconciliation')); + const reconciliationProse = reconciliation.replace(/\s+/g, ' '); expect( - /failure signal even when the agent exits successfully/i.test(activateProse), - 'The workflow must record why report_incomplete (not noop, not a comment) is the ' + - 'primitive that stops a truncated run being recorded as clean.', + /stable temporary ID, not a GitHub issue number/i.test(reconciliationProse), + 'For an item created this run the report must name its temporary ID, because no ' + + 'issue number exists yet.', + ).toBe(true); + expect( + /already existed and was recognized rather than created/i.test(reconciliationProse), + 'A real issue number may only be quoted for a pre-existing, recognized item.', ).toBe(true); + expect( + /Never predict, infer, or invent a number/i.test(reconciliationProse), + 'The report must forbid inventing an issue number to fill the identifier field.', + ).toBe(true); + }); + + it('states report_incomplete semantics accurately — a tracking record, not a red run', () => { + // Verified against pinned gh-aw v0.87.10 rather than gh-aw's own tool description + // (which says "treated as a failure signal even when the agent exits successfully" + // — misleading in exactly the way this issue is about): + // report_incomplete_handler.cjs -> core.warning only. + // handle_agent_failure.cjs -> contains no core.setFailed / process.exit; + // "failure handling" opens/updates an + // "[aw] ... reported incomplete result" issue. + // The run still concludes successfully. The workflow must say so, so the agent does + // not assume a red run is carrying the signal for it. + expect( + /reported incomplete result/i.test(activateProse), + 'The workflow must name the durable artifact report_incomplete actually produces.', + ).toBe(true); + expect( + /does \*\*not\*\* change the run's conclusion/i.test(activateProse), + 'The workflow must state that report_incomplete does not change the run conclusion.', + ).toBe(true); + expect( + /never rely on a red run/i.test(activateProse), + 'The workflow must forbid relying on a failed run to carry the incompletion signal.', + ).toBe(true); + // Guard against reintroducing the overclaim this test previously asserted. + expect( + /failure signal even when the agent exits successfully/i.test(activateProse), + 'The workflow must not repeat gh-aw\'s misleading "failure signal" phrasing: the ' + + 'pinned runtime emits a warning and a tracking issue, and never fails the run.', + ).toBe(false); }); it('forbids emitting a clean activation artifact when labels are missing', () => { diff --git a/workflows/squad.md b/workflows/squad.md index fade42f74..f85a738ac 100644 --- a/workflows/squad.md +++ b/workflows/squad.md @@ -102,10 +102,9 @@ safe-outputs: protected-files: allowed max-patch-files: 500 expires: 14d - # Capacity: largest supported activation is 50 issues. `max` counts safe-output - # ITEMS (tool calls), not label names within a call, and an over-limit item is - # dropped with only a warning. Derivation: `squad-plan-activate` > - # "Activation capacity budget". + # Capacity: max activation is 50 issues. `max` counts safe-output ITEMS (tool + # calls), not label names; enforced at invocation and collection, neither fails + # the run. Derivation: `squad-plan-activate` > "Activation capacity budget". create-issue: labels: [squad] # 50 worst-case issues + 25 bounded margin. @@ -1751,30 +1750,27 @@ Count expected issues before starting. If total > 50: recommend phased activatio **Activation capacity budget.** The largest activation supported in a single run is **50 issues** — the `enterprise` profile's `max_issues: 50` (the highest documented -profile limit) and the same threshold the phased-activation rule above enforces. Worst -case for an activation of that size: - -| Safe output | Worst case at 50 issues | Configured `max` | Bounded margin | -|---|---|---|---| -| `create-issue` | 50 items — one per epic/task | 75 | 25 | -| `add_labels` | 50 items — one per created issue | 110 | 60 | -| Labels *within* one `add_labels` call | 2 — `squad` + one `squad:{agent}` | not governed by `max` | — | -| Label names across the whole run | 100 — 50 × 2 | 110 | 10 | - -**`max` limits safe-output items (tool calls), not label names inside a call.** gh-aw's -collector counts NDJSON items per type and rejects the item that would exceed the cap -(`collect_ndjson_output.cjs`: `Too many items of type '{type}'. Maximum allowed: {max}.`). -One `add_labels` call carrying two labels therefore consumes **one** item, not two. -gh-aw's injected tool constraint phrases the same number as "Maximum 110 label(s) can be -added"; `max: 110` is deliberately sized to cover the worst case under **both** readings — -50 calls and 100 label names — so no interpretation of the cap can justify skipping a -label operation. Never batch several issues' labels into one call to conserve budget -(it breaks the per-issue correspondence rule), and never stop labeling early because the -budget looks tight: it is sized for the full documented maximum. - -**An over-limit item is dropped, not failed.** gh-aw logs a rejected item as a warning and -continues, so a run can finish green with label operations missing. Cap overflow does not -announce itself — Step 2e is the only thing that detects it. +limit) and the threshold the phased-activation rule above enforces. Worst case: + +| Safe output | Worst case | `max` | +|---|---|---| +| `create-issue` | 50 — one per epic/task | 75 | +| `add_labels` | 50 — one per created issue | 110 | +| Labels in one call | 2 — `squad` + `squad:{agent}` | not capped | +| Label names per run | 100 — 50 × 2 | 110 | + +**`max` limits safe-output items (tool calls), not label names inside a call.** One +`add_labels` call carrying two labels consumes **one** unit of budget, not two. gh-aw's +injected constraint still phrases that number as "Maximum 110 label(s) can be added", +which reads as a budget of *names*. That wording is the hazard 110 is sized against: it covers both readings — 50 calls and 100 names — so neither can justify +skipping a label operation. Never batch several issues' labels into one call to save +budget (it breaks per-issue correspondence), and never stop labeling early. + +**Reaching a cap is enforced twice, and neither layer fails the run.** A call past the +limit is rejected at invocation time with a JSON-RPC error (`E002: {type} limit +reached`) the agent *does* see; a surplus item is dropped at collection time with a +warning. Neither marks the run failed, so it can finish green with labels missing — +Step 2e, not the cap machinery, is what notices. ##### Label Pre-flight @@ -1871,16 +1867,16 @@ Root → Epics → Tasks. Phase-specific: filter to matching phase heading. **2d. Self-Validation:** Compare created/recognized task count vs expected (use the plan's declared total — not the safe-output cap). If created count is below expected: call `report_incomplete` immediately with `created={N}`, `expected={M}`, and the last verified issue number — never noop. Post: `N of M issues created so far — rerun the identical activation command to continue.` Re-runs are idempotent via title match. Never surface the `create-issue` or `add-comment` safe-output caps as a guessed reason for a partial run — name a cap only when Step 2e observed one actually being reached. -**2e. Label-Operation Reconciliation — an activation is not complete until every activated item's labels are accounted for.** +**2e. Label-Operation Reconciliation — no activation is complete until every activated item's labels are accounted for.** -While Steps 2b/2c run, keep two counts: `activated` (issues created or recognized this run) and `labeled` (issues whose `add_labels` call returned successfully). An `add_labels` call that was never made, was rejected, or returned an error counts as **unlabeled**. At the end of Step 2, compare them: +While Steps 2b/2c run, keep two counts: `activated` (issues created or recognized this run) and `labeled` (issues whose `add_labels` call returned successfully). An `add_labels` call that was never made, was rejected, or returned an error counts as **unlabeled**. At the end of Step 2: 1. `labeled == activated` → the activation is complete; proceed to Step 3. -2. `labeled < activated` → **this is an incomplete activation, not a successful one.** Call `report_incomplete` with a `reason` naming the shortfall (`{labeled} of {activated} activated issues received their labels`) and `details` listing **every affected work item — its issue number, its title, and the label set it should have received**. gh-aw treats `report_incomplete` as a failure signal even when the agent exits successfully, so this is what stops a truncated run from being recorded as a clean one. +2. `labeled < activated` → **this is an incomplete activation, not a successful one.** Call `report_incomplete` with a `reason` naming the shortfall (`{labeled} of {activated} activated issues received their labels`) and `details` listing **every affected work item — the identifier you used to target its `add_labels` call, its title, and the label set it should have received**. For an item created during this run that identifier is its stable temporary ID, not a GitHub issue number: issue creation is deferred to the post-agent job, so no real number exists yet. Quote a real number only for an item that already existed and was recognized rather than created. Never predict, infer, or invent a number. `report_incomplete` logs a warning and opens or updates a durable `[aw] ... reported incomplete result` tracking issue; it does **not** change the run's conclusion — the run still reports success. That record and the rule below keep a truncated activation from passing as clean, so never rely on a red run to carry the signal. -**Cap exhaustion is a reportable, nameable cause.** A safe-output item rejected for exceeding `max` is dropped with only a warning — it never fails the run on its own and it never appears as an error to the agent. If the Step 2e shortfall is because a cap was reached, say so explicitly in the `report_incomplete` `reason`, name which cap (`create-issue` 75 or `add_labels` 110) and list the work items that did not fit, and recommend `/squad plan activate phase {N}`. This is the one case where a cap may be named as the cause — because it was observed here, not guessed. +**Cap exhaustion is a reportable, nameable cause.** If the shortfall is because a cap was reached, say so in the `reason`, name which cap (`create-issue` 75 or `add_labels` 110) and list the work items that did not fit, and recommend `/squad plan activate phase {N}`. This is the one case where a cap may be named as the cause: it was observed, not guessed. Do not infer a cap from a rejection you never received, and do not treat the absence of an `E002` error as proof that every label landed: the count comparison, not the error stream, is the authority. -**Never report a clean activation you did not perform.** Emitting an `activated` or `phases-activated` artifact that lists every item as activated while `labeled < activated`, or that omits the shortfall entirely, is a false success report — precisely the silent truncation this reconciliation exists to prevent. +**Never report a clean activation you did not perform.** An `activated` or `phases-activated` artifact listing every item as activated while `labeled < activated`, or omitting the shortfall, is a false success report — the silent truncation this step exists to prevent. Labels must have descriptions and intentional colors when they already exist in the repository. A label auto-provisioned by `add-labels`'s `create-if-missing` on a fresh From 015b4fc07fa8e4bf87dc822215155c5eaae34fa8 Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Mon, 31 Aug 2026 15:53:27 -0700 Subject: [PATCH 3/5] test(gh-aw): use explicit .js extension on helper import Addresses Copilot review on PR #1964. The reviewer is correct on both counts. `tsc --module nodenext --moduleResolution nodenext` reports exactly one error on this file, and TypeScript names the fix itself: test/gh-aw-activation-capacity.test.ts(94,46): error TS2835: Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean './helpers/gh-aw-lock.js'? The repo is `"type": "module"` with `module`/`moduleResolution: NodeNext` in the root tsconfig, and the four other suites importing this helper already use the `.js` specifier. This file was the lone outlier; it now matches. Verified: 293/293 across all eight suites that import the helper, all four workflows compile with `--strict` (pre-existing warnings only), `npm run build` passes, and the NodeNext type-check on this file is now clean. Refs #1961. Parent #1957. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2471face-3b27-4419-bd48-0becd2642d7f --- test/gh-aw-activation-capacity.test.ts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/gh-aw-activation-capacity.test.ts b/test/gh-aw-activation-capacity.test.ts index f47ce870f..b6c26064f 100644 --- a/test/gh-aw-activation-capacity.test.ts +++ b/test/gh-aw-activation-capacity.test.ts @@ -91,7 +91,7 @@ import { afterAll, describe, expect, it } from 'vitest'; import { cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; import { execFileSync, spawnSync } from 'node:child_process'; import { join } from 'node:path'; -import { extractSafeOutputsConfigJson } from './helpers/gh-aw-lock'; +import { extractSafeOutputsConfigJson } from './helpers/gh-aw-lock.js'; const WORKFLOWS_DIR = join(process.cwd(), 'workflows'); const SQUAD_WORKFLOW = join(WORKFLOWS_DIR, 'squad.md'); From 4ec89dded9c722c9c60dd4561e7347ea3d850cdb Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Tue, 1 Sep 2026 09:01:23 -0700 Subject: [PATCH 4/5] fix(workflows): state Step 2e counts as accepted operations, not applied labels Copilot review flagged that Step 2e's reconciliation claimed issues "received their labels". #1963 (now merged to dev) settled the vocabulary: an accepted safe output is queued this turn and applied by the post-agent job, so the agent has evidence only that a call was accepted for a specific target -- never that a label reached GitHub. Step 2e was the remaining place still asserting application, which conflicts with the accepted-vs-applied rule the same skill now states. - `labeled` is defined as issues whose add_labels call was accepted. - The report_incomplete reason reads "had a label operation accepted". - "proof that every label landed" -> "every label operation was accepted". - Added an explicit statement that the counts track label operations, not labels present on GitHub. Tests: new assertion guards the positive and both retired over-claim phrasings; verified by reverting the prose (1 failed, 28 passed) rather than assuming. Tightened the sibling assertion to match the exact count definitions -- the previous `.*`-joined form matched a distant "was accepted" elsewhere in the flattened skill and so failed to detect a reverted definition. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0515ad5d-4c5b-48bb-92a6-4e1296594cab --- test/gh-aw-activation-capacity.test.ts | 19 ++++++++++++++++--- workflows/squad.md | 6 +++--- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/test/gh-aw-activation-capacity.test.ts b/test/gh-aw-activation-capacity.test.ts index b6c26064f..6c337e58a 100644 --- a/test/gh-aw-activation-capacity.test.ts +++ b/test/gh-aw-activation-capacity.test.ts @@ -333,15 +333,28 @@ describe('gh-aw: self-validation reconciles activated items with label operation ).toBe(true); }); - it('compares expected activated items against successful label operations', () => { - expect(activateProse).toMatch(/`activated`.*issues created or recognized/i); - expect(activateProse).toMatch(/`labeled`.*add_labels.*returned successfully/i); + it('compares expected activated items against accepted label operations', () => { + // Anchored to the exact definitions rather than `.*`-joined fragments: activateProse is + // the whole flattened skill, so a loose pattern can satisfy itself with wording from a + // different section (verified -- a distant "was accepted" masked a reverted definition). + expect(activateProse).toMatch(/`activated` \(issues created or recognized this run\)/i); + expect(activateProse).toMatch(/`labeled` \(issues whose `add_labels` call was accepted\)/i); expect( /labeled < activated/.test(activateSkill), 'The reconciliation must compare the two counts explicitly.', ).toBe(true); }); + it('states the counts track accepted operations, not labels present on GitHub', () => { + // #1963 settled the vocabulary: an accepted safe output is queued, not applied. Step 2e + // counts `add_labels` calls, so its reason string and count definition must not claim + // the labels reached the issue -- nothing in this run reads labels back. Guarding the + // negative too, because the over-claim reappears as an innocuous-looking verb. + expect(activateProse).toMatch(/never state or imply that a counted label was applied/i); + expect(activateProse).not.toMatch(/activated issues received their labels/i); + expect(activateProse).not.toMatch(/proof that every label landed/i); + }); + it('counts a missing, rejected, or errored label call as unlabeled', () => { // Absence-of-success, not presence-of-error, must be the failing condition. The agent // does see an E002 rejection at invocation time, but a call can also simply never be diff --git a/workflows/squad.md b/workflows/squad.md index cf71ff165..f0ed0f80e 100644 --- a/workflows/squad.md +++ b/workflows/squad.md @@ -1991,12 +1991,12 @@ Root → Epics → Tasks. Phase-specific: filter to matching phase heading. **2e. Label-Operation Reconciliation — no activation is complete until every activated item's labels are accounted for.** -While Steps 2b/2c run, keep two counts: `activated` (issues created or recognized this run) and `labeled` (issues whose `add_labels` call returned successfully). An `add_labels` call that was never made, was rejected, or returned an error counts as **unlabeled**. At the end of Step 2: +While Steps 2b/2c run, keep two counts: `activated` (issues created or recognized this run) and `labeled` (issues whose `add_labels` call was accepted). An `add_labels` call that was never made, was rejected, or returned an error counts as **unlabeled**. These counts track *label operations*, not labels present on GitHub: acceptance means the call was queued for a specific target this turn, and gh-aw applies it in the post-agent job. Never state or imply that a counted label was applied, landed, or was confirmed on the issue — nothing here reads labels back. At the end of Step 2: 1. `labeled == activated` → the activation is complete; proceed to Step 3. -2. `labeled < activated` → **this is an incomplete activation, not a successful one.** Call `report_incomplete` with a `reason` naming the shortfall (`{labeled} of {activated} activated issues received their labels`) and `details` listing **every affected work item — the identifier you used to target its `add_labels` call, its title, and the label set it should have received**. For an item created this run that identifier is the `temporary_id` you minted under the Temporary-ID Contract (`#aw_epic{K}` / `#aw_task{N}`) — not a GitHub issue number, because creation is deferred to the safe-output job and no real number exists yet. Quote a real number only where one is independently verified: an epic or task matched by dedup-by-title, or an issue recognized by Step 1's idempotent-rerun path. Never predict, infer, or invent a number. `report_incomplete` logs a warning and opens or updates a durable `[aw] ... reported incomplete result` tracking issue; it does **not** change the run's conclusion — the run still reports success. That record and the rule below keep a truncated activation from passing as clean, so never rely on a red run to carry the signal. +2. `labeled < activated` → **this is an incomplete activation, not a successful one.** Call `report_incomplete` with a `reason` naming the shortfall (`{labeled} of {activated} activated issues had a label operation accepted`) and `details` listing **every affected work item — the identifier you used to target its `add_labels` call, its title, and the label set it should have received**. For an item created this run that identifier is the `temporary_id` you minted under the Temporary-ID Contract (`#aw_epic{K}` / `#aw_task{N}`) — not a GitHub issue number, because creation is deferred to the safe-output job and no real number exists yet. Quote a real number only where one is independently verified: an epic or task matched by dedup-by-title, or an issue recognized by Step 1's idempotent-rerun path. Never predict, infer, or invent a number. `report_incomplete` logs a warning and opens or updates a durable `[aw] ... reported incomplete result` tracking issue; it does **not** change the run's conclusion — the run still reports success. That record and the rule below keep a truncated activation from passing as clean, so never rely on a red run to carry the signal. -**Cap exhaustion is a reportable, nameable cause.** If the shortfall is because a cap was reached, say so in the `reason`, name which cap (`create-issue` 75 or `add_labels` 110) and list the work items that did not fit, and recommend `/squad plan activate phase {N}`. This is the one case where a cap may be named as the cause: it was observed, not guessed. Do not infer a cap from a rejection you never received, and do not treat the absence of an `E002` error as proof that every label landed: the count comparison, not the error stream, is the authority. +**Cap exhaustion is a reportable, nameable cause.** If the shortfall is because a cap was reached, say so in the `reason`, name which cap (`create-issue` 75 or `add_labels` 110) and list the work items that did not fit, and recommend `/squad plan activate phase {N}`. This is the one case where a cap may be named as the cause: it was observed, not guessed. Do not infer a cap from a rejection you never received, and do not treat the absence of an `E002` error as proof that every label operation was accepted: the count comparison, not the error stream, is the authority. **Never report a clean activation you did not perform.** An `activated` or `phases-activated` artifact listing every item as activated while `labeled < activated`, or omitting the shortfall, is a false success report — the silent truncation this step exists to prevent. From a47bdb892baf240a5bfd14b206013048d5c2e14f Mon Sep 17 00:00:00 2001 From: Brady Gaster Date: Tue, 1 Sep 2026 09:07:41 -0700 Subject: [PATCH 5/5] test(gh-aw): anchor the false-success assertion, which was vacuous Copilot review caught that the regex ended in an `|activated.*artifact` alternative. Alternation binds loosest, so the whole pattern collapsed to that branch and matched any mention of the artifact -- the test would have passed with the `labeled < activated` condition deleted from the prose, which is the entire invariant it names. Anchored to the full sentence and verified by mutation: removing the condition from Step 2e now fails (1 failed / 28 passed) where it previously passed. Second instance of this defect class in this suite; the sibling count assertions were tightened in 4ec89dde for the same reason. Audited the remaining alternations here -- `/finish green|still succeeds/` is legitimate (two accepted phrasings, both specific). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0515ad5d-4c5b-48bb-92a6-4e1296594cab --- test/gh-aw-activation-capacity.test.ts | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/test/gh-aw-activation-capacity.test.ts b/test/gh-aw-activation-capacity.test.ts index 6c337e58a..35aac3440 100644 --- a/test/gh-aw-activation-capacity.test.ts +++ b/test/gh-aw-activation-capacity.test.ts @@ -442,7 +442,13 @@ describe('gh-aw: self-validation reconciles activated items with label operation 'Emitting an activated/phases-activated artifact while labeled < activated must be ' + 'named as a false success report.', ).toBe(true); - expect(activateProse).toMatch(/phases-activated.*artifact.*while.*labeled < activated|activated.*artifact/i); + // Anchored to the whole sentence. The previous form ended in an `|activated.*artifact` + // alternative, and since alternation binds loosest the entire pattern collapsed to that + // branch -- it matched any mention of the artifact and would have passed with the + // `labeled < activated` condition deleted. Verified by deleting it. + expect(activateProse).toMatch( + /An `activated` or `phases-activated` artifact listing every item as activated while `labeled < activated`/i, + ); }); it('still forbids blaming a cap that was never actually reached', () => {