Repository navigation
E2E runbook commands ship untested — a non-functional query vacuously passes its gate #1822
Description
Problem
E2E runbook procedures ship as documented shell commands that are never executed-tested. A command that cannot run at all can sit in a runbook indefinitely, and because failures are suppressed it will report the passing answer rather than an error.
Concrete instance (found 2026-08-21)
.squad/e2e/E4-agent-binding-verification.md Phase 3b, introduced in a2e5c18c (PR #1791):
gh issue list --repo $FIXTURE --state all --limit 100 \
--json number,title,labels,createdAt \
--jq --arg t "$E4_START" '.[] | select(.createdAt > $t) | ...' 2>$null
gh has no --arg flag. --jq consumes --arg as its expression and the remainder become stray positionals:
$ gh issue list ... --jq --arg t "2026-01-01T00:00:00Z" '...'
unknown arguments ["t" "2026-01-01T00:00:00Z" ".[] | select(...)"]
exit=1
With the trailing 2>$null, the captured result is 0 rows, silently, on every run. Verified independently twice (Flight, Coordinator).
Why it wasn't caught
Three properties compounded:
- The pass condition is universally quantified. "Every
squad:{x}on a newly created issue must be a roster Name" is vacuously true over the empty set. - No "confirm it returned rows" instruction. Phase 3a has exactly this instruction and is therefore safe; 3b did not.
- Empty output manufactures the passing values for the two fields the verdict schema marks
# dispositive(squad_devrel_present,squad_reviewer_present), wheretrueis the FAIL evidence.
So an infrastructure failure did not merely fail to inform the verdict — it wrote the green answer for the gate whose entire purpose is catching falsely-minted labels (#1812).
Measured against the bar recorded in .squad/decisions.md (2026-08-20, Test bar for the gh-aw workstream): "A test that passes while the system is broken is decoration, not a gate." Phase 3b passed while broken, by construction, for its entire lifetime.
Scope note
The recorded E4 n=1 verdict is not invalidated. Its Condition 2 scored a timeline labeled events query that returned 15 events — a different instrument. The exposure was prospective.
PR #1821 fixes this specific instance. This issue is about the absence of a mechanism that would have caught it, and would catch the next one.
Suggested direction
- A smoke pass that executes every command in an E-scenario against the fixture and asserts non-zero exit or zero rows is surfaced, not swallowed.
- Runbook convention: a gate with a universally-quantified pass condition must assert its input set is non-empty before evaluating.
- Related: the
$LASTEXITCODEhouse rule (separate issue).
Owners
FIDO (test/CI gate) with Sims (runbook convention).
Activity
bradygaster commentedon Aug 23, 2026
1. Owner & rationale
Owner:
squad:sims. The defect is in.squad/e2e/runbook reliability and measurement design; Sims owns E2E test engineering.Labels reconciled as:
gh-aw,type:bug,priority:p1,wave:1-next,squad:sims,triaged,go:spec-ready.2. Files in scope
Verified existing paths:
.squad/e2e/E4-agent-binding-verification.md.squad/e2e/E1-merge-continuation-relay.md.squad/e2e/README.md.squad/e2e/windows-test-baseline.mdtest/gh-aw-quality.test.ts3. Reproduction / measurement
This command was execution-tested and uses no output suppression:
It exits with
unknown arguments ["t" "2026-01-01T00:00:00Z" ...];gh issue listhas no--argflag. Any runbook command using this shape plus error suppression can manufacture an empty result set.4. Verification bar
The runbook gate must fail or become explicitly INCONCLUSIVE when its data-extraction command exits non-zero or returns zero rows for a universally quantified assertion. A command-smoke check must execute every documented
ghcommand shape in the E-series runbooks and fail on unknown flags, non-zero$LASTEXITCODE, or suppressed stderr.5. Traps
No
2>$null, no2>/dev/null, and no other output suppression in the proposed checks. Do not reintroduce--argundergh --jq; interpolate in PowerShell or use a tool that actually supports jq args. A universal condition like "every returned label is valid" must first prove the input set is non-empty, otherwise empty output is a false green.6. Out of scope
Do not rerun or rewrite the historical E4 verdict; the issue body records that its n=1 score used a different timeline-events instrument. Do not fix product owner-label behavior here; that is #1812.
7. Session-stall guard
docs(e2e): repoint E1/E4 to the live fixture, preserve evidence recor…
test(e2e): gate the class of defect that let a broken query pass (git…