Repository navigation
docs(e2e): repoint E1/E4 to the live fixture, preserve evidence records - #1856
Merged
Merged
Conversation
The E2E runbooks targeted `bradygaster/aspiregregator-squad-e2e`, which was deleted on 2026-08-24 along with every other test repo. Runs now bill to `octodemo`, and the fixture was recreated from the pristine read-only origin `bradygaster/Aspiregregator`. Moved the PROCEDURE, not the EVIDENCE. E1:4 and E4:27/:256 record which repo produced the readings they sit beside; rewriting those would falsify the records. They keep the retired name and are annotated as retired. Only the re-run inputs (`$FIXTURE`) were repointed. Three findings the repoint surfaced, none fixable by find-and-replace: 1. The repo NAME was reused across owners, so E1's `blocked_by_job_url` cross-check -- which matched the bare name -- could no longer identify the fixture, and would accept evidence from the deleted repo. Tightened to match owner AND name. 2. The new fixture is a bare application repo with no `.squad/` directory, so it has no roster. E4's central criterion (which leaked tokens are dispositive vs ambiguous) is DERIVED from roster contents, not fixed vocabulary. Added a re-derivation gate stating the reasoning rule, so a re-run cannot inherit the retired fixture's classification. 3. Phase 0 refreshes an activated fixture; it cannot create one. Added Phase 0-pre, which asserts activation state before choosing a path rather than letting 0b report UNREADABLE on all four files and burning a diagnostic cycle. Verification: the E2E runbook hygiene linter (#1822 / PR #1852) passes, and was mutation-tested against the new fenced block -- removing the exit-code pairing fails it at E4:390, so the new block is genuinely covered rather than vacuously passing. Closes #1851 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 20afe6d2-444e-414e-8a39-e67ab67ca6df
Contributor
|
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | |
| ✅ | No merge conflicts | No merge conflicts |
| ❌ | Copilot threads resolved | 1 unresolved Copilot thread(s) — fix and resolve before merging |
| ✅ | CI passing | All checks passing |
Files Changed (3 files, +68 −8)
| File | +/− |
|---|---|
.squad/e2e/E1-merge-continuation-relay.md |
+5 −3 |
.squad/e2e/E4-agent-binding-verification.md |
+55 −4 |
.squad/e2e/README.md |
+8 −1 |
Total: +68 −8
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Contributor
🟢 Impact Analysis — PR #1856Risk tier: 🟢 LOW 📊 Summary
🎯 Risk Factors
📦 Modules Affectedsquad-state (3 files)
This report is generated automatically for every PR. See #733 for details. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
E4’s new Phase 0-pre commentary currently equates any non-zero gh api exit with a 404/missing directory, which can mislead operators when the failure is actually auth/network/permissions-related.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the E2E runbook documentation to repoint E1/E4 re-run inputs to the new live fixture (octodemo/aspiregregator-squad-e2e) while explicitly preserving the original evidence records that reference the now-deleted retired fixture (bradygaster/aspiregregator-squad-e2e).
Changes:
- Repointed fixture targets used for re-runs to
octodemo/aspiregregator-squad-e2eand documented why the move occurred. - Hardened E1’s evidence cross-check wording to require matching owner + repo name (not just the repo name).
- Added an E4 “Phase 0-pre” gate to detect an unactivated fixture (missing
.squad/) and require re-derivation of roster-dependent criteria.
File summaries
| File | Description |
|---|---|
| .squad/e2e/README.md | Updates the documented fixture and records the rationale/implications of the forced move. |
| .squad/e2e/E4-agent-binding-verification.md | Repoints $FIXTURE, preserves retired-fixture evidence annotations, and adds Phase 0-pre to detect unactivated fixtures and require re-derivation. |
| .squad/e2e/E1-merge-continuation-relay.md | Preserves retired-fixture evidence header, repoints re-run $FIXTURE, and tightens the blocked_by_job_url cross-check to owner+repo matching. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+392
to
+395
| # 0 -> activated; skip to 0a and refresh normally. | ||
| # !=0 -> 404 / absent; Squad has never been activated here. Run activation FIRST. | ||
| # Do NOT interpret a 404 as "clean" and proceed: 0b would then report UNREADABLE | ||
| # for all four files, which is the correct failure but wastes a cycle diagnosing it. |
This was referenced Aug 24, 2026
bradygaster
added a commit
that referenced
this pull request
Aug 24, 2026
…un (#1862) Executing E4 end-to-end against the new fixture surfaced five documented procedures that are stale or wrong. Two are silent-failure traps: they do not error, they produce a confident wrong answer. E4 Phase 2 -- short path 5 -> 8 commands. The bold "do not continue to `plan validate` / `plan accept ...`; those verify nothing additional" was false. `plan activate` hard-blocks without an `impl-accepted` artifact while reporting all six jobs `success`, so it looks clean and does nothing. On the old path Phases 3b/3c/3e had no artifact to read and only 3a scored. Added the two newly-required accept steps and the `plan validate` first-pass-FAIL cycle. E4 Phase 0b -- compare content, not byte size. Three provisioning artifacts each produce a false STALE: the `source:` line `gh aw add` injects into the top-level workflow, CRLF vs LF, and `gh aw add` stripping the trailing newline. With only the first two handled the main workflow still read 1391 vs 1390. Normalizing all three takes all four surfaces to MATCH; verified live against the fixture. E4 3b -- the contamination table and its "guaranteed false FAIL" warning described the fixture that was deleted before this one existed. The new fixture started with 12 labels and zero `squad:*`, which inverts the guidance. Missed when #1856 repointed the runbooks. E4 3a -- warn against substituting `Out-File` for `Set-Content`. `Out-File` wraps at console width and splits markdown table rows mid-cell; it split the roster name `Keaton` into `Ke`/`aton` and silently corrupted a roster check. E1 -- worker PRs are not parked in `action_required`. GitHub suppresses workflow triggers entirely for `GITHUB_TOKEN`-authored PRs, so no run is created. The documented `runs/<ID>/approve` override cannot apply to a run that does not exist. Same consequence, different cause; the PAT/App-token fix stands. Also records the 2026-08-24 amended-procedure verdict at the head of E4: 3a and 3c PASS, 3b/3e PARTIAL -> #1859, #1860. Verified: `test/e2e-runbook-hygiene.test.ts` (3/3), `npm run lint`. Phase 0b's replacement block was executed against the live fixture, not just reviewed. Docs only; no changeset. Closes #1861 Co-authored-by: brady gaster <bradygaster@github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 20afe6d2-444e-414e-8a39-e67ab67ca6df
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1851.
What changed
The E2E runbooks pointed at
bradygaster/aspiregregator-squad-e2e, deleted 2026-08-24 with every other test repo. Re-run targets now point atoctodemo/aspiregregator-squad-e2e, recreated from the pristine read-only originbradygaster/Aspiregregator. Runs bill tooctodemobecause thebradygasterorg was near its Actions-minute limit.Procedure moved; evidence did not.
E1:4andE4:27/E4:256state which repo produced the readings printed beside them. Rewriting them would falsify an evidence record, so they keep the retired name and are annotated as retired. Only$FIXTURE— the re-run input — was repointed.Three things a find-and-replace would have gotten wrong
1. The repo name was reused across owners. E1's
blocked_by_job_urlcross-check matched the bare nameaspiregregator-squad-e2e. After the move that string no longer identifies the fixture — it matches the deleted repo equally well, so the check would accept evidence from a repo that does not exist. Tightened to match owner and name.2. E4's central criterion is derived, not fixed. The dispositive/ambiguous split (
devrelandreviewerdispositive,leadambiguous) is not vocabulary — it is a function of the fixture's roster: a token is dispositive when it appears nowhere in the roster, ambiguous when the roster's own Role column contains it. Measured: the new fixture has no.squad/directory at all (gh api repos/octodemo/aspiregregator-squad-e2e/contents/.squad→ 404, identical to the origin), so it has no roster, and casting will not reproduce Keaton/McManus/Fenster/Hockney/Kint. Added a re-derivation gate that states the reasoning rule so a re-run re-classifies instead of inheriting.3. Phase 0 cannot create a fixture, only refresh one. It assumes
.github/workflows/squad.mdis already present. Added Phase 0-pre, which asserts activation state up front. Without it, 0b reportsUNREADABLEon all four files — the correct failure, but only after a wasted diagnostic cycle.Verification
test/e2e-runbook-hygiene.test.ts(the E2E runbook commands ship untested — a non-functional query vacuously passes its gate #1822 linter): 3/3 pass.$LASTEXITCODEpairing from the new Phase 0-pre block fails the linter atE4:390with the exact suppression diagnostic — so the new fenced block is genuinely covered, not vacuously passing.aspiregregatorreference. The 4 remainingbradygaster/...hits are all explicitly annotated as retired. The 4 hits intest/gh-aw-quality.test.tsare unaffected::520/:539reference a localtest-fixtures/planning/aspiregregator/directory, and:1215/:1246are comments citing historical run IDs.Remaining, and deliberately not in this PR
The fixture is repointed but not yet runnable. It still needs Squad activation (which generates
.squad/and the roster) and repo secrets, which were not copied from the origin and cannot be recovered from the deleted fixture. That is operator work for the E2E run itself, not a documentation change — and Phase 0-pre now detects and states it rather than letting a run discover it late.