Repository navigation
fix(test): budget the delivered prompt, not authored source - #1849
Conversation
The prompt-budget gate asserted a "hard 100 KB gh-aw prompt ceiling" that does not exist, and measured a quantity gh-aw never sends. The cited 102400 is defaultRepoMemoryMaxFileSize (gh-aw pkg/workflow/repo_memory.go) — a repo-memory FILE cap, unrelated to prompts. gh-aw's token-optimization.md treats prompt size as a cost/quality concern, never a byte limit. It also summed raw source. gh-aw strips every inline `## skill:` block during setup/interpolation (gh-aw skills.md), so ~65 KB of the 97 KB measured never enters the initial request, while the boilerplate gh-aw injects (xpia.md, markdown.md, safe_outputs_*.md) was not counted at all. Wrong on both sides, the gate reported ~26 bytes of headroom and blocked correct changes on byte count alone. The real budget was already asserted correctly by "keeps the ambient prompt under 40 KB" in the inline-skill-extraction suite, which measures post-extraction ambient bytes (currently ~31.8 KB). This keeps a single source-GROWTH guard at 160 KB so unbounded authoring growth stays visible, with a failure message that points at the ambient check first. Closes #1842 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 20afe6d2-444e-414e-8a39-e67ab67ca6df
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | No Copilot review threads |
| ✅ | CI passing | All checks passing |
Files Changed (1 file, +28 −21)
| File | +/− |
|---|---|
test/gh-aw-quality.test.ts |
+28 −21 |
Total: +28 −21
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
🟢 Impact Analysis — PR #1849Risk tier: 🟢 LOW 📊 Summary
🎯 Risk Factors
📦 Modules Affectedtests (1 file)
This report is generated automatically for every PR. See #733 for details. |
There was a problem hiding this comment.
🟢 Approval recommended
The change is test-only and replaces an incorrect, misleading budget assertion with checks aligned to the workflow’s actual prompt assembly behavior.
Pull request overview
This PR fixes the gh-aw quality “prompt budget” gate to budget what is actually delivered (ambient prompt after inline skill extraction) rather than raw authored markdown bytes, and removes a misleading/incorrect 100KB “prompt ceiling” assertion.
Changes:
- Removed the fabricated 100KB prompt-ceiling constant and the headroom-based regression assertion tied to it.
- Consolidated the budget checks into the existing ambient-prompt (<40KB) gate plus a separate, explicit source-growth guard (160KB) that is documented as not a gh-aw limit.
- Updated test messaging/comments to explain the distinction between authored source size and delivered ambient prompt size.
File summaries
| File | Description |
|---|---|
| test/gh-aw-quality.test.ts | Removes incorrect prompt-ceiling/headroom checks and replaces them with a documented source-growth guard while relying on the canonical ambient prompt budget. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Closes #1842
The bug
The prompt-budget gate in
test/gh-aw-quality.test.tsasserted a gh-aw prompt ceiling that does not exist, and measured a quantity gh-aw never sends. Both sides of the comparison were wrong, which left it reporting ~26 bytes of headroom and failing correct changes on byte count alone.1. The ceiling is misattributed
The comment claimed "gh-aw enforces a hard 100 KB prompt ceiling (102 400 bytes)". Verified against gh-aw:
102400isdefaultRepoMemoryMaxFileSizeingithub/gh-aw→pkg/workflow/repo_memory.go— "the default maximum file size in bytes (100KB)" for repo-memory files. Unrelated to prompts..github/aw/workflow-constraints.mdnever mentions a prompt ceiling..github/aw/token-optimization.mdtreats prompt size purely as cost/quality ("Strip redundant instructions, examples, and pleasantries from the prompt body"), never as a byte limit.2. It measured source, not the prompt
gh-aw strips every inline
## skill:block during the setup/interpolation step — per gh-aw.github/aw/skills.md:Measured composition of what the gate was summing:
shared/squad-planning-ontology.md(16,420 B) andshared/squad-planning-policy.md(4,538 B) are entirely skill blocks — they start with## skill:on line 1 and are fully extracted.workflows/squad.mdis 69,608 B of which only ~25,167 B is prompt body; 21 skill blocks begin at L540.3. It ignored what gh-aw does inject
Compiling for real (
gh aw compile .github/workflows/squad.md --strict, exit 0) showsGH_AW_PROMPT_CONFIGcarrying ~17 prompt items, including eight gh-aw boilerplate files the test counts at zero:xpia.md,temp_folder_prompt.md,markdown.md,safe_outputs_prompt.md,safe_outputs_create_pull_request.md,mcp_cli_tools_prompt.md,cli_proxy_with_safeoutputs_prompt.md,pr_context_prompt.md. The lock also defers our four files via{{#runtime-import}}(GH_AW_PROMPT_CONTENT_0005–0008), confirming they are assembled at runtime rather than inlined at compile time.The fix
The correct budget already existed in this file. The
gh-aw: inline skill extractionsuite runs a faithful mirror of gh-aw's extractor (extractInlineSkills, itself modeled onsetup/js/extract_inline_skills.cjs) and assertskeeps the ambient prompt under 40 KBagainst post-extraction ambient bytes. That is the real gate, and it passes at ~31.8 KB.So this change does not add a second budget — it stops the redundant one from lying:
GH_AW_PROMPT_CEILING_KB/102400constant and the false comment.readText()'s CRLF normalization — it is why this math is platform-independent (.gitattributespinseol=lffor many types but not*.md, so Windows worktrees would otherwise read +2,093 bytes).Impact
Real headroom is ~70 KB, not 26 bytes. This unblocks #1801, #1780, #1730, #1757, and #1608, none of which needed to compact shell one-liners or delete rationale to buy bytes.
It also changes #1834: the shell-input security contract (
workflows/squad.mdL245–299, 2,695 B) can move into a## skill:block rather than being deleted for budget reasons. #1834 is no longer a prerequisite for this issue.Validation
The 13 skips are pre-existing and tracked in #1833 (
HAS_POSIX_SHELLgating at L30 skips twodescribeblocks on Windows).Test-only change — no
packages/*/src/touched, so no changeset required.