Skip to content

fix(test): budget the delivered prompt, not authored source - #1849

Merged
bradygaster merged 1 commit into
devfrom
squad/1842-prompt-budget-measures-source-not-prompt
Aug 24, 2026
Merged

bradygaster merged 1 commit into
devfrom
squad/1842-prompt-budget-measures-source-not-prompt

Conversation

@bradygaster

Copy link
Copy Markdown
Collaborator

Closes #1842

The bug

The prompt-budget gate in test/gh-aw-quality.test.ts asserted a gh-aw prompt ceiling that does not exist, and measured a quantity gh-aw never sends. Both sides of the comparison were wrong, which left it reporting ~26 bytes of headroom and failing correct changes on byte count alone.

1. The ceiling is misattributed

The comment claimed "gh-aw enforces a hard 100 KB prompt ceiling (102 400 bytes)". Verified against gh-aw:

  • 102400 is defaultRepoMemoryMaxFileSize in github/gh-aw → pkg/workflow/repo_memory.go — "the default maximum file size in bytes (100KB)" for repo-memory files. Unrelated to prompts.
  • .github/aw/workflow-constraints.md never mentions a prompt ceiling.
  • .github/aw/token-optimization.md treats prompt size purely as cost/quality ("Strip redundant instructions, examples, and pleasantries from the prompt body"), never as a byte limit.

2. It measured source, not the prompt

gh-aw strips every inline ## skill: block during the setup/interpolation step — per gh-aw .github/aw/skills.md:

Extraction runs in the setup/interpolation step (not at compile time): gh-aw writes each block to engine-specific skill locations and removes it from the main prompt body.

Measured composition of what the gate was summing:

bytes
What the test measured (raw source sum) 97,254
What actually enters initial context (ambient) 31,855
Extracted to on-demand skills 65,398 (67%)

shared/squad-planning-ontology.md (16,420 B) and shared/squad-planning-policy.md (4,538 B) are entirely skill blocks — they start with ## skill: on line 1 and are fully extracted. workflows/squad.md is 69,608 B of which only ~25,167 B is prompt body; 21 skill blocks begin at L540.

3. It ignored what gh-aw does inject

Compiling for real (gh aw compile .github/workflows/squad.md --strict, exit 0) shows GH_AW_PROMPT_CONFIG carrying ~17 prompt items, including eight gh-aw boilerplate files the test counts at zero: xpia.md, temp_folder_prompt.md, markdown.md, safe_outputs_prompt.md, safe_outputs_create_pull_request.md, mcp_cli_tools_prompt.md, cli_proxy_with_safeoutputs_prompt.md, pr_context_prompt.md. The lock also defers our four files via {{#runtime-import}} (GH_AW_PROMPT_CONTENT_0005–0008), confirming they are assembled at runtime rather than inlined at compile time.

The fix

The correct budget already existed in this file. The gh-aw: inline skill extraction suite runs a faithful mirror of gh-aw's extractor (extractInlineSkills, itself modeled on setup/js/extract_inline_skills.cjs) and asserts keeps the ambient prompt under 40 KB against post-extraction ambient bytes. That is the real gate, and it passes at ~31.8 KB.

So this change does not add a second budget — it stops the redundant one from lying:

  • Removed the fabricated GH_AW_PROMPT_CEILING_KB / 102400 constant and the false comment.
  • Collapsed the two byte assertions into a single source-growth guard at 160 KB, explicitly documented as not a gh-aw limit, whose failure message points the reader at the ambient check first.
  • Kept readText()'s CRLF normalization — it is why this math is platform-independent (.gitattributes pins eol=lf for many types but not *.md, so Windows worktrees would otherwise read +2,093 bytes).

Impact

Real headroom is ~70 KB, not 26 bytes. This unblocks #1801, #1780, #1730, #1757, and #1608, none of which needed to compact shell one-liners or delete rationale to buy bytes.

It also changes #1834: the shell-input security contract (workflows/squad.md L245–299, 2,695 B) can move into a ## skill: block rather than being deleted for budget reasons. #1834 is no longer a prerequisite for this issue.

Validation

npx vitest run test/gh-aw-quality.test.ts
Tests  91 passed | 13 skipped (104)

The 13 skips are pre-existing and tracked in #1833 (HAS_POSIX_SHELL gating at L30 skips two describe blocks on Windows).

Test-only change — no packages/*/src/ touched, so no changeset required.

The prompt-budget gate asserted a "hard 100 KB gh-aw prompt ceiling" that
does not exist, and measured a quantity gh-aw never sends.

The cited 102400 is defaultRepoMemoryMaxFileSize (gh-aw
pkg/workflow/repo_memory.go) — a repo-memory FILE cap, unrelated to
prompts. gh-aw's token-optimization.md treats prompt size as a
cost/quality concern, never a byte limit.

It also summed raw source. gh-aw strips every inline `## skill:` block
during setup/interpolation (gh-aw skills.md), so ~65 KB of the 97 KB
measured never enters the initial request, while the boilerplate gh-aw
injects (xpia.md, markdown.md, safe_outputs_*.md) was not counted at
all. Wrong on both sides, the gate reported ~26 bytes of headroom and
blocked correct changes on byte count alone.

The real budget was already asserted correctly by "keeps the ambient
prompt under 40 KB" in the inline-skill-extraction suite, which measures
post-extraction ambient bytes (currently ~31.8 KB). This keeps a single
source-GROWTH guard at 160 KB so unbounded authoring growth stays
visible, with a failure message that points at the ambient check first.

Closes #1842

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 20afe6d2-444e-414e-8a39-e67ab67ca6df
Copilot AI lite review requested due to automatic review settings August 24, 2026 17:18
@github-actions

github-actions Bot commented Aug 24, 2026 •

Copy link
Copy Markdown
Contributor

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit 6a517bc

PR Scope: 🔧 Infrastructure

⚠️ 1 item(s) to address before review

Status Check Details
✅ Single commit 1 commit — clean history
✅ Not in draft Ready for review
✅ Branch up to date Up to date with dev
❌ Copilot review No Copilot review yet — it may still be processing
✅ Changeset present No source files changed — changeset not required
✅ Scope clean No .squad/ or docs/proposals/ files
✅ No merge conflicts No merge conflicts
✅ Copilot threads resolved No Copilot review threads
✅ CI passing All checks passing

Files Changed (1 file, +28 −21)

File +/−
test/gh-aw-quality.test.ts +28 −21

Total: +28 −21


This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

@github-actions

Copy link
Copy Markdown
Contributor

🟢 Impact Analysis — PR #1849

Risk tier: 🟢 LOW

📊 Summary

Metric Count
Files changed 1
Files added 0
Files modified 1
Files deleted 0
Modules touched 1

🎯 Risk Factors

  • 1 files changed (≤5 → LOW)
  • 1 module(s) touched (≤1 → LOW)

📦 Modules Affected

tests (1 file)
  • test/gh-aw-quality.test.ts

This report is generated automatically for every PR. See #733 for details.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is test-only and replaces an incorrect, misleading budget assertion with checks aligned to the workflow’s actual prompt assembly behavior.

Pull request overview

This PR fixes the gh-aw quality “prompt budget” gate to budget what is actually delivered (ambient prompt after inline skill extraction) rather than raw authored markdown bytes, and removes a misleading/incorrect 100KB “prompt ceiling” assertion.

Changes:

  • Removed the fabricated 100KB prompt-ceiling constant and the headroom-based regression assertion tied to it.
  • Consolidated the budget checks into the existing ambient-prompt (<40KB) gate plus a separate, explicit source-growth guard (160KB) that is documented as not a gh-aw limit.
  • Updated test messaging/comments to explain the distinction between authored source size and delivered ambient prompt size.
File summaries
File Description
test/gh-aw-quality.test.ts Removes incorrect prompt-ceiling/headroom checks and replaces them with a documented source-growth guard while relying on the canonical ambient prompt budget.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@bradygaster
bradygaster merged commit 99afdda into dev Aug 24, 2026
17 checks passed
@bradygaster
bradygaster deleted the squad/1842-prompt-budget-measures-source-not-prompt branch August 24, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

workflows/squad.md is ~37 bytes from the prompt-budget floor — the next change to it fails CI on byte count alone

2 participants