[zwanski-start]

Closes #157

What changed

Regression testing used to mean a testing agent re-running every feature by hand in the browser (snapshots, screenshots, reading files), which uses up a lot of quota. Now the agent that builds a feature also leaves behind a script that later regression runs can just execute.

• Coding agents (non-YOLO; single, batch and legacy prompts) get a new STEP 5.8, placed right before STEP 6 (mark passing). After verifying the feature in the browser, the agent writes a headless @playwright/test script at tests/regression/feature-<ID>.spec.ts (or .js) and commits it with the feature. The script must be deterministic: it reads the app URL from BASE_URL with a default, uses unique test data that it cleans up, and relies on auto-waiting assertions.
• Testing agents get a header that is added to the start of their prompt. It tells them to run npx playwright test tests/regression/feature-<ID>.spec.ts --reporter=line first:
  • The script passes: move on to the next feature, with no manual browser work.
  • The script fails: either handle it as a regression with the existing fix flow, or fix a stale script.
  • There is no script: verify manually once, then write the script so the next run is cheap. Existing projects build up their scripts this way over time.
• The instructions are added in prompts.py at runtime rather than only in the templates. Existing projects keep copies of their prompts in .autoforge/prompts/, so they get the change too. Customized coding prompts without a ### STEP 6: heading get the section added at the end.
• New file test_prompts.py with 7 tests. CLAUDE.md now documents the convention.

Verification

• python -m pytest test_prompts.py: 7 passed
• ruff check . and mypy prompts.py test_prompts.py: clean
• python test_security.py: all passed
• Checked that bash_security_hook allows npx playwright test ... and ls tests/regression/....
• No UI changes, so no browser check.

Assumptions

• The scripts use Playwright, because the generated apps are web apps and agents already test them with Playwright. API-only features use Playwright's request fixture. Agents install @playwright/test in the project as needed.
• Scripts are an addition to the coding agent's own browser verification, not a replacement, so coding sessions are just as thorough as before. The savings come from regression runs.
• The orchestrator still starts a testing agent to run the script, because something has to make sure the app server is running and to fix regressions. Having the orchestrator run the scripts itself, with no LLM session at all, would be a bigger follow-up.
• YOLO mode is unchanged: it doesn't write scripts and has no testing agents.

🤖 Generated with Claude Code

[zwanski-start]

🔍 QA report: ✅ Passed

Tester: Ada (cubefarm agent, in a fresh session) · Round: 1 · Author: Dennis

Meets issue #157: non-YOLO coding prompts (plain, single-feature, batch, and existing projects' saved prompts) now include STEP 5.8, placed once right before STEP 6, which tells the agent to save a headless @playwright/test script at tests/regression/feature-.spec.ts. Testing prompts in every mode start with a header telling the agent to run that script first and test by hand only when it is missing or failing. New tests (7), ruff and the security suites pass. The full pytest and mypy failures are identical on a clean copy of master. A sample script run with the exact npx command from the prompt passed against a running server and failed correctly against a dead one.

	Check	Details
✅	Coding prompt gets STEP 5.8 right before STEP 6	Built the coding prompt for a freshly created project (default templates). The step appears once, right before '### STEP 6: UPDATE FEATURE STATUS'. The single-feature and batch prompts also contain it exactly once.
✅	YOLO mode unchanged	The test suite confirms the coding, single-feature and batch prompts have no STEP 5.8 when yolo_mode=True.
✅	Testing prompt starts with the 'run the script first' header	The header comes first in batch, single and unassigned modes. The {{TESTING_FEATURE_IDS}} placeholder is still filled in (no '{{' left). The manual workflow and the 'If the feature FAILS' section it points to are still present.
✅	Existing and custom project prompts	Prompts saved in .autoforge/prompts/ get both changes because they are added at load time. A custom prompt with no STEP 6 heading gets the step added at the end.
✅	Security allowlist intact and allows the new commands	security.py is unchanged. bash_security_hook allows 'npx playwright test tests/regression/feature-3.spec.ts --reporter=line', 'ls tests/regression/feature-3.spec.*', 'npm install -D @playwright/test', 'npx playwright install chromium' and './init.sh'.
✅	The script convention works in practice	Wrote a sample tests/regression/feature-1.spec.ts that uses the request fixture and reads BASE_URL, with no Playwright config. 'npx playwright test tests/regression/feature-1.spec.ts --reporter=line' gave 1 passed against a running local server and 1 failed (ECONNREFUSED) with BASE_URL pointing at a dead port. No processes were left running.
✅	Python lint, type checks and tests	ruff is clean, mypy on the changed files is clean, test_prompts.py has 7 passed, and the security suites pass. Full pytest has 21 failures and full mypy has 25 errors; both are identical on a clean copy of origin/master. The 21 failures are in test_devserver_security.py, which uses @pytest.mark.asyncio, but the pytest-asyncio plugin is not installed in the environment.
⏭️	UI build, browser check and e2e tests	The PR touches no UI files (only prompts.py, test_prompts.py and CLAUDE.md), so there was nothing to check in the browser and no screenshots were taken.
✅	Edge cases (non-blocking)	In a generated app that uses Vitest or Jest, the default test patterns may also pick up tests/regression/*.spec.ts. The testing header always shows the run command with .spec.ts, even when the script is .spec.js. A project-level Playwright config with a different test folder could make the file report 'No tests found'. An agent can work around all three, so none of them blocks the PR.

🧪 Commands run

Command	Result
git diff origin/master...HEAD	3 files changed: prompts.py, test_prompts.py, CLAUDE.md
uv venv -p 3.12 && uv pip install -r requirements.txt	Scratch virtualenv created
python -m pytest test_prompts.py	7 passed
python -m pytest (full)	151 passed, 21 failed; master: 144 passed, the same 21 failed (pytest-asyncio not installed)
ruff check .	All checks passed
mypy prompts.py test_prompts.py	No issues
mypy .	25 errors in 10 files, identical on master (none in the changed files)
python test_security.py	169 passed, 0 failed
python test_security_integration.py	9 passed, 0 failed
bash_security_hook probe on the new commands	All 5 allowed
Prompt output for a freshly created project (single, batch, testing)	STEP 5.8 appears once, before STEP 6; testing header comes first; placeholder filled in
npx playwright test tests/regression/feature-1.spec.ts --reporter=line (sample project)	1 passed against a running server; 1 failed with ECONNREFUSED when BASE_URL pointed at a dead port

No browser screenshots were taken in this round.

_{Posted by cubefarm · merges automatically once GitHub's checks pass}