[zwanski-start]

Closes #136

What changed

• .claude/templates/testing_prompt.template.md: STEP 2 now asks the testing agent to choose a strategy:
  • Strategy A: Browser verification (the existing Playwright CLI flow) for web UI features
  • Strategy B: Command-line verification, which follows the steps in the issue:
    • B0 plan success criteria (command, input, expected exit code/output/files, at least one error case)
    • B1 make results observable by adding exit codes, stderr error messages and optional verbose logging to the program
    • B2 prepare simulated user input (fixtures in tests/fixtures/, stdin via printf/heredoc, args, env)
    • B3 run the program from bash and capture stdout/stderr/exit code separately (inside the project, .test-output/)
    • B4 compare the results with the criteria ("runs without crashing" is not a pass)
    • B5 keep the cases as an automated test and run the existing suite
  • The regression-fix steps, the commit message, the quality bar and AVAILABLE TOOLS (new "Command Line (Bash)" entry) now cover both strategies.
• prompts.py:
  • project_has_web_ui(project_dir) reads the app spec's <frontend> block and ignores HTML comments. It returns False when the spec has a <technology_stack> but no frontend, or the frontend is empty/"none"/"CLI…"/"terminal…"/"headless…". It returns None when there is no spec or no tech stack.
  • get_testing_prompt(): when a project has no web UI, it adds a "PROJECT TYPE: NO WEB UI" header, removes the browser strategy and the Playwright tool reference, and keeps command-line verification as the only strategy. Existing projects with an older browser-only testing_prompt.md get STEP 2 replaced with Strategy B, pulled from the base template so there is a single source of truth.
• test_testing_prompt.py: 16 pytest tests for detection, web/unknown/CLI prompt output, and the legacy-prompt path.
• CLAUDE.md: documents the two strategies and the new test file.

Verification

• python -m pytest test_testing_prompt.py: 16 passed
• ruff check .: clean. mypy prompts.py test_testing_prompt.py: clean. python test_security.py: all pass
• Rendered the testing prompt for a scratch project with a CLI-only spec. The header and Strategy B appear, and nothing from Playwright is left.
• No UI changes, so no browser check was needed.

Assumptions / notes

• I went with a strategy inside the existing testing agent rather than a separate agent type. The orchestrator, the MCP tools and the per-agent settings stay the same, and mixed projects (CLI + web) can use both strategies.
• An unknown project type (no spec, or a spec without a tech stack) keeps the full prompt with both strategies, and the agent picks one per feature.
• The bash allowlist is not changed. For example, python is not on the global allowlist, so Python CLI projects still need it in .autoforge/allowed_commands.yaml. The prompt tells the agent to use allowed alternatives or record the limitation, and not to work around it.
• The coding agent prompt is unchanged. This issue is about the testing agent.

🤖 Generated with Claude Code

[zwanski-start]

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 1 · Author: Dennis

The PR adds a command-line verification strategy (B0–B5: plan criteria, make results observable, simulated input, run from bash capturing stdout/stderr/exit code, compare, keep as automated tests) to the testing prompt, covering every step the issue asks for. For projects whose app spec declares no web frontend, it removes the browser instructions; older browser-only project prompts get Strategy B injected. The new tests, ruff and the security tests pass, the other test and mypy failures are the same on master, and probing with real inputs (the shipped spec template and master's old prompt) shows web projects unchanged and CLI projects correctly adapted. Minor non-blocking notes: a frontend description starting with 'Headless…' or 'Terminal…' is misclassified as having no web UI, a few browser phrases remain when an old project prompt is adapted, and a non-UTF-8 spec raises UnicodeDecodeError (the same issue already exists in has_project_prompts).

	Check	Details
✅	Issue requirements: plan criteria, error/success reporting, bash execution, simulated input, retrieve results	Strategy B in testing_prompt.template.md covers B0 planning (including an error case), B1 exit codes and stderr, B2 fixtures/stdin/args/env, B3 capturing stdout/stderr/exit code separately, B4 comparing against the criteria, and B5 keeping the cases as tests.
✅	CLI project gets command-line-only prompt (current template)	Rendered get_testing_prompt for a spec with technology_stack and no frontend: it starts with 'PROJECT TYPE: NO WEB UI', has Strategy B and no Strategy A, no 'playwright' text, and STEP 3 and AVAILABLE TOOLS are kept.
✅	CLI project with legacy (master) testing_prompt.md	Used master's real template as the project prompt: STEP 2 is replaced with Strategy B from the base template, there is no playwright text and no Browser Automation section, and IDs '4, 5' are filled in. Minor: workflow line 3 and STEP 3 still mention the browser ('Test the feature in the browser', 'Test your fix using browser automation'); the header overrides them.
✅	Web projects unaffected	app_spec.template.txt is detected as web (True). A web project with master's old prompt gets exactly the same prompt as before (checked with ==). An unknown project type (no spec) keeps the full template.
✅	project_has_web_ui edge cases	'No frontend', 'N/A', 'None', empty and commented-out frontends give False; React/Next.js, 'Node.js + EJS', Electron and Tkinter give True; a UTF-8 BOM is handled. Non-blocking false negatives: 'Headless CMS w/ Next.js storefront' and 'Terminal-styled web app' give False.
✅	Encoding robustness	A CP1252 spec containing é raises UnicodeDecodeError from get_testing_prompt, because get_app_spec only catches OSError. Not blocking: has_project_prompts in prompts.py already reads the spec the same way. Suggest catching ValueError in project_has_web_ui.
✅	Python test suite / lint / types vs master	test_testing_prompt.py: 16 passed. Full pytest: 160 passed, 21 failed, with exactly the same 21 failures on an origin/master worktree. ruff clean. mypy: 24 errors on both branches, none in prompts.py or test_testing_prompt.py.
✅	Security not weakened	security.py and the allowlist are not touched. The prompt tells the agent not to work around blocked commands. test_security.py passes.
✅	Server boot + UI smoke	uvicorn on 127.0.0.1:5247: /api/projects and / return 200. The SPA renders in Playwright with 0 console errors. Server stopped afterwards and the port is free.
⏭️	UI build / e2e	The PR changes no files under ui/ (prompt template, prompts.py, tests and docs only).

🧪 Commands run

Command	Result
python -m pytest test_testing_prompt.py -q	16 passed
ruff check .	All checks passed
mypy .	24 errors in 9 files, same as origin/master; none in changed files
python test_security.py	ALL TESTS PASSED
python -m pytest -q (PR vs origin/master worktree)	160 passed, 21 failed; same 21 failures on master
python probe.py (custom script: legacy-prompt adaptation, spec template detection, edge cases, encoding)	Legacy and new CLI prompts adapted correctly; web unchanged; CP1252 spec raises UnicodeDecodeError (minor)
uvicorn server.main:app --host 127.0.0.1 --port 5247 + curl /api/projects, /	200 / 200

No browser screenshots were taken in this round.

_{Posted by cubefarm · merges automatically once GitHub's checks pass}