Skip to content

Save agent validation screenshots to .autoforge/validation/ - #289

Open
zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-170-ada
Open

zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-170-ada

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #170

What changed

  • Prompt templates (coding_prompt, testing_prompt, auto_improve_prompt): screenshot instructions now use playwright-cli screenshot --filename=.autoforge/validation/<name>.png. They also tell the agent never to save screenshots anywhere else in the project and never to commit them.
  • autoforge_paths.py: validation/ added to the generated .autoforge/.gitignore. New helper get_validation_dir().
  • prompts.py:
    • scaffold_project_prompts creates .autoforge/validation/.
    • New migration step v1 → v2 (CURRENT_MIGRATION_VERSION = 2). It refreshes .autoforge/.gitignore, creates the directory, and rewrites the default screenshot wording in existing projects' copied coding_prompt.md / testing_prompt.md.
    • v0 projects go through v1 then v2, because the v1 replacement blocks contain the same default wording.
  • CLAUDE.md: .autoforge/validation/ documented in the generated-project structure.

Verification

  • ruff check, mypy on the changed modules, test_security.py, and the pytest suites (test_client, test_dependency_resolver, test_rate_limit_utils) all pass.
  • Migration: I built a project from the old (master) templates, set it to migration version 1, and ran migrate_project_to_current. Both prompts came out byte-identical to the new templates, .autoforge/.gitignore contained validation/, and the directory was created. A second run was a no-op.
  • Scaffold: a fresh scaffold creates .autoforge/validation/. In a git repo, a file placed there does not appear in git status.
  • Path handling: I checked that playwright-cli screenshot --filename=.autoforge/validation/x.png writes the file relative to the cwd, and creates the directory if it is missing.
  • No UI changes, so there was nothing to check in the browser.

Assumptions

  • The migration only matches the exact default template wording. Prompts that users have customized are left alone rather than rewritten heuristically.
  • Snapshots (playwright-cli snapshot) still go to .playwright-cli/. They are element-ref lookups rather than validation evidence, and that directory is already gitignored.
  • auto_improve_prompt is not copied into projects (it is loaded from the template), so it needs no migration.

🤖 Generated with Claude Code

Coding/testing/auto-improve prompt templates now tell agents to pass
--filename=.autoforge/validation/<name>.png to playwright-cli screenshot.
The directory is created on scaffold, added to .autoforge/.gitignore, and
existing projects are migrated (v1 -> v2) by rewriting the default
screenshot wording in their copied prompts.

Closes AutoForgeAI#170

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Barbara (cubefarm agent, in a fresh session) · Round: 1 · Author: Ada

Agents are now told to save screenshots under .autoforge/validation/, which is gitignored and documented. A new migration step (version 1 to 2) brings existing projects onto the new prompt wording. I tested that migration on version 0 and version 1 projects: the prompts came out byte-identical to the new templates, the gitignore and folder were added, a second run did nothing, and customized prompts were left alone. Running the real playwright-cli screenshot --filename command wrote the file into the gitignored folder, and ruff, mypy (no new errors) and the unit tests are clean.

Check Details
✅ Templates point screenshots at .autoforge/validation/ coding_prompt, testing_prompt and auto_improve_prompt now use --filename=.autoforge/validation/.png and tell the agent never to save screenshots elsewhere or commit them. Snapshots still go to .playwright-cli/, which is already gitignored.
✅ Validation folder is gitignored _GITIGNORE_CONTENT now contains validation/. In a git-initialized freshly scaffolded project, a file placed in .autoforge/validation/ did not appear in git status.
✅ Fresh scaffold creates the folder scaffold_project_prompts creates .autoforge/validation/ and sets .migration_version to 2.
✅ Migration from version 1 to 2 Project scaffolded with master's code and set to version 1. After migrate_project_to_current, coding_prompt.md and testing_prompt.md were byte-identical to the new templates, .gitignore contained validation/, the folder existed and the version was 2. A second run returned [].
✅ Migration from version 0 (prompts in old root prompts/ folder) Ran through version 1 and then 2. Both prompts were byte-identical to the new templates.
✅ Customized prompts left alone A custom coding_prompt.md was unchanged; only the default testing_prompt.md was rewritten.
✅ Real playwright-cli --filename behaviour playwright-cli screenshot --filename=.autoforge/validation/feature-1-a.png created the missing folder and wrote the PNG relative to the working folder, and git did not list it. The agent runs inside the project folder (client.py:478).
✅ Windows line endings Checked in code, not run on Windows: Path.read_text uses universal newlines, so prompts with Windows line endings still match the \n replacement strings.
✅ Backend boots uvicorn on 127.0.0.1:5443: /api/health, /api/projects and /api/settings return 200; an unknown project returns 404. Server stopped afterwards.
⏭️ UI in a real browser The PR changes no UI code, so I did not test in a browser.
⏭️ Unit tests for the new migration step There are none. Manual testing covered it; not blocking.
🧪 Commands run
Command Result
ruff check . All checks passed
mypy . (PR compared with origin/master) No errors in changed files; the same 24 errors as master plus 1 from the local venv/ folder in this worktree (setup artifact)
python test_security.py ALL TESTS PASSED
python -m pytest test_client.py test_dependency_resolver.py test_rate_limit_utils.py 55 passed
python -m pytest test_security_integration.py test_devserver_security.py test_ws_security.py 21 failed, 76 passed: identical on origin/master, so already failing on master, not caused by this PR
python scratchpad/mig.py (migration and scaffold probe script) Version 0, version 1 and customized-prompt cases behave as expected; second run does nothing
playwright-cli screenshot --filename=.autoforge/validation/feature-1-a.png PNG written into the folder, which was created; git did not list it
uvicorn server.main:app --port 5443 + curl 200 for health, projects and settings; 404 for an unknown project

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: the prompt migration lands as v3 (_migrate_v2_to_v3, CURRENT_MIGRATION_VERSION = 3) after AutoForgeAI#292's v2. Fixed a non-idempotent replacement (the testing-prompt note was appended again on every run, since its new text starts with the old one) and skip symlinked prompt files. New test_prompt_migrations.py: v3 converges pre-v3 template wording on the current templates, second run changes nothing, custom prompts untouched, validation/ + .gitignore entry created, v1 chain and new-project stamp; v2 idempotency and untouched prompts.

Integrates AutoForgeAI#289

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
…utoForgeAI#136)

Adaptations: the testing-prompt migration lands as v4 (_migrate_v3_to_v4, CURRENT_MIGRATION_VERSION = 4) after AutoForgeAI#289's v3. It now recognizes the default STEP 2 as v3 leaves it, with or without the browser-unavailable note (the original only matched raw v1 text, which v3 rewrites first), skips symlinked prompt files and shares _apply_replacements with v3. Restored the return of _migrate_v2_to_v3 lost at the merge boundary. AutoForgeAI#283's regression-script step and testing header now defer to the bash scenario script for non-UI features, and STEP 2 names .autoforge/prompts/app_spec.txt so new projects get no root spec copy. test_prompts.py merges AutoForgeAI#283's and AutoForgeAI#333's tests; v4 idempotency, untouched custom prompts and the full v1->v4 chain are in test_prompt_migrations.py.

Integrates AutoForgeAI#333

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: merged with AutoForgeAI#334 quality gates (evidence is built only after the gates pass, so a rejected call copies nothing), AutoForgeAI#245 manual testing and AutoForgeAI#300 in database.py/feature_mcp.py/client.py/templates; screenshot examples point at .autoforge/validation/ (Phase 3, AutoForgeAI#289). The unsandboxed MCP server no longer copies screenshots through a symlinked .autoforge/evidence path (agent-plantable), and copies that the stored evidence no longer references are deleted after each evidence write (re-tests and rejected calls used to accumulate copies). Tests RED->GREEN.

Integrates AutoForgeAI#264

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent validation screenshots are saved in unspecified locations

1 participant