Repository navigation
Add Manual Testing lane for code-complete features agents can't verify - #245
zwanski-start wants to merge 2 commits into
Conversation
…fy (AutoForgeAI#11) Agents building apps they cannot run (e.g. native iOS) had no way to hand off code-complete work, so features sat unfinished and invisible to the user. - New needs_manual_testing / manual_testing_notes columns with migration - MCP tool feature_mark_needs_manual_testing for coding agents - Features awaiting manual testing satisfy dependencies and count as done for the orchestrator, so agents keep working on everything else - Features API returns a needs_manual_testing list; graph status added - POST /features/{id}/manual-test records a human pass/fail verdict - Coding prompts explain when to use the hand-off Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Violet 'Manual Testing' column appears when features await manual testing - Feature modal shows the agent's testing notes with Mark as Passed / Mark as Failed (feedback is sent back to the agent) - 'Mark Passed' override for idle pending features, with confirmation - Graph view, progress totals and completion checks include the new state Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🔍 QA report: ✅ PassedTester: Ken (cubefarm agent, in a fresh session) · Round: 1 · Author: Dennis The PR adds a needs_manual_testing state for features the agent finished but couldn't verify, a Manual Testing Kanban lane, a POST /features/{id}/manual-test pass/fail endpoint, and a new MCP tool. Dependents of these features count as unblocked and the orchestrator treats them as done. Backend tests (the 12 new ones included), ruff, the security suites, UI lint and build all pass; mypy and the full pytest run fail only in the same places as master. In a real browser the lane, the testing notes, fail-with-feedback, pass and the manual sign-off of an idle feature all worked with no console errors. Small cosmetic notes: the sidebar progress badge only updates after a refresh, and the violet notes box is slightly too wide at 390px.
🧪 Commands run
No browser screenshots were taken in this round. Posted by cubefarm · merges automatically once GitHub's checks pass |
Adaptations: merged with AutoForgeAI#334 (quality gates) and AutoForgeAI#300 (regression_count) in database.py, client.py, feature_mcp.py, prompts and the coding template. User decision 08.11.2026: dependents of a feature awaiting manual testing stay blocked until a human passes it (get_satisfied_ids dropped), and the manual verdict endpoint/UI accepts only features in the Manual Testing lane (footer Mark Passed removed). Orchestrator waits (stall state human_input) instead of reporting a blocked queue, explains manual-testing blockers, does not chain a manual-testing feature into a batch, and no longer prints 'All features complete!' while features await manual testing. Migration also creates the needs_manual_testing index. Integrates AutoForgeAI#245 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Adaptations: none beyond listing the test in CLAUDE.md; merged on top of AutoForgeAI#245's FeatureModal changes. Integrates AutoForgeAI#305 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Adaptations: AssistantChat merged with my-features' messagesContainerRef scrolling; feature_report_bug also clears needs_manual_testing (AutoForgeAI#245) so a feature in the Manual Testing lane returns to the agents (test RED->GREEN), and writes with an in_progress = 0 guard so it cannot reset a feature an agent claimed after the check. AutoForgeAI#278's move-to-front not taken (optional in the plan). Integrates AutoForgeAI#331 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Adaptations: merged with AutoForgeAI#334 quality gates (evidence is built only after the gates pass, so a rejected call copies nothing), AutoForgeAI#245 manual testing and AutoForgeAI#300 in database.py/feature_mcp.py/client.py/templates; screenshot examples point at .autoforge/validation/ (Phase 3, AutoForgeAI#289). The unsandboxed MCP server no longer copies screenshots through a symlinked .autoforge/evidence path (agent-plantable), and copies that the stored evidence no longer references are deleted after each evidence write (re-tests and rejected calls used to accumulate copies). Tests RED->GREEN. Integrates AutoForgeAI#264 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Adaptations: merged with AutoForgeAI#245 (manual-testing/in-progress header, ClipboardCheck icon) and AutoForgeAI#264 (TestEvidencePanel) in FeatureModal/FeatureCard. UI-only; no JS unit test framework in the repo, verified in the browser in the Phase 4 pass. Integrates AutoForgeAI#321 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Closes #11
Problem
When the agent builds something it can't run (the reporter's case is a native iOS app with no Xcode or simulator), it can't verify features, so it never marks them passing. Those features stay unfinished, the user can't tell which ones are code-complete, and anything that depends on them stays blocked.
What changed
New "awaiting manual testing" state
api/database.py: newneeds_manual_testingandmanual_testing_notescolumns, with a migration for existing databases.mcp_server/feature_mcp.py: new MCP toolfeature_mark_needs_manual_testing(feature_id, notes). The coding agent calls it when the code is done but it can't verify it. The tool clearsin_progressand saves notes on what the human should test.get_satisfied_idsinapi/dependency_resolver.py, used by the orchestrator, the MCP tools and the API), so the agent keeps working on everything else.get_all_completecounts these features as done for the agent, so a run doesn't loop forever on work only a human can finish.feature_mark_passingandfeature_mark_failingclear the flag.feature_get_statsreports aneeds_manual_testingcount.CODING_AGENT_TOOLSand is disallowed for the read-only assistant. The coding prompt template and the single and batch prompt headers explain when to use it.API
GET /featuresreturns a newneeds_manual_testinglist. The graph endpoint has aneeds_manual_testingstatus.POST /features/{id}/manual-testtakes{result: "pass" | "fail", feedback?}:passmarks the feature passing. It also works on any unfinished feature that no agent is working on, so users can sign off features stuck in Pending from before this change. It returns 409 if an agent is working on the feature.fail(only for features awaiting manual testing) sends the feature back to Pending and writes the tester's feedback intomanual_testing_notes. The next agent sees it in the feature details.UI
Verification
test_manual_testing.py(12 tests). It covers dependency resolution, migrating a legacy DB, the list endpoint, pass, fail with feedback, a manual pass on an idle feature, rejected requests (400/409/422/404), and the MCP tool's claim guards, ready list, stats and clear-on-pass.python -m pytest test_manual_testing.py test_client.py test_dependency_resolver.py test_rate_limit_utils.pyall pass.test_security.pyandtest_security_integration.pypass.ruff check .is clean.mypy .reports only errors that already exist on master (none from this change).npm run linthas no errors (6 warnings, all pre-existing) andnpm run buildsucceeds.GET /featuresput the seeded feature inneeds_manual_testingand its dependent inpendingwithout being blocked. I started browser checks in Playwright, but the browser tools disconnected before I got to the Manual Testing column, so the new UI hasn't been checked in a browser yet.Assumptions and notes
.autoforge/prompts/coding_prompt.md, so their agents won't mention the new tool in that file. The tool is still exposed to them, and the batch and single-feature prompt headers (generated inprompts.py) do mention it.mcp_server.feature_mcpcan't be imported.requirements.txtdoesn't capmcp, and a fresh install pulls mcp 2.x, wheremcp.server.fastmcpno longer exists. That's an existing problem; it needsmcp<2pinned, which this PR doesn't do.zwanski-startfork because that account can't push to AutoForgeAI/autoforge.🤖 Generated with Claude Code