Skip to content

Add Manual Testing lane for code-complete features agents can't verify - #245

Open
zwanski-start wants to merge 2 commits into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-11-dennis
Open

zwanski-start wants to merge 2 commits into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-11-dennis

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #11

Problem

When the agent builds something it can't run (the reporter's case is a native iOS app with no Xcode or simulator), it can't verify features, so it never marks them passing. Those features stay unfinished, the user can't tell which ones are code-complete, and anything that depends on them stays blocked.

What changed

New "awaiting manual testing" state

  • api/database.py: new needs_manual_testing and manual_testing_notes columns, with a migration for existing databases.
  • mcp_server/feature_mcp.py: new MCP tool feature_mark_needs_manual_testing(feature_id, notes). The coding agent calls it when the code is done but it can't verify it. The tool clears in_progress and saves notes on what the human should test.
  • A feature awaiting manual testing can't be claimed again. Features that depend on it count as unblocked (get_satisfied_ids in api/dependency_resolver.py, used by the orchestrator, the MCP tools and the API), so the agent keeps working on everything else.
  • The orchestrator's get_all_complete counts these features as done for the agent, so a run doesn't loop forever on work only a human can finish.
  • feature_mark_passing and feature_mark_failing clear the flag. feature_get_stats reports a needs_manual_testing count.
  • The tool is in CODING_AGENT_TOOLS and is disallowed for the read-only assistant. The coding prompt template and the single and batch prompt headers explain when to use it.

API

  • GET /features returns a new needs_manual_testing list. The graph endpoint has a needs_manual_testing status.
  • POST /features/{id}/manual-test takes {result: "pass" | "fail", feedback?}:
    • pass marks the feature passing. It also works on any unfinished feature that no agent is working on, so users can sign off features stuck in Pending from before this change. It returns 409 if an agent is working on the feature.
    • fail (only for features awaiting manual testing) sends the feature back to Pending and writes the tester's feedback into manual_testing_notes. The next agent sees it in the feature details.

UI

  • A violet Manual Testing column appears on the Kanban board when any feature is awaiting manual testing.
  • The feature modal shows the agent's testing notes with Mark as Passed and Mark as Failed buttons. Failing asks for feedback first.
  • Unfinished features that no agent is working on get a Mark Passed button, with a confirmation step.
  • Cards, the graph view (node color, icon, minimap), progress totals, the celebration check and the dashboard all handle the new state.

Verification

  • Added test_manual_testing.py (12 tests). It covers dependency resolution, migrating a legacy DB, the list endpoint, pass, fail with feedback, a manual pass on an idle feature, rejected requests (400/409/422/404), and the MCP tool's claim guards, ready list, stats and clear-on-pass.
  • python -m pytest test_manual_testing.py test_client.py test_dependency_resolver.py test_rate_limit_utils.py all pass. test_security.py and test_security_integration.py pass. ruff check . is clean.
  • mypy . reports only errors that already exist on master (none from this change).
  • UI: npm run lint has no errors (6 warnings, all pre-existing) and npm run build succeeds.
  • Ran the server against a seeded project. GET /features put the seeded feature in needs_manual_testing and its dependent in pending without being blocked. I started browser checks in Playwright, but the browser tools disconnected before I got to the Manual Testing column, so the new UI hasn't been checked in a browser yet.

Assumptions and notes

  • Dependencies: a feature awaiting manual testing counts as satisfying its dependents. If it counted as unsatisfied, a mobile project would stall until a human tested everything. If a manual test later fails, the feature goes back to Pending.
  • Existing projects: they keep their copied .autoforge/prompts/coding_prompt.md, so their agents won't mention the new tool in that file. The tool is still exposed to them, and the batch and single-feature prompt headers (generated in prompts.py) do mention it.
  • MCP tests: they skip when mcp_server.feature_mcp can't be imported. requirements.txt doesn't cap mcp, and a fresh install pulls mcp 2.x, where mcp.server.fastmcp no longer exists. That's an existing problem; it needs mcp<2 pinned, which this PR doesn't do.
  • Push target: this branch is pushed from the zwanski-start fork because that account can't push to AutoForgeAI/autoforge.

🤖 Generated with Claude Code

zwanski-start and others added 2 commits October 22, 2026 03:33
…fy (AutoForgeAI#11)

Agents building apps they cannot run (e.g. native iOS) had no way to hand off
code-complete work, so features sat unfinished and invisible to the user.

- New needs_manual_testing / manual_testing_notes columns with migration
- MCP tool feature_mark_needs_manual_testing for coding agents
- Features awaiting manual testing satisfy dependencies and count as done
  for the orchestrator, so agents keep working on everything else
- Features API returns a needs_manual_testing list; graph status added
- POST /features/{id}/manual-test records a human pass/fail verdict
- Coding prompts explain when to use the hand-off

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Violet 'Manual Testing' column appears when features await manual testing
- Feature modal shows the agent's testing notes with Mark as Passed /
  Mark as Failed (feedback is sent back to the agent)
- 'Mark Passed' override for idle pending features, with confirmation
- Graph view, progress totals and completion checks include the new state

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Ken (cubefarm agent, in a fresh session) · Round: 1 · Author: Dennis

The PR adds a needs_manual_testing state for features the agent finished but couldn't verify, a Manual Testing Kanban lane, a POST /features/{id}/manual-test pass/fail endpoint, and a new MCP tool. Dependents of these features count as unblocked and the orchestrator treats them as done. Backend tests (the 12 new ones included), ruff, the security suites, UI lint and build all pass; mypy and the full pytest run fail only in the same places as master. In a real browser the lane, the testing notes, fail-with-feedback, pass and the manual sign-off of an idle feature all worked with no console errors. Small cosmetic notes: the sidebar progress badge only updates after a refresh, and the violet notes box is slightly too wide at 390px.

Check Details
✅ Issue #11: code-complete features show in a Kanban lane with a mark-passed button Seeded an iOS-style project with features #1 and #4 flagged needs_manual_testing. The Kanban showed a violet 'Manual Testing' column with 2 cards labelled 'Ready for Manual Testing'. The modal showed the agent's notes and Mark as Passed / Mark as Failed buttons. The column hides when empty.
✅ Fail flow returns the feature to Pending with tester feedback Clicked Mark as Failed, typed feedback and clicked Send Back to Agent. The feature moved to Pending and its modal showed 'Manual testing FAILED. Tester feedback: ...'. That text is stored in manual_testing_notes, which the agent receives in to_dict().
✅ Pass flow and manual sign-off of an idle pending feature Mark as Passed moved #1 to Done. On the pending #4 the footer 'Mark Passed' button asked for confirmation, then moved it to Done.
✅ Dependency resolution GET /features: #2 (depends on manual-testing #1) is pending and not blocked; #5 (depends on #2) is blocked. The graph endpoint gave #1 and #4 the status needs_manual_testing. The same satisfied-IDs helper is used by the orchestrator, the MCP server and the API.
✅ Orchestrator and agent handling Code review: get_all_complete counts manual-testing features as done; the ready-features list skips them; the claim and mark-in-progress SQL guard on needs_manual_testing=0; mark_passing and mark_failing clear the flag. All runs go through the orchestrator (non-auto-improve), so the agent can't loop on features only a human can finish. The tool is in CODING_AGENT_TOOLS and is blocked for the read-only assistant.
✅ REST validation and errors fail on a feature not awaiting testing -> 400; pass on an already-passing feature -> 400; result 'maybe' -> 422; missing result -> 422; feedback over 5000 characters -> 422; feature 999 -> 404; unknown project -> 404; pass while an agent has the feature in progress -> 409.
✅ Backward-compatible SQLite migration _migrate_add_manual_testing_columns adds both columns with ALTER TABLE ... DEFAULT 0/NULL. test_migration_adds_columns_to_legacy_db passes.
✅ Graph view Manual-testing nodes are violet with a clipboard icon, and the minimap is violet too. The legend has no entry for the new state, but it already lacked one for needs-input.
✅ Phone layout (390x844) The modal and its pass/fail buttons work at 390px, but the violet notes box is slightly too wide: its text and buttons spill past its border on the right (cosmetic). The board is squeezed by the always-expanded sidebar, which predates this PR.
✅ Progress totals The /api/projects stats were correct (3/5, 60%) after the passes. The sidebar badge stayed at 20% until refresh because the project list is cached and the new mutation doesn't refresh it. Other feature mutations on master behave the same way. Minor.
✅ Security not weakened security.py and the localhost middleware are unchanged; test_security and test_security_integration pass. No secrets are logged. Requests from 127.0.0.1 with a spoofed Host header were accepted, because the middleware checks the client IP; that is master behaviour and unrelated to this PR.
✅ Browser console 0 errors. 3-4 warnings, all Radix 'Missing Description for DialogContent', which also appear on master.
⏭️ UI e2e All 15 specs skipped themselves (they need a live project or assistant setup); none failed. Run with the system Chrome because the bundled headless shell couldn't be downloaded.
🧪 Commands run
Command Result
venv/bin/python -m pytest test_manual_testing.py test_client.py test_dependency_resolver.py test_rate_limit_utils.py -q 67 passed, 0 failed
venv/bin/python -m pytest test_manual_testing.py -v -rs 12 passed, 0 skipped (MCP tests ran with mcp 1.30.0)
venv/bin/python -m pytest -q (full) 156 passed, 21 failed; the 21 failures are all in test_devserver_security.py and also fail on master (master: 144 passed, 21 failed)
venv/bin/python test_security.py ALL TESTS PASSED
venv/bin/python test_security_integration.py ALL INTEGRATION TESTS PASSED
venv/bin/ruff check . All checks passed
venv/bin/mypy . (PR vs origin/master) 24 errors on both; same error set, no new errors
npm --prefix ui run lint 0 errors, 6 warnings (all pre-existing)
npm --prefix ui run build success (tsc -b && vite build)
uvicorn server.main:app --port 5345 (HOME sandboxed, no API key) + curl GET/POST /api/projects/iosapp/features[/graph|/{id}/manual-test] Correct buckets and graph statuses; 200/400/404/409/422 as expected
VITE_API_PORT=5345 npx playwright test (system Chrome via a throwaway config, deleted afterwards) 15 skipped, 0 failed

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: merged with AutoForgeAI#334 (quality gates) and AutoForgeAI#300 (regression_count) in database.py, client.py, feature_mcp.py, prompts and the coding template. User decision 08.11.2026: dependents of a feature awaiting manual testing stay blocked until a human passes it (get_satisfied_ids dropped), and the manual verdict endpoint/UI accepts only features in the Manual Testing lane (footer Mark Passed removed). Orchestrator waits (stall state human_input) instead of reporting a blocked queue, explains manual-testing blockers, does not chain a manual-testing feature into a batch, and no longer prints 'All features complete!' while features await manual testing. Migration also creates the needs_manual_testing index.

Integrates AutoForgeAI#245

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: none beyond listing the test in CLAUDE.md; merged on top of AutoForgeAI#245's FeatureModal changes.

Integrates AutoForgeAI#305

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: AssistantChat merged with my-features' messagesContainerRef scrolling; feature_report_bug also clears needs_manual_testing (AutoForgeAI#245) so a feature in the Manual Testing lane returns to the agents (test RED->GREEN), and writes with an in_progress = 0 guard so it cannot reset a feature an agent claimed after the check. AutoForgeAI#278's move-to-front not taken (optional in the plan).

Integrates AutoForgeAI#331

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: merged with AutoForgeAI#334 quality gates (evidence is built only after the gates pass, so a rejected call copies nothing), AutoForgeAI#245 manual testing and AutoForgeAI#300 in database.py/feature_mcp.py/client.py/templates; screenshot examples point at .autoforge/validation/ (Phase 3, AutoForgeAI#289). The unsandboxed MCP server no longer copies screenshots through a symlinked .autoforge/evidence path (agent-plantable), and copies that the stored evidence no longer references are deleted after each evidence write (re-tests and rejected calls used to accumulate copies). Tests RED->GREEN.

Integrates AutoForgeAI#264

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
Adaptations: merged with AutoForgeAI#245 (manual-testing/in-progress header, ClipboardCheck icon) and AutoForgeAI#264 (TestEvidencePanel) in FeatureModal/FeatureCard. UI-only; no JS unit test framework in the repo, verified in the browser in the Phase 4 pass.

Integrates AutoForgeAI#321

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Features go missing in the Kanban

1 participant