Skip to content

Fix #13: Stop endless regression-testing sessions when no coding work remains - #296

Open
zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-13-ada
Open

zwanski-start wants to merge 1 commit into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-13-ada

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #13

Context

Most of the fix the issue proposes is already on master: agent.py exits when every feature passes, and the parallel orchestrator (now the only entry point) stops on get_all_complete(). The coding prompt no longer uses feature_get_next, so the prompt change the issue suggests doesn't apply anymore.

One path still produced the reported symptom: a stalled project. If nothing is ready and no coding agent is running (remaining features need human input, failed MAX_FEATURE_RETRIES times, or depend on such features or on a cycle), get_all_complete() returns False. The orchestrator then looped forever printing "No ready features available", and _maintain_testing_agents() kept spawning regression-only testing sessions. That is the endless token burn from the issue.

Changes

  • parallel_orchestrator.py
    • _maintain_testing_agents() only spawns testing agents while coding agents are running. With no code changing, more regression runs can't find anything new.
    • New get_stall_state() explains why nothing can start right now: work (the snapshot was stale), testing (wait for running testers, which may flip a feature back to failing), human_input, or blocked.
    • On blocked, the orchestrator exits with a clear message. On human_input, it waits without spawning any agents and prints the waiting notice once instead of every 10s.
  • agent.py: after every session (whatever its status), completion is read from the features DB. This replaces matching "all features are passing" / "no more work to do" in the agent's response text: no tool emits those strings anymore, and a model's own wording could trigger them by accident. It also replaces the later check in the continue branch, which ran only after "auto-continue in Ns" had already been printed.
  • test_orchestrator_completion.py: 9 new tests covering stall classification and when testing agents are spawned.

Verification

  • ruff check .: clean. mypy reports no errors in the changed files (the errors it shows elsewhere were already on master).
  • pytest test_orchestrator_completion.py test_client.py test_dependency_resolver.py test_rate_limit_utils.py: all pass. python test_security.py: all pass.
  • End-to-end: ran the real run_loop() on a temporary project with one passing feature and two pending features in a dependency cycle. It now prints "No remaining features can be started…" and exits with 0 testing agents spawned. Before this change, that setup looped forever and kept spawning testing agents.
  • No UI changes, so no browser check.

Assumptions

  • I treated a fully blocked project (no human input pending) as terminal and made it exit. Retry counters live in memory, so restarting the agent retries the failed features. Projects waiting on human input keep the orchestrator alive (idle, no agents) so it resumes once the user answers in the UI.
  • I didn't add a --continuous-regression flag. Nothing in the codebase relies on testing agents running without coding agents.

🤖 Generated with Claude Code

…coding work remains

- Orchestrator only spawns testing agents while coding agents are running,
  so a stalled project no longer burns tokens on repeated regression runs.
- When nothing is ready or running, classify the stall: exit if remaining
  features can never start (failed too often, cyclic/unsatisfiable deps),
  wait quietly (one notice) if they are waiting on human input.
- agent.py: detect completion from the features DB after every session
  instead of matching 'no more work to do' in the agent's response text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 1 · Author: Ada

The PR stops the endless regression-testing loop from #13. In a mocked run on master, a project whose remaining features form a dependency cycle never exits and spawned 10 testing agents in 4 seconds. On this branch it prints a clear message and exits with no testing agents, and a project waiting on human input stays alive without spawning agents. agent.py now reads completion from the features DB: an errored session where every feature passes now stops, and a response that merely says 'all features are passing' no longer ends the run early. Lint, the new tests and the security tests pass; the remaining mypy errors and the 21 test_devserver_security failures are the same on master.

Check Details
✅ Issue repro: stalled project no longer loops / burns tokens Ran the real ParallelOrchestrator.run_loop() on a temp project (passing #1, cycle #2<->#3) with agent spawning mocked. Master: still running after 4s with 10 testing agents spawned and 'No ready features available' printed every loop. PR: prints 'No remaining features can be started...' and exits with 0 testing agents spawned.
✅ Blocked by a permanently failed dependency #3 depends on #2, which has failed MAX_FEATURE_RETRIES times: the orchestrator exits with 0 testing agents spawned.
✅ All features passing Orchestrator prints 'All features complete!' and exits immediately with no testing agents.
✅ Waiting on human input Orchestrator stays alive, spawns no agents, and prints the waiting notice once instead of every poll. After needs_human_input is cleared in the DB, it starts #2 and #3 and finishes.
✅ Normal progress unaffected With ready features, start_feature is called for #2 then #3 (after its dependency passes) and the loop ends with 'All features complete!'. The unit test checks that testing agents still spawn alongside running coding agents.
✅ agent.py completion read from the DB Session mocked, max_iterations=3. Session ends in an error while every feature passes: master ran 3 sessions, PR stops after 1. Response text claims 'All features are passing! No more work to do.' while features still fail: master quit after 1 session, PR runs 3. Normal case (continue + all passing): 1 session on both.
✅ Code review (agent.py, parallel_orchestrator.py, new tests) The 'work' stall path re-runs the loop without waiting, but it re-reads the DB on every pass (expire_all), so it can't busy-spin. Auto-improve uses max_iterations=1, so the new post-session check has no effect there. No security, path or encoding changes, and no leftover debug code.
✅ Backend boots and REST routes respond uvicorn server.main:app on 127.0.0.1:5247: /api/health 200, /api/projects 200, /api/settings 200, /api/projects/does-not-exist 404.
✅ UI loads in a real browser Served SPA loaded in Playwright with 0 console errors or warnings. The PR doesn't touch the UI, so the UI build and e2e tests were not required.
🧪 Commands run
Command Result
venv/bin/python -m ruff check . All checks passed
venv/bin/python -m mypy . 24 errors in 9 files, the same count as master; none in the changed files
venv/bin/python -m pytest -q test_orchestrator_completion.py test_client.py test_dependency_resolver.py test_rate_limit_utils.py 64 passed
venv/bin/python -m pytest -q 153 passed, 21 failed (all in test_devserver_security.py; master: 144 passed, 21 failed, so the failures are pre-existing)
venv/bin/python test_security.py ALL TESTS PASSED
venv/bin/python test_security_integration.py ALL INTEGRATION TESTS PASSED
python scratchpad/probe.py (run_loop scenarios, mocked spawning) on PR and master PR: cycle and failed-dependency exit with 0 testing spawns, human-input waits then resumes after the answer; master: cycle never exits, 10 testing spawns in 4s
python scratchpad/probe_agent.py (run_autonomous_agent with mocked session) on PR and master PR: 1/1/3 sessions vs master: 1/3/1 (all-pass, error+all-pass, false-positive text)
venv/bin/python -m uvicorn server.main:app --host 127.0.0.1 --port 5247 + curl on REST routes 200/200/200/404 as expected; server stopped afterwards

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
…coding work remains

Adaptations: the ParallelOrchestrator docstring for testing_agent_ratio now says testing agents run only while a coding agent runs; new test_agent_completion.py covers the agent.py change (completion read from the database, not from the reply text).

Integrates AutoForgeAI#296

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
)

Adaptations: combined with AutoForgeAI#296's stall states, the report is printed for 'blocked' and 'human_input' stalls before the orchestrator stops (blocked) or waits (human input), replacing AutoForgeAI#296's one-line notices. Ported AutoForgeAI#255's 2-line rec_stack/path reset in _detect_cycles, which reported features that merely depend on a cycle as cyclic; new regression test. New test for the report's content and de-duplication.

Integrates AutoForgeAI#306

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Agent continues indefinitely with regression tests after all features complete

1 participant