Skip to content

Stop endless regression testing when no coding progress is possible - #244

Open
arkaccess wants to merge 1 commit into
AutoForgeAI:masterfrom
arkaccess:swarm/issue-13-ken
Open

arkaccess wants to merge 1 commit into
AutoForgeAI:masterfrom
arkaccess:swarm/issue-13-ken

Conversation

@arkaccess

Copy link
Copy Markdown

Summary

Most of the fix proposed in #13 is already on master: agent.py exits when all features pass, and the orchestrator breaks out of its loop on get_all_complete(). feature_get_next no longer exists, so the prompt change proposed in the issue doesn't apply anymore.

One version of the bug was still there. If every remaining feature was waiting for human input, had permanently failed, or was blocked by dependencies that can't be met (for example a dependency on a failed or missing feature), get_all_complete() returned False. The orchestrator then sat in its "No ready features available" branch. Every time a testing agent finished, _maintain_testing_agents() started another one, so the same passing features were regression-tested over and over. That is the same token drain the issue describes.

Changes

  • parallel_orchestrator.py
    • New has_coding_work(): returns True if a coding agent is running or a feature can be resumed or started.
    • _maintain_testing_agents() no longer starts testing agents when there's no coding work.
    • In the stalled branch (no ready features, no coding agents):
      • re-checks a fresh DB snapshot for features that just became ready;
      • waits for any testing agents still running, since a regression they find makes that feature ready again;
      • if features are waiting for human input, it keeps polling quietly with no agents. It prints the notice once instead of every 10s. The user can answer from the UI and the loop picks the feature up;
      • otherwise it exits and lists the failed and blocked features, with a hint to fix them and restart.
  • agent.py: the completion check now runs before every session, not only the first. Retries after an error or rate limit no longer start a session on a finished project.
  • test_orchestrator_completion.py: 10 new tests for has_coding_work, testing-agent spawning, and how run_loop exits or waits. Agent spawns are stubbed.

Verification

  • python -m pytest test_orchestrator_completion.py: 10 passed
  • python -m pytest test_client.py test_dependency_resolver.py test_rate_limit_utils.py: 55 passed
  • python test_security.py: all passed
  • ruff check .: clean
  • mypy on the changed files: no new errors. The errors it reports are already on master, in temp_cleanup.py and client.py.
  • No UI changes, so no browser check.

Assumptions

  • Exiting is the right move when nothing can progress without outside changes. Failure counts live in memory, so restarting the agent retries failed features.
  • Waiting for human input is the one stalled state where the orchestrator stays alive, because the UI can unblock it.
  • No --continuous-regression flag was added. Regression testing still runs alongside active coding exactly as before.

Closes #13

🤖 Generated with Claude Code

…sible

The orchestrator kept spawning regression testing agents forever when every
remaining feature was waiting on human input, permanently failed, or blocked by
unmet dependencies, re-testing the same passing features and burning tokens.

- Add ParallelOrchestrator.has_coding_work() and skip testing-agent spawns
  when it is False
- When stalled with no agents running: wait quietly (one notice) for features
  awaiting human input, otherwise exit with a summary of failed/blocked features
- agent.py: check for completion before every session, not just the first,
  so error/rate-limit retries don't start sessions on a finished project
- Add test_orchestrator_completion.py

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@arkaccess

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Ada (cubefarm agent, in a fresh session) · Round: 1 · Author: Ken

The PR fixes the part of #13 that was still broken: when every remaining feature is waiting for human input, has permanently failed, or is blocked by dependencies that can't be met, the orchestrator no longer starts regression testing agents over and over. It now exits with a list of failed and blocked features, or waits quietly without agents when a human can unblock it. agent.py also checks for completion before every session, so retries after an error no longer start sessions on a finished project. The new tests fail on master and pass on this branch, extra edge-case probes pass, and lint, type check and the test suite show no new failures.

Check Details
✅ Bug reproduced on master, fixed on PR Ran the PR's test_orchestrator_completion.py against origin/master's code: 3 failed there (testing agents keep starting when the only remaining feature waits for human input; the loop never exits when the only remaining feature depends on a missing feature; the human-input wait starts agents). The same tests pass on the PR.
✅ Exit when the rest is failed or blocked Probe test: feature #2 at MAX_FEATURE_RETRIES and #3 depending on #2. The loop exits, prints 'Failed after N attempts: #2' and 'Blocked by unmet dependencies: #3', and starts no testing agents.
✅ Dependency cycle Probe test: #2 and #3 depend on each other. The loop exits and lists both as blocked.
✅ Waiting for human input, then answered Probe test: the loop prints the waiting notice once and starts no agents. After needs_human_input is cleared in the DB from another thread, it starts coding feature #2 and testing agents start again.
✅ Waits for a running testing agent With a testing agent registered as running and the remaining feature blocked, the loop waits instead of exiting, since a regression the agent finds could make a feature ready again.
✅ agent.py completion check on retries Stubbed client whose session marks the last feature passing and returns an error. On master the agent ran 4 sessions (capped with max_iterations=4). On the PR it ran 1 session, printed 'ALL FEATURES COMPLETE!' and stopped.
✅ Regression testing during active coding unchanged Code review: has_coding_work returns True while coding agents run or features are resumable or ready, so _maintain_testing_agents behaves as before. The test test_spawns_while_coding_work_exists passes.
✅ Security and schema security.py and the localhost-only middleware are untouched, there is no DB schema change, and nothing new is written to logs beyond feature IDs.
✅ Backend boot and REST smoke test uvicorn on 127.0.0.1:5571: /api/health 200, /api/projects 200, path-traversal project name returns 404, project creation and the features endpoint work.
✅ UI renders a stalled project Loaded a project with 2 passing features, 1 waiting for input and 1 blocked. The dashboard, Kanban and graph views rendered with 0 console errors. The PR doesn't change the UI.
⏭️ UI Playwright e2e The PR doesn't touch the UI.
🧪 Commands run
Command Result
python -m pytest test_orchestrator_completion.py 10 passed
python -m pytest -q (full suite) 154 passed, 21 failed; the 21 are in test_devserver_security.py and master has the identical set
python -m pytest test_orchestrator_completion.py (on an origin/master checkout) 3 failed, 3 passed (bug reproduced)
python -m pytest qa probe tests (4 edge cases, not committed) 4 passed
python probe_agent.py (stubbed agent.py retry scenario, PR vs master) PR: 1 session then stop; master: 4 sessions (capped)
ruff check . All checks passed
mypy . 25 errors, the same set as on master
python test_security.py ALL TESTS PASSED
python test_security_integration.py ALL INTEGRATION TESTS PASSED
npm --prefix ui run build built successfully, no type errors
npm --prefix ui run lint 0 errors, 6 warnings
uvicorn server.main:app --host 127.0.0.1 --port 5571 + curl smoke tests health 200, projects 200, traversal 404; server stopped afterwards

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Agent continues indefinitely with regression tests after all features complete

2 participants