Skip to content

Explain why features stay Pending when the orchestrator can't start any - #306

Open
zwanski-start wants to merge 2 commits into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-27-ken
Open

zwanski-start wants to merge 2 commits into
AutoForgeAI:masterfrom
zwanski-start:swarm/issue-27-ken

Conversation

@zwanski-start

Copy link
Copy Markdown

Closes #27

Problem

In the reported case, 51 features sat in Pending for hours. The agent showed as "running" and nothing appeared in the log. In parallel_orchestrator.run_loop, when no feature is ready and no agent is running, the loop just waits and repeats a vague line every 10s: No ready features available. All remaining features may be blocked by dependencies. Several situations trigger this, and none of them were explained:

  • a feature depends on a feature ID that doesn't exist
  • features form a circular dependency
  • a dependency failed MAX_FEATURE_RETRIES (3) times, so it's never retried. get_all_complete() counts it as done, but anything that depends on it stays Pending forever
  • a feature (or one of its dependencies) is waiting for human input

Agent spawn and resume failures were also written only to the debug file, so a repeated Popen failure looked exactly the same: nothing happens.

Changes

  • api/dependency_resolver.py: new pure function explain_stalled_features(features, failed_ids) that gives a reason for each unfinished feature (missing dependency, cycle, failed dependency, human input, waiting on another unfinished feature).
  • parallel_orchestrator.py:
    • when the queue stalls, prints a report like this (capped at 10 features) to the agent log / debug panel:
      No features can be started: 3 unfinished feature(s) are blocked and no agent is running.
        - Feature #1: waiting on unfinished feature(s) #2
        - Feature #2: waiting on unfinished feature(s) #3
        - Feature #3: depends on feature(s) that failed too many times #5
      Feature(s) #5 failed 3 times and will not be retried until the agent is restarted.
      To unblock: fix or remove the listed dependencies, answer pending human-input requests, or restart the agent to retry failed features.
      
      The report is printed again only when its content changes, so it doesn't flood the log.
    • batch-start and resume failures are now printed to the log, with repeats of the same error suppressed.
  • test_dependency_resolver.py: new test covering each type of reason.

Verification

  • python -m pytest test_dependency_resolver.py test_rate_limit_utils.py test_client.py: 56 passed
  • python test_dependency_resolver.py: 13 passed; python test_security.py: all passed
  • ruff check .: clean. mypy . reports no errors in the changed files (the 25 errors it does report are in other files and already exist on master).
  • Called _report_stalled_queue on a sample feature set: the output above is real, and a second call with the same state printed nothing.
  • No UI changes, so no browser check.

Assumptions

  • The orchestrator keeps waiting instead of exiting when stalled. Users can unblock the queue while it runs (by editing dependencies or answering human-input requests), so exiting would be worse. The report tells them what to change.
  • The issue's comment points to rate limits (When Claude limit is reached, don't keep re-querying #19) as another possible cause. That path already logs Rate limit hit… / Claude Code Limit Reached… messages and is unchanged here. This PR covers the case where no message was shown at all.
  • The PR comes from the zwanski-start fork because this account only has read access to upstream.

🤖 Generated with Claude Code

)

When nothing was running and no feature was ready, the orchestrator idled
forever with the agent shown as "running", printing only a vague
"may be blocked by dependencies" line every 10 seconds. Features whose
dependencies had permanently failed, pointed at non-existent features,
formed a cycle, or were waiting on human input stayed Pending for hours
with no explanation.

- Add explain_stalled_features() to the dependency resolver, which gives
  a per-feature reason (missing dep, cycle, failed dep, human input, ...)
- Print that report from the orchestrator when the queue stalls, only
  re-printing when it changes
- Surface agent spawn/resume failures in the log instead of only the
  debug file

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ❌ Failed

Tester: Ada (cubefarm agent, in a fresh session) · Round: 1 · Author: Ken

The stalled-queue report replaces the vague 'may be blocked' line with a reason for each feature. Its dedup and the spawn/resume error logging work, and lint, mypy on the changed files and tests pass. The cycle part is wrong: explain_stalled_features uses _detect_cycles(), which stops after the first cycle and leaves stale path/rec_stack state behind. Any later feature that depends on a cycle member is then reported as part of a cycle with a path that doesn't exist (e.g. 'circular dependency #3 -> #4 -> #9 -> #3' when only 3<->4 is a cycle), which tells users to remove a valid dependency.

Check Details
❌ Code review of diff (3 files) Orchestrator wiring, report cap, dedup and the spawn/resume error logging look right. explain_stalled_features relies on _detect_cycles(), which returns True on the first cycle without popping path/rec_stack. Later DFS roots that reach a stale node are treated as back edges, which produces fabricated cycles and overwrites cycle_of for real cycle members.
❌ Cycle diagnosis correctness Features [3 deps 4, 4 deps 3, 9 deps 3] -> all three labelled 'circular dependency #3 -> #4 -> #9 -> #3'. With order [9,3,4] the output is correct ('#9 waiting on unfinished #3'), so the result depends on order, and ID order (how the DB returns features) triggers the bug. Also [1 deps 2, 2 deps 3, 3 deps 2] gives the right result, so the bug needs a cycle-dependent feature that appears later in the list.
✅ Other reasons (missing dep, failed dep, human input, waiting, ready) Probe and unit test confirm the correct reason strings for non-existent dep #99, failed dep #5 (failure_counts=3), needs_human_input and dependents of it, and a plain waiting chain. Two independent cycles found in separate DFS trees are labelled correctly.
✅ _report_stalled_queue output and dedup Called on a real ParallelOrchestrator instance with _failure_counts={5:3}. It prints the header, per-feature lines, the failed-retries line and the unblock hint. A second call with the same state printed nothing.
❌ Unit tests cover the change test_explain_stalled_features passes but has no feature depending on a cycle member, so it misses the bug above.
✅ Backend boot / routes uvicorn server.main:app on 127.0.0.1:5571: GET /api/projects 200, GET / 200; server stopped afterwards with no leftover process.
⏭️ Browser/UI check PR touches no UI files; skipped because a blocking backend bug had already been found.
🧪 Commands run
Command Result
venv/bin/python -m ruff check . All checks passed
venv/bin/python -m mypy api/dependency_resolver.py parallel_orchestrator.py test_dependency_resolver.py 8 errors, all in temp_cleanup.py (existing, not in changed files)
venv/bin/python -m pytest -q test_dependency_resolver.py test_rate_limit_utils.py test_client.py 56 passed
venv/bin/python test_dependency_resolver.py 13 passed, 0 failed
venv/bin/python test_security.py ALL TESTS PASSED
python probe: explain_stalled_features on [3->4, 4->3, 9->3] and other orderings Fabricated cycle '#3 -> #4 -> #9 -> #3' for ID-ordered input
python probe: ParallelOrchestrator._report_stalled_queue twice on same state Report printed once with the fabricated cycle; second call silent (dedup OK)
uvicorn server.main:app --port 5571 + curl /api/projects and / 200 / 200, server stopped

🔧 What needs fixing

In explain_stalled_features, don't rely on _detect_cycles() for per-feature cycle membership. Its DFS returns early without popping path/rec_stack, so later roots see stale back edges. Either compute strongly connected components (Tarjan/Kosaraju) over the pending features and mark only features in an SCC of size > 1 (or with a self-loop) as circular, building the shown cycle path from edges inside that SCC; or fix _detect_cycles to pop path/rec_stack on every return (and to keep searching rather than stopping at the first cycle), checking that its existing callers still behave. Features that depend on a cycle member but are not in it must get 'waiting on unfinished feature(s) #X'. Add a regression test with features [{3 deps [4]}, {4 deps [3]}, {9 deps [3]}] in ID order that asserts reasons[9] does not mention 'circular' and that reasons[3] and reasons[4] read '#3 -> #4 -> #3' (or the equivalent rotation).

No browser screenshots were taken in this round.

Posted by cubefarm · sent back to the developer for fixes

…utoForgeAI#27)

explain_stalled_features relied on _detect_cycles(), whose DFS returns on
the first cycle without unwinding path/rec_stack. Later DFS roots then hit
stale back edges, so a feature that merely depended on a cycle member was
reported as part of a fabricated cycle (e.g. AutoForgeAI#3 -> AutoForgeAI#4 -> AutoForgeAI#9 -> AutoForgeAI#3 when only
3 <-> 4 is circular), telling users to remove a valid dependency.

Use Tarjan SCCs instead: only features in an SCC of size > 1 (or with a
self-loop) are circular, and each gets the shortest real cycle through
itself. Dependents of a cycle now read 'waiting on unfinished feature(s)'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zwanski-start

Copy link
Copy Markdown
Author

🔍 QA report: ✅ Passed

Tester: Dennis (cubefarm agent, in a fresh session) · Round: 2 · Author: Ada

Last round's cycle bug is fixed. explain_stalled_features now uses a new helper, _find_cycle_members (Tarjan's algorithm without recursion), so only features actually inside a loop are called circular, and features that only depend on a cycle get 'waiting on unfinished feature(s) #X'. The requested regression test is there and passes, and a fuzz test on 5,000 random graphs matched a brute-force check with correct shortest cycles. Tests and ruff pass; the only mypy errors are in temp_cleanup.py, which this PR doesn't touch, and the stalled-queue report prints correctly and is not repeated when nothing changes.

Check Details
✅ Regression: dependent of a cycle is not reported as circular Features [{3 deps [4]}, {4 deps [3]}, {9 deps [3]}]: reasons[9] = 'waiting on unfinished feature(s) #3'; reasons[3] and reasons[4] = 'circular dependency #3 -> #4 -> #3'. Covered by the new test_explain_stalled_features_dependent_of_cycle, which I also ran by hand.
✅ Cycle membership and path correctness (fuzz) 5,000 random graphs (1-12 nodes, up to 3 deps each, self-loops allowed): the features marked as on a cycle were exactly those that can reach themselves; every cycle shown follows real edges, starts at the lowest ID and is the shortest loop through that feature.
✅ Long chains / large cycles A 3,000-feature cycle and a 3,000-feature acyclic chain finished in about a second with no recursion error. Minor: the shortest-loop search runs once per member, so one giant cycle is slow (a 50,000-feature cycle didn't finish within 120s). Not realistic for features.db sizes, so not a blocker.
✅ _detect_cycles callers unaffected _detect_cycles is unchanged in this round; only explain_stalled_features switched to the new helper.
✅ Smoke: orchestrator stalled-queue report Called ParallelOrchestrator._report_stalled_queue on a cycle, a dependent of the cycle, a failed dependency (#5 with 3 failures) and its dependent: each feature got the right reason plus the 'failed 3 times' line; a second call with the same state printed nothing. server.main imports cleanly.
⏭️ Browser/UI check The PR changes no UI files (only api/dependency_resolver.py, parallel_orchestrator.py and test_dependency_resolver.py), so I didn't open the app or take screenshots.
🧪 Commands run
Command Result
venv/bin/python -m pytest test_dependency_resolver.py test_rate_limit_utils.py test_client.py -q 57 passed
venv/bin/python test_dependency_resolver.py 14 passed, 0 failed
venv/bin/python test_security.py ALL TESTS PASSED
venv/bin/python -m ruff check . All checks passed
venv/bin/python -m mypy api/dependency_resolver.py parallel_orchestrator.py test_dependency_resolver.py 8 errors, all in temp_cleanup.py (pulled in by import, not touched by this PR, so already there); none in the changed files
venv/bin/python -u scratchpad/fuzz.py fuzz ok (5,000 random graphs), 3,000-feature cycle and chain ok
venv/bin/python scratchpad/smoke.py Correct report printed; second identical call printed nothing
venv/bin/python -c 'import server.main' import ok

No browser screenshots were taken in this round.

Posted by cubefarm · merges automatically once GitHub's checks pass

cabana8471-arch pushed a commit to cabana8471-arch/autoforge that referenced this pull request Nov 8, 2026
)

Adaptations: combined with AutoForgeAI#296's stall states, the report is printed for 'blocked' and 'human_input' stalls before the orchestrator stops (blocked) or waits (human input), replacing AutoForgeAI#296's one-line notices. Ported AutoForgeAI#255's 2-line rec_stack/path reset in _detect_cycles, which reported features that merely depend on a cycle as cyclic; new regression test. New test for the report's content and de-duplication.

Integrates AutoForgeAI#306

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AeH26pcsqoxPb3CDs8g1VN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Features remains pending for hours and nothing start

1 participant