Skip to content

Parallel Scheduling: One Turn Per Phase and Dependency-Aware Fan-Out - #113

Open
suprie wants to merge 3 commits into
mainfrom
run/parallel-scheduling-one-turn-per-phase-plus5-d13a619fe69b
Open

suprie wants to merge 3 commits into
mainfrom
run/parallel-scheduling-one-turn-per-phase-plus5-d13a619fe69b

Conversation

@suprie

@suprie suprie commented Oct 30, 2026

Copy link
Copy Markdown
Owner

Summary

The run queue no longer merges every queued phase into one LLM turn. prd-parallel-scheduling makes the phase the unit of execution: each run-plan phase runs in its own executor turn, in its own worktree and branch cut from the default branch, with its own provider session, token budget, and wall-clock watchdog. A new dependency-aware scheduler validates the plan graph, holds dependents in Waiting until every upstream lands green, then fans out ready phases concurrently — four by default, configurable per plan — while a non-green upstream fails only its dependents and leaves independent in-flight phases running. Restart reconciliation now handles multiple in-flight phases, auto-resuming only when the worktree and saved provider session both survive and parking the rest with a reason for the morning report.

What changed

  • Add scheduler.rs: graph validation (cycles, self-edges, dangling edges), Waiting gate release/fail, ready-set selection, per-phase watchdog, restart classification, and generated-graph property tests.
  • Rewrite the run executor: one PhaseTask and one turn per phase, a JoinSet bounded by RunPlan.max_concurrency, per-phase worktrees, per-phase token/wall-clock budgets, and serialized plan bookkeeping.
  • Add RunPhaseStatus::Waiting, RunPlan.max_concurrency (default 4), and per-phase worktree/session fields to run-plan.yaml, with legacy plans loading unchanged.
  • Extend restart reconciliation to auto-resume recoverable in-flight phases via a resume-aware session spawn.
  • Seed upstream handoff artifacts into each dependent phase's worktree so dependents never read an unmerged sibling branch.
  • Expose a concurrent-phases control in the plan-tonight wizard and surface waiting in the run surfaces.
  • Add the six loop artifacts under .loopdeck/handoff-artifacts/.

PRD

docs/epics/role-based-orchestration/prd-parallel-scheduling.md

Decisions

  • 2026-10-30 — Execute run-plan phases one turn each with a ready-set scheduler (accepted): each phase runs in its own worktree and turn under max_concurrency, with a Waiting gate and fail-dependents policy.
  • 2026-10-10 — Use write-capable Codex policy for unattended runs (accepted): autonomous turns use workspaceWrite with on-request, subject to the destructive-command floor.
  • 2026-10-09 — Correlate Codex turn failures across the run watchdog (accepted): lifecycle logs now distinguish tool hangs, protocol failures, watchdogs, and cleanup failures.

Test plan

  • cargo test (from src-tauri/)
  • cargo clippy -- -D warnings (from src-tauri/)
  • npx tsc --noEmit
  • npm run test:frontend
  • Manual: queue a two-phase plan with disjoint dependencies and confirm two phase worktrees run at once, each opening its own draft PR.

Verify Verdict

PRD Verification — prd-parallel-scheduling.md

Verdict: PASS

# Criterion Status Evidence
1 One executor turn per run-plan phase, replacing the combined batch (P0) PASS src-tauri/src/commands/run_queue.rs:2218 (execute_run) spawns one PhaseTask per ready phase; :1803 builds the prompt from a single-loop slice; next_queued_batch and the combined-batch turn are deleted
2 Concurrent execution of phases whose depends_on are satisfied and disjoint, each in its own worktree, under a bounded concurrency limit (P0) PASS src-tauri/src/scheduler.rs:304 (select_ready); run_queue.rs:2239 (JoinSet), :2293 (max_concurrency), :202 (each branch cut from the default branch); default 4 in limits.rs:114; picker PlanTonightWizard.tsx:659
3 Gated starts: dependent phases hold in Waiting until upstream completes, then release or fail per the stall policy (P0) PASS runplan.rs:54 adds Waiting; scheduler.rs:153 (mark_waiting), :186 (reconcile_gates), :272 (fail_dependents leaves Running phases alone); gate edges in transition.rs
4 Restart reconciliation covering multiple in-flight phases (P1) PASS run_executor.rs:582 iterates every Running phase; scheduler.rs:354 classifies resume vs park on worktree+session survival; the resumed turn passes the saved id through agent.rs:1947
5 Per-phase token and wall-clock budgets per in-flight phase, not per batch (P1) PASS run_queue.rs:1705 creates one TokenBudget per PhaseTask; scheduler.rs:333 (phase_watchdog); run_queue.rs:1991 kills only that phase and fails its dependents
6 Add scheduler property tests (diamond graphs, cycle rejection, mid-flight crash recovery) and worktree cleanup race checks (Phase 4 loop) PASS scheduler.rs:721/:773/:799 generated-DAG, cycle, and restart properties; run_executor.rs:1006/:1520 crash recovery; run_queue.rs:3216 disjoint-worktree cleanup check
7 Must reject cycles at plan creation (Design) PASS run_queue.rs:636 validates before persisting; scheduler.rs:69 rejects cycles, self-edges, and dangling edges; also checked at run start (run_queue.rs:2232)

Non-goals audit

  • No scope creep detected. Cron/delays, cross-project scheduling, the one-run-per-project lock, and branch auto-merge are untouched.

Roll-up: PASS — 0 FAIL, 0 PARTIAL, 7 PASS.

Run metadata

  • Phases:
    • parallel-scheduling/one-turn-per-phase — "Replace the combined queued-batch turn with one executor turn per run-plan phase" (role-based-orchestration / prd-parallel-scheduling / Phase 1 — Per-phase turn execution)
    • parallel-scheduling/parallel-batches — "Run phases with satisfied, disjoint depends_on concurrently in separate worktrees under a bounded concurrency limit" (role-based-orchestration / prd-parallel-scheduling / Phase 2 — Dependency-aware fan-out)
    • parallel-scheduling/gated-start — "Hold dependent phases in Waiting until upstream phases complete, releasing or failing per the stall policy" (role-based-orchestration / prd-parallel-scheduling / Phase 2 — Dependency-aware fan-out)
    • parallel-scheduling/restart-reconcile — "Extend restart reconciliation to multiple in-flight phases after a crash or restart" (role-based-orchestration / prd-parallel-scheduling / Phase 3 — Reconciliation + budgets)
    • parallel-scheduling/budget-split — "Apply per-phase token and wall-clock budgets per in-flight phase instead of per combined batch" (role-based-orchestration / prd-parallel-scheduling / Phase 3 — Reconciliation + budgets)
    • parallel-scheduling/tests — "Add scheduler property tests (diamond graphs, cycle rejection, mid-flight crash recovery) and worktree cleanup race checks" (role-based-orchestration / prd-parallel-scheduling / Phase 4 — Tests/verification)
  • Budgets: 3000000 tokens total · 32400s wall-clock cap for this turn · 28800s total-run wall-clock cap

suprie and others added 2 commits October 30, 2026 13:36
The run queue no longer merges every queued phase into one LLM turn. Each run-plan phase runs in its own executor turn, in its own worktree and branch cut from the default branch, with its own provider session, token budget, and wall-clock watchdog. A new dependency-aware scheduler validates the plan graph, holds dependents in Waiting until every upstream lands green, then fans out ready phases concurrently — four by default, configurable per plan — while a non-green upstream fails only its dependents and leaves independent in-flight phases running. Restart reconciliation now handles multiple in-flight phases, auto-resuming only when the worktree and saved provider session both survive and parking the rest with a reason.

Rubric: PASS — 7/7 criteria

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@suprie
suprie marked this pull request as ready for review October 30, 2026 06:39
Integrate bounded artifact injection and citation attribution into isolated
phase turns. Reconcile waiting-only restarts, unify Waiting states and
transitions, and retain the dependency-aware scheduler.

Validated: cargo test (762 passed, 9 ignored), cargo clippy -- -D warnings,
TypeScript type-check, 34 frontend tests, cargo fmt, and git diff --check.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant