Skip to content

subordinate_status and the console still read the raw status column — a supervisor waits on an agent the platform knows has stopped #589

Closed

Description

@serge-ivo

The health verdict reaches 2 of 5 loop-run readers; the three it misses are the supervisor and the owner

f01d5b4 (#580 AC3) added runHealth so a client would stop inferring liveness. It has exactly two production consumers (grep -rn "runHealth\|isStalled(\|waitClause" workers/api/src workers/mcp/src store | grep -v test → 8 hits, 2 call sites): routes/tools.ts:1046 and work-report.ts:203.

The five readers of agent_loop_runs:

reader applies verdict?
routes/tools.ts:1055,1063 (/loop) yes
agent-think.ts:451 → describeLoopRun yes
tool-registry.ts:231,247 (check_work) yes
connectors/supervision.ts:714 (check_delegation) no
lib/instance-work.ts:223 → subordinate_status no
store/console/src/lib/workInFlight.ts:29 + tabs/LoopRunsSection.tsx:96 no

Three answers, one run, same instant

For a run parked 4.35 h on an engine limit:

  • check_instance_loop → health:"waiting", waitNote:"WAITING, not stalled and not working…"
  • subordinate_status → activity:"working", quietForMinutes: 261
  • console "Autonomous runs" (LoopRunsSection.tsx:152) → step 1/40 · started 4h ago + a Stop button

Mechanism

  • subordinate-payload.ts:177 and :228 — activity = runs.some((r) => r.status === "running") ? "working" : "idle", the raw column.
  • subordinate-observation.ts:152 — quietForMinutes: minutesSince(r.lastProgressAt ?? r.startedAt, now), the column agent-loop-store.ts:69-74 explicitly calls "NOT a liveness signal, and it was read as one for two releases".
  • instance-work.ts:233-234 does not SELECT last_alive_at, waiting_reason, waiting_until — the supervision path cannot produce a verdict.

The description sells the wrong surface hardest (supervision.ts:550): "What every agent you supervise is doing RIGHT NOW… 'which ones are idle' is roster[].activity."

Two comments are now false

  • run-sweeper.ts:67 — "the same rule summarizeSubordinates uses, so the sweeper and the supervisor still agree on 'quiet'". The sweeper uses COALESCE(last_alive_at, last_progress_at, started_at) (:70); summarizeSubordinates uses lastProgressAt ?? startedAt.
  • workInFlight.ts:28 — "A run is live only while the server says running; every terminal state is a different word." Since fd1c323, running covers working, waiting and stalled.

Why this misleads into a wrong action

run-sweeper.ts:9-11 already names the victim: "subordinate_status reports it as work in flight, so a supervisor keeps waiting on an agent that stopped hours ago." A Coder Lead blocks on a dead subordinate; the owner sees a Stop button for a run that stopped itself.

Acceptance criteria

  1. Every reader of agent_loop_runs applies the same verdict, or explicitly declines to make a liveness claim. subordinate-payload.ts:177 is the single highest-value line.
  2. instance-work.ts's SELECT carries the columns a verdict needs.
  3. A guard asserts every reader of that table derives health from one implementation — denominator stated per ADR 0002 as readers-found, not a hand-listed set. Two surfaces deriving one fact independently is the A run whose engine died at step 1 reported 'running' for 4.35 hours with a fresh lastProgressAt, and wrote nothing to the trace #580 defect; this is the same defect at a different layer.
  4. The two false comments are corrected or deleted — a comment asserting an agreement that no longer holds is worse than none.
  5. Red demonstrated: a parked run must not be reported as working by subordinate_status.

Verified vs inferred

  • Verified: the grep and its counts; all six reader call sites; the three live answers; every quoted comment.
  • Inferred: nothing load-bearing.

Activity

  1. serge-ivo commented on Aug 15, 2026

    @serge-ivo
    ContributorAuthor

    Done in dc68e2a (with a wording correction in b4d6ea3). CI green on both; API + Host workers deployed successfully.

    Readers before and after

    The issue named 5 readers. The guard's own sweep — needles over workers/api/src, workers/mcp/src, store/console/src, agents/coder/web/src, packages/sdk/src — finds 25 loop-run readers over 576 source files. That gap is the point: the reader set is FOUND, not listed.

    before after
    readers that apply runHealth's verdict 2 every reader that makes a liveness claim
    readers that derive one from the raw status column 3 (subordinate-payload.ts ×2, subordinate-observation.ts, console) 0
    readers that compare status === "running" for OPENNESS — 9, all legitimate, none producing a liveness word

    AC by AC

    1. subordinate-payload.ts:177 → activityFromRuns(runs), over health. Same at :228 (rosterLines). subordinate-observation.ts now calls runHealth/waitClause — it already owned supervision's judgement, so this is a call rather than a second rule. check_delegation (supervision.ts:714) ships health/waitNote/healthLegend. The console reads the field the server was already sending.
    2. instance-work.ts's SELECT carries last_alive_at, waiting_reason, waiting_until. It genuinely could not form a verdict before; a guard arm asserts all six columns a verdict needs stay in that SELECT.
    3. Guard: workers/api/src/lib/run-health-readers.test.ts. Denominator per ADR 0002 — 8 needles, every one asserted non-empty (a renamed export fails the guard loudly instead of shrinking it silently), reader count floored at 15 with the reason beside the floor, and the offender set compared EXACTLY so removing the last one fails too. The rule: within the reader set, no file outside work-report.ts may put a status === "running" test and a liveness word on the same line. Openness tests stay legal — that distinction is why the console had one sentence for two questions.
    4. Both false comments corrected. run-sweeper.ts:67 claimed it shared a rule with summarizeSubordinates; they read different columns deliberately (work-report.ts:136-141), so the claim was arguing for making one of them worse — it now names the agreement that does hold, with runHealth. workInFlight.ts:28 is gone: the function is isOpen, and runActivity/activityLabel answer the other question.
    5. Red demonstrated on both arms. Reintroducing runs.some(r => r.status === "running") ? "working" : "idle" → workers/api/src/lib/subordinate-payload.ts:230: out.activity = …. Reverting the observation's verdict → 5 failures including expected 'working' to be 'waiting'.

    A vocabulary change the legends had to state

    activity gains waiting, stalled and unknown; only idle now means an agent is free for work. Each licenses a different next action, and folding waiting into idle would have been the same over-claim facing the other way. subordinate_status's description and the payload legend both say so.

    Verified vs not

    • Verified live: the deployed console bundle at proagentstore.online/console/ contains the four-member vocabulary and the new parked-run label — so the console half is running in production.
    • Verified in test: 9,016 tests green on the pushed tree, plus every CI gate (11 checks, biome, docs:drift, 6 typechecks, console build).
    • NOT observed live: the API-side activity value on a real roster. subordinate_status is an agent-side connector tool — reading it means messaging the instance, which is a state change. Recorded as a non-observation rather than assumed.
    • The conditions that distinguish the words are present in production right now: runs 70ea298e (PAS Coder, ~5.1h) and 0cbf401f (Heartfull, ~5.4h) are both sitting in notify-coding-wait. 70ea298e is the run this issue was filed about.

    Closing: every AC is delivered and guarded. If the roster is later observed still saying working for a parked subordinate, that is a new fact and worth reopening on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions