Spec: view subscription Plan usage and route Steps across Harnesses #583

Settled in a grill-with-docs session: #2178 (Q1.1–Q3.4, ADR 0031).

Problem Statement

When a coding subscription hits a Plan window limit or spend cap, unattended thirdshift work fails even when another paid-for subscription Harness on the same machine still has allowance available:

• Long Spec runs can land every Ticket on one Harness and then fail on the final Spec review or Delivery when a weekly window or monthly spend cap runs out, branding the issue agent-failed and counting toward the repository's Failure limit even though nothing is wrong with the issue.
• Scheduled Passes keep firing against an exhausted subscription, burning startup attempts or failing Ready issues in a loop.
• Conversely, running reactive-only without an optional reserve ceiling drains 100% of the primary subscription overnight and leaves nothing for interactive Day shift work in the morning, while provider credit balances (such as Codex credits or Antigravity personal Google AI credits) can be spent silently before switching to another subscription.
• Meanwhile, the Day shift has no single place in the Console or CLI to see current Plan usage across the Fleet's shared subscription logins, which Plan windows are near or at their ceiling, when they reset, and which configured route each machine will use next.

Solution

Introduce Plan usage (composed of Plan windows and any spend cap) across the five subscription Harnesses—claude, codex, muse, agy and grok—leaving API-key Harnesses (opencode and kimi) out of subscription monitoring:

1. Passive + official non-billed observation: Every running session passively updates its machine's local Plan usage state as structured Harness wire-frame events arrive (including Claude rate_limit_event frames at all utilization levels, Codex session token_count events, and terminal UsageLimit rejections across all five subscription Harnesses). On demand (with short TTL caches, never on an idle Console timer), thirdshift enriches local state with official zero-token CLI/RPC reads where supported (codex app-server account/rateLimits/read, muse serve usage/read and account/read, and claude auth status --json for plan/account identity), never scraping undocumented web/OAuth endpoints or touching Harnesses' private auth tokens.
2. CLI and Console visibility (thirdshift plan-usage and /plan-usage): A dedicated CLI command (--fleet, --machine <name>, --json, --clear [<harness>]) and Console page (/plan-usage, /api/v1/plan-usage, defaulting to Fleet when peers exist) show configured routes and observed subscription Harnesses, grouping shared logins across Fleet machines by account identity when known, caching offline peer snapshots, piggybacking local Plan usage pushes onto post-Command console report, and surfacing a Console header warning pill whenever a configured route is exhausted or at/above its reserve ceiling.
3. Proactive + reactive per-Step routing: Under [harness] (routes, max_utilization, per-route allow_credits) and [security] (routes), factory Commands using User config defaults select a route per session Step from machine-local Plan usage—routing proactively around cooldowns and max_utilization ceilings (treating included Plan windows as the boundary ahead of credit pools by default, including --settings '{"useG1Credits":false}' on headless agy launches), and routing reactively within a Step when a session stops on structured provider Plan-usage evidence. Mid-Step fallback keeps local commits, discards uncommitted worktree edits cleanly, and starts a fresh session on the next route with zero added quota-anxiety prompt text. Explicit CLI harness/model flags, Step harness pins, Experiments and Scoring runs stay pinned, and cybersecurity safeguard refusals still never route and still hold Self-merge.
4. Clean exhaustion without failing issues: When every allowed route is unavailable, scheduled Passes skip quietly (one Activity log line per kind on reason change, plus a status banner on 1 Inbox and thirdshift status), and live Commands end as Paused Commands (outcome = "paused"), stopping queued sibling Spec Tickets while active ones finish or salvage, and releasing the issue's Claim without agent-failed (restoring ready-for-agent subject to the Trusted account edit check, except Security fixes). A Paused Command that restored ready-for-agent sits in Done rather than Needs you and saves its Run notification without emailing under only_unmerged, because Pickup continues it automatically once a Plan window resets.

User Stories

1. As the Day shift, I want thirdshift plan-usage and /plan-usage to show each configured route and observed subscription Harness with its Plan windows, utilization, reset times and cooldown status, so that I can see available subscription headroom at a glance.
2. As the Day shift, I want thirdshift plan-usage and /plan-usage to default to --fleet when Fleet peers exist and group shared subscriptions across machines by account identity (claude, codex, muse), showing the freshest observation and naming the machines sharing that login, so that one shared subscription is not repeated six times or shown with stale per-host numbers.
3. As the Day shift, I want distinct subscription logins or unknown-identity Harnesses (agy, grok) kept as separate rows per machine/account, so that employer or secondary accounts are never conflated with my personal login.
4. As the Day shift, I want NATted machines that push via console report to include their local Plan usage snapshot after each Command, and reachable Consoles to cache last-seen peer snapshots when a Fleet host is offline, so that viewing from my phone reflects laptop VM usage too.
5. As the Day shift, I want a warning pill in the Console header whenever any configured route is exhausted or at/above max_utilization, linking to /plan-usage, while keeping 9 Resources strictly for per-host RAM, CPU and disk.
6. As the Day shift, I want /plan-usage and thirdshift plan-usage to omit subscription Harnesses that are neither configured in routes, previously observed, nor signed in on the queried machine/Fleet, while visibly surfacing any configured route whose CLI is missing or signed out, so that headless servers without unused CLIs stay uncluttered without hiding misconfiguration.
7. As the Day shift, I want Claude rate_limit_event stream frames recorded passively at all utilization levels (not only >= 85%) alongside Codex token_count frames and terminal UsageLimit rejections across claude, codex, muse, agy and grok, so that every session keeps local Plan usage fresh for free.
8. As the Day shift, I want active non-billed reads (codex app-server, muse serve, claude auth status) to run on demand with short TTL caches (~60s when viewing, ~5m or past resets_at before routing) and never on an idle Console background timer, so that an idle Console stays near-zero CPU and network.
9. As the Day shift, I want opencode and kimi left out of subscription Plan-usage monitoring, and thirdshift never to scrape undocumented web/OAuth endpoints or touch Harnesses' private auth tokens, so that API-billed Harnesses and private credentials are untouched.
10. As the Day shift, I want to configure an ordered harness.routes list (supporting both "claude" strings and inline tables with harness, model, effort, max_utilization and allow_credits) and an optional harness.max_utilization ceiling in User config, editable through thirdshift config and Console /config, so that I can hold back a Day shift reserve (such as 85%) and route across Harnesses or same-Harness model pools.
11. As the Day shift, I want harness.routes = [] (the default) to preserve single-Harness behavior on harness.default with max_utilization = 100, so that reserves and fallbacks are strictly opt-in and never route to an unlisted Harness or paid API credential.
12. As the Day shift, I want included Plan windows treated as the routing boundary ahead of Codex credit balances and Antigravity personal Google AI credits (passing --settings '{"useG1Credits":false}' on headless agy launches) unless a route explicitly sets allow_credits = true, so that unattended factory runs use my other paid subscriptions before burning extra credits.
13. As the Day shift, I want [security] to support an optional security.routes list and otherwise walk harness.routes when security.harness is plan-limited or fails before first turn, while never routing away from a cybersecurity safeguard refusal (which still holds Self-merge), so that Security reviews and Security runs keep working under plan limits without weakening ADR 0021's safety hold.
14. As the Day shift, I want factory Commands using User config defaults to route per session Step—leaving completed Steps and landed Spec Tickets untouched and starting a fresh session on the new Harness (never a cross-Harness Resume)—so that a Spec run whose 17 Tickets landed on Claude can finish its Spec review and Delivery on Codex.
15. As the Day shift, I want explicit CLI harness/model flags, explicit Custom-process Step harness pins, Experiments and Scoring runs to stay pinned without silent cross-Harness routing, so that targeted manual runs and Evals remain faithful to what was asked.
16. As the Day shift, I want reactive mid-Step fallback (before_first_turn == false) to keep local commits, discard uncommitted worktree edits cleanly (switching Implement from fresh to continuation once commits or an open PR exist, and aborting plus re-applying any pinned conflict merge on conflict Repair), link every configured route's Factory skills, and add zero quota-warning text to the Session prompt, so that the next Harness starts from clean git state without rushing over quota anxiety.
17. As the Day shift, I want routing decisions to stay strictly machine-local (no Fleet HTTP calls during Pass or Step admission), learning shared-login exhaustion on passive-only Harnesses via ~1s before_first_turn rejections and revalidating expired or timestamp-less cooldowns with exponential backoff (1h → 2h → 4h → 6h cap) plus manual thirdshift plan-usage --clear [<harness>], so that overlapping weekly windows and monthly spend caps recover safely without hammering depleted accounts.
18. As the Day shift, I want scheduled Passes to skip quietly when every allowed route on the machine is unavailable (logging once on reason change and showing a status banner on 1 Inbox and thirdshift status), so that cron never loops or fails Ready issues while subscriptions are exhausted.
19. As the Day shift, I want a live factory Command whose allowed routes are all exhausted to end immediately as a Paused Command (outcome = "paused") without sleeping in-process—stopping queued sibling Spec Tickets while active ones finish or salvage, and releasing its Claim without agent-failed (putting ready-for-agent back unless an untrusted edit requires needs-triage or a Security fix requires agent-failed)—so that account exhaustion never trips pickup.failure_limit and Pickup self-heals after reset.
20. As the Day shift, I want a Paused Command that restored ready-for-agent for automatic Pickup continuation to sit in Done rather than Needs you (and save its Run notification without emailing when only_unmerged is set), while Paused Commands that Pickup will not retake wait in Needs you and send their notification, so that Needs you and overnight emails stay strictly actionable.
21. As the Day shift, I want pull-request and notification Built with lines to keep their single primary tuple for Scoring compatibility while .models.json, the Command record and the Run notification's Session models list record every Step's actual Harness and Model, so that multi-Harness runs are transparently attributed without breaking Scoring.

Implementation Decisions

• Harnesses in scope and data sources:
  • Scope covers the five subscription Harnesses: claude, codex, muse, agy and grok. opencode and kimi are excluded as API-key Harnesses.
  • Passive session-stream capture updates the machine's saved Plan usage state atomically on every running session: claude rate_limit_event frames across all statuses (allowed, allowed_warning, rejected) and all utilization percentages plus ClaudeApiError limit facts (overageStatus, overageDisabledReason, resetsAt, overageResetsAt), codex session token_count rate_limits, and terminal UsageLimit protocol errors across all five subscription Harnesses.
  • Active zero-token CLI/RPC probes run only on demand: codex app-server stdio JSON-RPC account/rateLimits/read (accountId, planType, ordinaryUsageAllowed, rateLimits / rateLimitsByLimitId, credits, spendControlReached), muse serve stdio JSON-RPC usage/read (tier, observedAtMs, window, weekly) and account/read (state, label, credentialRequired), and claude auth status --json (loggedIn, subscriptionType, planTier, orgId, email). grok and agy currently expose no working headless non-billed window probe and rely on passive session-stream capture (with grok ready to adopt its CLI JSON output if implemented upstream later).
  • Never call undocumented web/OAuth endpoints or read Harnesses' private auth tokens.
• Refresh cadence and caching:
  • Viewing (thirdshift plan-usage or /plan-usage / /api/v1/plan-usage) refreshes local active probes when the saved snapshot is older than ~60 seconds.
  • Routing refreshes candidate codex/muse probes only when a candidate's saved snapshot is missing, past resets_at, or older than ~5 minutes.
  • thirdshift console runs no idle background timer for Plan usage (ADR 0016).
• User config schema (reusable by add fallback model setting for provider issues, like codex model at capacity #1546):

  [harness]
  default = "claude"
  routes = []            # ordered entries: "claude" or { harness = "agy", model = "...", effort = "...", max_utilization = 90, allow_credits = false }
  max_utilization = 100  # 1..=100; 100 switches proactively only on confirmed exhaustion
  
  [security]
  harness = ""
  routes = []            # optional ordered entries in the same format; when empty, plan-limited security.harness walks harness.routes

  • All new keys appear in DEFAULTS so thirdshift config and Console /config inventory and validate them automatically.
  • With harness.routes empty, thirdshift uses harness.default (or Claude Code) with no cross-Harness fallback. String entries inherit [harness.<name>]'s model and effort; inline table entries override optional model, effort, max_utilization (1–100) and allow_credits (default false).
• Credit and overage boundaries:
  • Included Plan windows (usedPercent / utilization) govern proactive routing by default (allow_credits = false), routing away at max_utilization or 100% before spending Codex credit balances (credits.balance).
  • Headless agy invocations pass --settings '{"useG1Credits":false}' unless the chosen route sets allow_credits = true.
• Per-Step routing, pinning, and mid-Step clean reset:
  • Routing is evaluated per session Step on factory Commands whose Harness comes from User config defaults, using strictly machine-local state (zero Fleet HTTP calls during Pass or Step admission).
  • Explicit CLI harness/model flags, explicit Custom-process Step harness settings, Experiments and Scoring runs stay pinned. Cybersecurity SafeguardRefusal never routes to another Harness and still holds Self-merge (ADR 0021).
  • Only structured Harness wire-frame events and official non-billed CLI/RPC facts count as Plan-usage exhaustion—never bare HTTP 429 transient throttling and never agent-writable assistant/tool text (ADR 0030, ADR 0031).
  • Each configured route is tried at most once per Step. A cross-Harness switch always starts a fresh session (never a cross-Harness Resume), and every configured route's Factory skills are linked into the worktree.
  • When a session stops on a confirmed Plan-usage limit mid-Step after doing work (before_first_turn == false), local commits are kept, uncommitted worktree edits are discarded (git reset --hard HEAD and git clean -fd, preserving .git/info/exclude handover files; for merge-conflict Repair, git merge --abort followed by re-merging the pinned conflict commit), Implement switches from fresh to the existing continuation prompt when commits or an open PR now exist on the branch, and zero new quota-warning text is added to any Session prompt.
• Cooldown clearance and revalidation:
  • Exhaustion cooldowns hold until resets_at when reported, or for an initial 1-hour revalidation backoff (doubling on repeated before_first_turn limit rejections: 1h → 2h → 4h → 6h cap) when resets_at is absent.
  • Past resets_at or backoff expiry, a route enters revalidating state: codex and muse re-probe before selection (requiring ordinaryUsageAllowed != false and spendControlReached == false on Codex), while passive-only Harnesses (claude, agy, grok) try once on the next Step and immediately re-arm the doubled backoff and fall back if rejected before_first_turn (handling monthly spend caps that outlive weekly resets).
  • thirdshift plan-usage --clear [<harness>] (and its authenticated Console action) clears recorded cooldown/backoff state manually.
• CLI, Console, Fleet grouping, and console report:
  • thirdshift plan-usage [--fleet | --machine <name>] [--json] [--clear [<harness>]] and Console /plan-usage (Plan usage in navigation with jump key p, /api/v1/plan-usage in OpenAPI with full CLI/API parity) default to Fleet when Fleet peers/reports exist.
  • Fleet view groups shared subscriptions across machines by account identity when known (claude orgId/email, codex accountId, muse account label), displaying the freshest observation and the list of machines using that login, while keeping distinct logins and unknown-identity Harnesses (agy, grok) separate per machine.
  • Rows include every Harness named in harness.default, harness.routes, security.harness or security.routes (in route order, marking the active primary route and max_utilization) plus any subscription Harness with a saved observation or signed-in CLI, omitting unused/uninstalled ones.
  • Reachable peer reads cache last-seen per-machine snapshots for offline fallback, and post-Command console report pushes include the reporting host's local Plan usage snapshot under the report scope.
  • Console header shows a warning pill beside host resources when any configured route is exhausted or at/above max_utilization.
• All-routes-exhausted Pass skips, Paused Command, Claim, Inbox, and Built with:
  • When every configured default route on a machine is unavailable, scheduled Passes skip before acquiring worktrees or GitHub claims, writing one Activity log line per Pass kind on reason change, and 1 Inbox plus thirdshift status show a status banner naming the exhausted routes and earliest known reset.
  • A live factory Command whose allowed routes are all exhausted ends immediately without sleeping in-process as a Paused Command (outcome = "paused"), recording the stopping Harness, Plan window, utilization, reset time and Continuation/resume command on its Command record. In a Spec run, queued sibling Tickets stop starting while active Tickets finish or salvage.
  • Ending an issue's Claim for a Paused Command removes in-progress without adding agent-failed, putting ready-for-agent back if the Claim took it off—unless an untrusted account edited the issue since its last Trusted ready-for-agent labelling (which leaves needs-triage) or the Run is a Security fix (which swaps for agent-failed).
  • A Paused Command whose Claim put ready-for-agent back lands in Done/history rather than Needs you, and under only_unmerged saves its Run notification without emailing; any Paused Command that Pickup will not retake automatically (Experiment, Scoring run, PR Fix/Review, Security fix, untrusted-edit needs-triage, or a hand-started Run that did not restore ready-for-agent) enters Needs you as Paused and emails its notification when configured.
  • Built with keeps its single parseable tuple (the Harness/Model/Effort of the primary implementation session, or resolved first route if rerouted earlier), while .models.json, CommandRecord and the Run notification's Session models list record every Step's actual Harness and Model.

Testing Decisions

• Test external user-visible behavior at existing top-level seams—CLI output/JSON, Console HTTP/HTML/SSE responses, git worktree state, GitHub label/PR effects, Command records, Activity logs and Run notifications—never internal helper details or real sleeps.
• Fake Harness executables and session-stream seam: Extend scripted fake Harness CLIs to emit Claude rate_limit_event frames across all utilization levels and ClaudeApiError spend/window rejections, claude auth status --json, codex app-server account/rateLimits/read and session token_count rate_limits + credits, muse serve usage/read + account/read, agy --settings '{"useG1Credits":false}' verification + UsageLimit errors, and grok UsageLimit errors. Assert TTL cache reuse, proactive max_utilization routing, allow_credits gating, reactive before_first_turn and mid-Step clean worktree resets (including Implement switching to continuation once commits/PR exist and conflict-Repair merge reset), exponential cooldown revalidation, and --clear. Prior art: existing Harness choice, interpretation, fresh-run and spec-run integration suites.
• Console HTTP, Fleet, parity and browser seam: Test /plan-usage, /api/v1/plan-usage, CLI/API parity, Fleet shared-login grouping vs distinct/unknown-identity separation, offline peer cache fallback, console report push under the report scope, header warning pill visibility, and Console Playwright interaction. Prior art: Console Metrics, Fleet Metrics report, Resources, and route-table parity suites.
• Pass admission, Claim, Inbox and notification seam: Using fake gh and git, assert Pass skips + Activity log dedupe + 1 Inbox / thirdshift status banner when all routes are exhausted, Paused Command (outcome = "paused") across Ticket runs, Spec runs (halting queued Tickets) and Custom processes, Claim release without agent-failed (including the untrusted-edit needs-triage guard, Security-fix agent-failed guard, and rejection of agent-forged prose), and Done vs Needs you + only_unmerged notification suppression when ready-for-agent is restored. Prior art: Pickup run, Out-of-disk Claim release, Status/Inbox, and Run notification suites.

Out of Scope

• Bounded same-Harness retries on temporary HTTP 503/529/overload errors (Recover from temporary Harness service failures within a bounded retry budget #584).
• Routing or model fallback for provider model-at-capacity errors, expired OAuth sessions, or cybersecurity safeguard refusals (add fallback model setting for provider issues, like codex model at capacity #1546, A Harness's cybersecurity refusal on a Security finding fails the Triage run and counts toward the Failure limit #1972)—though [harness] routes and [security] routes are shaped so add fallback model setting for provider issues, like codex model at capacity #1546 reuses the same configuration table.
• Subscription Plan usage monitoring for API-billed Harnesses (opencode and kimi).
• Scraping undocumented web or OAuth usage endpoints or reading Harnesses' private authentication tokens.
• Cross-Harness session Resume (--resume stays strictly within one Harness) or adding un-evaluated quota-handoff instructions to Session prompts.
• Cross-machine Fleet HTTP calls inside Pass or Step routing decisions.

Further Notes

• Domain glossary (Plan usage, Plan window, Paused Command, Claim, Failure limit, Harness, Needs you), grill record and ADR 0031 landed in Record the #583 Plan usage grill, glossary terms and ADR 0031 #2178.
• Closes add coding subscription monitor to thirdshift console web ui #969 into this Spec's Console /plan-usage Ticket.