Skip to content

[bug] A repo has been marked broken for five days by a dropped WebSocket — a transport failure is stored as the repo's state, and no verdict is ever re-taken #440

Closed

Description

@serge-ivo

A healthy repo has been marked broken for five days, and nothing has re-asked

pas/platform on the Coder Home instance says the checkout cannot be used. The checkout is fine.
The verdict was taken on 2026-08-03 by something that was not looking at the checkout at all, and
nothing has replaced it since.

Measured, production, twice, ten minutes apart

GET /v1/instances/12ebf1f0-…/coding/repos
  pas/platform   cloneStatus: "error"
                 cloneError:  "No runner connected — run `pags up`"
                 workdir:     ~/dev/stores/pas/platform
                 updatedAt:   2026-08-03 01:44:25      ← five days ago

On disk, right now: ~/dev/stores/pas/platform holds 18 entries and
git rev-parse --is-inside-work-tree → true. The machine is connected —
GET /v1/relay/12ebf1f0-…/status?node=Mac → {"connected":true}, runner 0.4.44 (which has
/coding/repo-check). Two consecutive reads of the repos list left the row byte-identical.

updateRepoClone bumps updated_at on every write (lib/coding-store.ts:155-171), so the
untouched timestamp is proof that no verdict has been written for this row in five days, not
merely that none changed it.

Meanwhile the chat prompt reads that row and tells the model the repository is unusable
(lib/repo-status-prompt.ts, attachedReposPrompt). #405 exists because a wrong ready makes an
agent invent code; a wrong error makes it refuse work it could do.

Two defects, and one correction to the report

1. A transport failure is stored as a property of the repo.
lib/coding-session-open.ts:112-120:

} catch (e) {
	const msg = e instanceof Error ? e.message.slice(0, 300) : String(e);
	await updateRepoClone(env, repo.id, { cloneStatus: "error", cloneError: msg });

Any exception from POST /coding/start becomes the repo's clone status — including
No runner connected, which is a fact about a WebSocket, not about a directory. clone_status
answers "is there usable code at this path"; a closed laptop is not an answer to that question.
The platform already models this distinction correctly one file over: verdictFromCheck has a
dedicated unverified state for exactly "the machine could not tell us", and
cloneStatusForVerdict returns null for it — deliberately no write (lib/coding-workdir.ts:73-84, 122-133). runnerUnreachable-shaped failures should take that path, not the error path.

2. There is no way to re-take a verdict, and no record of when one was taken.

A correction first, because it changes what to build: the repo list DOES re-verify —
routes/coding-repos.ts:139-155 runs verifyLocalWorkdir over up to MAX_VERIFY_PER_LIST = 12
local repos on every GET …/coding/repos. So "the check only runs at add and at workdir change" is
not true of the code. What is true is that the re-check is:

  • conditional — if (!conn) return c.json({ repos }). No resolvable runner connection at that
    instant, no re-check, no trace of the attempt;
  • incidental — the only trigger is someone opening that instance's repo list. A row belonging to
    an instance nobody opens is never re-examined by anything;
  • one-directional in practice — onUnverified: null at list time is right (an offline laptop
    must not condemn a path it verified yesterday), but it means the only way out of a wrong verdict is
    a successful check, which requires the conditional path above to fire.

And an open question I could not close. With the relay reporting connected: true for that very
instance and a 0.4.44 runner, two list reads still wrote nothing — so either
getBoundRunnerConn returned null (a pin, a registration row under another node name, #379
alias resolution) or /coding/repo-check answered unverified. Both are invisible: the route
neither logs the attempt nor reports it in the response. Whoever picks this up should instrument
that first
— it may be the whole bug, and it is unknowable from outside. For contrast, another
instance's row (bd43f4de… / apps/chess-academy) was written today at 06:21:55, so the write
path does work somewhere.

A second live row, reported as the same thing but genuinely different

stores/fds (same instance, workdir ~/dev/stores/fds) reads ready; the directory holds one
entry and no .git of its own. That is not a stale verdict — it is what today's check would
say: ~/dev/stores is itself a git work tree, so git rev-parse --is-inside-work-tree is true
for the subfolder, and #405 chose that test on purpose ("~/dev/monorepo/apps/thing is a
legitimate workdir and an existence test would have condemned every package in every monorepo
checkout"
). Worth knowing while working here, not worth changing on this evidence.

What to do — cheapest first

  1. Never write clone_status: "error" for a transport failure. In coding-session-open.ts:117-120,
    route an unreachable-runner exception (isRunnerUnreachable / relayFailureIsDisconnect already
    exist in lib/runner-availability.ts) to no write — the same answer unverified gets. A repo's
    state should only be written by something that looked at the repo.
  2. Add clone_checked_at (migration + a write in verifyLocalWorkdir). repo-status-prompt.ts:113-129
    already asks for it and explains why updated_at cannot stand in:

    "updated_at is bumped by any edit to the row (rename, launch URLs, merge policy), so rendering
    it as 'checked 3 minutes ago' would be a precise-looking claim the platform cannot actually
    support … A real clone_checked_at column would fix that."

    With it, FRESHNESS_NOTE can say when instead of only "the LAST RECORDED verdict", the
    repo row can show it, and "this verdict is five days old" becomes visible rather than
    reconstructed from a timestamp that means something else.

  3. Make re-checking reachable on purpose. A POST …/coding/repos/:id/recheck (or a "Re-check"
    control beside the diagnosis banner ReposList.tsx already renders) that runs verifyLocalWorkdir
    with onUnverified: "unknown" and returns the verdict — so a user looking at a wrong verdict has
    an action, and so the check can be exercised deliberately when diagnosing.
  4. Say when a re-check did not happen. When the list route returns early for want of a
    connection, that is information: the rows shown are last-known, not current. It is one field on
    the response and one sentence in the UI, and it is what makes the difference between "the
    platform checked and it is broken" and "nobody has looked since Monday".

Alternatives considered and rejected

Acceptance criteria

  • A POST /coding/start that fails because no runner is reachable leaves clone_status
    unchanged (test: unreachable-runner error → no updateRepoClone call).
  • coding_repos.clone_checked_at exists, is written by every verdict, and is surfaced in the
    repo row and in attachedReposPrompt's freshness note.
  • An owner can re-check a repo on demand and see the verdict, with no session and no runner
    restart.
  • The live row above reads ready again after a re-check, and stores/fds still reads ready
    (the monorepo-subfolder case must not regress).
  • When the list route cannot re-check, the response says so.

Regression risk

  • clone_status is a plain string column with an additive union (ready / error /
    needs_attention / unknown); readers are repo-status-prompt.ts, repo-status.ts and
    ReposList.tsx. Adding a column is safe; changing what a value means is not — leave the
    existing four alone.
  • Not writing on a transport failure means a genuinely broken clone that also happens to fail at the
    transport keeps its previous status. That is the correct trade (silence over a false claim) and it
    is the same trade onUnverified: null already makes, but it should be stated in the code, not
    inferred.
  • A recheck route runs a relay command on user input: rate-limit it with the existing buckets and
    scope it to the owner, like every other :instanceId route.

Files: workers/api/src/lib/coding-session-open.ts:112-120, workers/api/src/routes/coding-repos.ts:100-160,181,599,
workers/api/src/lib/coding-workdir.ts:73-133, workers/api/src/lib/coding-store.ts:155-171,
workers/api/src/lib/repo-status-prompt.ts:102-133, store/console/src/…/ReposList.tsx.
Related: #405 (which built the check; still open pending live confirmation — this is why reading
the list cannot confirm it), #416 (which decided against checking at chat time), #410/#411.

Activity

  1. serge-ivo commented on Aug 8, 2026

    @serge-ivo
    ContributorAuthor

    Shipped in 52d1561 + 25cabf2, deployed — and the open question is closed: the instance is pinned to an offline machine

    The mechanism, which the issue explicitly left unresolved

    getBoundRunnerConn returns null on the repos-list route, so if (!conn) return c.json({ repos })
    fires and no verdict is ever re-taken. Measured live just now, GET …/runtime/status:

    relay:      { connected: false, runnerNode: "Sergeys-Mac-mini.local", live: false }
    attachment: { state: "pinned-machine-offline",
                  message: "This agent is pinned to Sergeys-Mac-mini.local, which isn't connected.
                            Mac is connected — set \"Runs on\" to Mac (or Automatic)…" }
    

    The instance carries a "Runs on" pin to a machine that is not running pags up.
    getBoundRunnerConn honours the pin and deliberately does not fall back (#379/#380), so null is the
    correct answer — the route was right and silent, which is the worst combination.

    Why this looked impossible from outside, and why the report's own evidence misled: every surface
    that reported this agent healthy bypasses the pin. /coding/diagnostics uses getRunnerConn on the
    runtime row's node (it answered reachable:true, health.ok:true), /coding/browse uses
    getDefaultRunnerConn, and /v1/terminals/nodes calls relayConnected per node. I confirmed the
    runner itself is fine by probing it directly — POST /coding/repo-check for that exact path returns
    {"checked":true,"exists":true,"isDirectory":true,"entryCount":18,"insideWorkTree":true}
    instantly. So suspects (b) relay transport and (c) a swallowed verdict are both excluded: the
    transport works and the code never reaches the verdict.

    One more piece of provenance: the stored cloneError "No runner connected — run \pags up`"iscallRunner's **pre-#341 503 message, deleted in b347f68** — so it can only have been written by coding-session-open.ts`'s catch, which is defect 1.

    Verified in production, after the deploy

    The list now answers the question it used to duck:

    GET …/coding/repos
      recheck: { "ran": false, "checked": 0,
                 "reason": "The machine is online but this agent isn't attached — another runner on it
                            may already hold this agent. Try: `pags up --force`" }
    

    And the on-demand re-check works, end to end, on the row from the report:

    POST …/coding/repos/repo_e9f74506-…/recheck
      → checked: false
        verdict: { state: "unverified", path: "~/dev/stores/pas/platform",
                   detail: "`~/dev/stores/pas/platform` has not been checked — no machine is connected.
                            Run `pags up` on the machine that has this checkout." }
    

    It persisted. That row's updated_at is now 2026-08-08 09:22:16 — the first write in five days
    — and clone_status has gone from a false error to an honest unknown. The platform has
    stopped claiming a healthy 18-entry git checkout is broken, and attachedReposPrompt will stop
    telling the model the repository is unusable. clone_checked_at is correctly still NULL: no machine
    looked, and an absent time is exactly what that means.

    Acceptance criteria

    • An unreachable-runner /coding/start leaves clone_status unchanged — isRunnerUnreachable
      routes it to the same no-write path unverified takes. Tests assert updateRepoClone is
      not called, in both the typed-error and serialised-marker forms.
    • clone_checked_at exists (migration 0110), is written by every definite verdict including
      an unchanged one, and is surfaced in the repo row and in attachedReposPrompt's freshness note.
      NULL for existing rows with no backfill — inventing a time would be the false precision the
      column exists to remove.
    • An owner can re-check on demand and see the verdict, with no session and no runner restart.
    • When the list cannot re-check, the response says so and why.
    • stores/fds does not regress — the monorepo-subfolder case is pinned by a unit test
      (insideWorkTree:true, 1 entry → ok).
    • The live row reads ready. Still blocked, and NOT by this code: nothing can verify that
      path until a runner the pin resolves to is attached. The remedy is one owner action —
      PUT /v1/instances/12ebf1f0-…/runner-node {"runnerNode":"Mac"} (or clear it for Automatic),
      or start pags up on Sergeys-Mac-mini.local. Not applied here: repointing which machine
      an agent's coding sessions run on is an operational choice this ticket did not ask for.

    Keeping the ticket open on that last box. What has changed is that the failure is now visible and
    actionable
    instead of silent — which was the actual complaint.

    Not written, deliberately

    A resolved connection that answers nothing is the other invisible outcome, and no probe from outside
    can see it: the check degrades to unverified, onUnverified: null declines to write, and the
    response looks identical to a successful re-check. That case now emits a warn to the durable error
    log naming the node and the count, so it is evidence rather than a shrug.

    Gates

    pnpm typecheck · pnpm vitest run (412 files, 6849 tests, 0 failures) · biome
    --error-on-warnings (1046 files) · design-tokens · agents-allowlist · qa-config · file-size ·
    check-migrations --require-history (111 migrations, 110 distinct) · test-isolation · docs:drift
    — all green, re-run after the final rebase onto 85a39d9.

    pnpm test:e2e ran: 277 passed, chromium + webkit, including a new #440 block at 320px and
    390px in WebKit
    . It caught a real defect before it shipped: the Re-check control measured
    23.98px in WebKit, under WCAG 2.5.8's 24px floor — fixed py-1 → py-1.5. One earlier failure
    (docs/browser-runtime/ 404) was the gitignored generated store/docs tree missing from a fresh
    worktree; I ran pnpm docs:build and re-ran rather than reporting it as pre-existing, and it passes.

    The CodingTab.tsx size pin moved 1302 → 1309 with the reason written at the pin: the tab now
    carries the server's own recheck answer instead of deriving one from runnerOnline — which would
    have been the bug, since the runner status dot does not honour the "Runs on" pin and the repos route
    does. A component guessing from the dot would report "checked" for a list that checked nothing.

  2. serge-ivo commented on Aug 8, 2026

    @serge-ivo
    ContributorAuthor

    Three of five criteria delivered and verified live. Staying open: one needs the owner, one is now #461.

    Re-audited independently against the deployed API (52d1561 and 25cabf2 are ancestors of
    b6f6c9f6, deployed 13:02 UTC today).

    Delivered, and confirmed in production

    • A transport failure no longer becomes the repo's state. lib/coding-session-open.ts:151-153
      now guards the write with if (!isRunnerUnreachable(e)), and the trade is written down rather
      than inferred. Live proof: the row this ticket is about no longer reads
      error / "No runner connected — run \pags up`"`. It reads:

      ```
      pas/platform   cloneStatus: "unknown"
      cloneError:    "`~/dev/stores/pas/platform` has not been checked — no machine is connected"
      ```
      
      An honest "nobody has looked", where five days of a false accusation used to be.
      
    • clone_checked_at exists and is written by every verdict — migration 0110, stamped even
      when the verdict is unchanged (routes/coding-repos.ts:118-126), which is the whole point: a
      correct ready and a ready nobody has re-confirmed since Monday are now distinguishable.
      Observed live on three other instances, e.g. cd489be0-… → recheck {"ran": true, "checked": 5}
      with cloneCheckedAt: 2026-08-08 13:16:11 on every local row.

    • An owner can re-check on demand — POST …/coding/repos/:repoId/recheck
      (routes/coding-repos.ts:242) with onUnverified: "unknown", and the control at
      ReposList.tsx:205-218.

    • (the second half of criterion 4) stores/fds still reads ready — correctly. The
      monorepo-subfolder case has not regressed: git -C ~/dev/stores/fds rev-parse --is-inside-work-tree → true, toplevel ~/dev/stores.

    Not met — and it needs the owner, not an agent

    • "The live row above reads ready again after a re-check."

    The list route now correctly reports why it did not re-check, and the reason is the root cause this
    ticket's open question asked whoever picked it up to instrument. It was the pin:

    GET /v1/instances/12ebf1f0-…/runner-node
      runnerNode:   "Sergeys-Mac-mini.local"     ← connected=false, nodeOnline=false
      nodes:        ["Mac", "Sergeys-Mac-mini.local"]
      Mac:          connected=true, nodeOnline=true
      resolvedNode: null
    

    getBoundRunnerConn honours the pin and does not fall back — correctly, by #379/#380 design — so the
    list route returns early, POST …/recheck will do the same, and pas/platform cannot leave unknown
    while the pin points at a machine that is switched off.

    This is one click and it is the owner's: on the "Coder Home" instance, set Runs on →
    Mac (or Automatic). The re-check then runs on the next open of the Coding tab and the row
    returns to ready. Nothing in the codebase can do this for them — repointing a pin is a decision about
    where the owner's code runs, and an agent silently changing it would defeat the guarantee the pin
    exists to provide.

    Confirmed as of 2026-08-08 13:2x UTC: it has not been done.

    Criterion 5 is met in form and wrong in content — split to #461

    • [~] "When the list route cannot re-check, the response says so."

    It does. But what it says, on this exact instance, is:

    recheck.reason: "The machine is online but this agent isn't attached — another runner on it may
                     already hold this agent. Try: `pags up --force`"
    

    Both halves are false. Nothing else holds the agent, and --force cannot repoint a pin — pags up is
    already running on Mac with a live socket for this very instance. diagnoseAttachment has the
    right answer (pinned-machine-offline, runtime-attachment.ts:63-73, which names both machines and
    says to change "Runs on"), but describeFacts (runner-availability.ts:110-116) never passes
    pinnedNode / liveNodeExcludedByPin, so the branch is dead on every surface but /runtime/status.

    Filed as #461 rather than reopened here, because it defeats three call sites — this list route,
    and both Workflow pause announcements — and is a defect in a shared adapter rather than in this
    ticket's work. Worth noting the test that let it through: coding-repos.test.ts:377 asserts only that
    the reason is non-empty.

    Summary

    Everything that was an engineering task on this ticket is done. What remains is one owner action
    (repoint "Runs on"), after which the last criterion verifies itself. Staying open until that is done,
    because the criterion is a live-state check nobody has performed — not an assumption I am willing to
    close on.

  3. serge-ivo commented on Aug 8, 2026

    @serge-ivo
    ContributorAuthor

    Re-measured in production today — the code is complete; the last box still needs a connected runner

    Picked this up in the Coder/runner batch (#461 #460 #450 #449 #448 #440 #432 #454). Nothing further
    was built here: 52d1561 + 25cabf2 already deliver every code item. This is the fresh measurement,
    plus one thing the audit turned up that has its own ticket now.

    Live, 2026-08-08 ~21:1x UTC, instance 12ebf1f0-… ("Coder Home")

    GET /v1/instances/12ebf1f0-…/runner-node
      runnerNode:  "Sergeys-Mac-mini.local"
      nodesDetail: Mac                     connected=false  nodeOnline=false
                   Sergeys-Mac-mini.local  connected=false  nodeOnline=false
      resolvedNode: null                                   ← NO machine is connected at all now
    
    GET /v1/instances/12ebf1f0-…/coding/repos
      recheck: { ran: false, checked: 0,
                 reason: "The runner for this agent isn't running. Try: `pags up`" }
    
      pas/platform          unknown   "`~/dev/stores/pas/platform` has not been checked —
                                       no machine is connected."   updatedAt 2026-08-08 09:22:16
      fws/platform          ready                                  updatedAt 2026-08-03 09:53:12
      heartfull/platform    ready                                  updatedAt 2026-08-03 09:51:40
      pags/platform         ready                                  updatedAt 2026-08-03 09:51:34
      fgs/platform          ready                                  updatedAt 2026-08-03 02:36:36
      apps/chess-academy    ready                                  updatedAt 2026-08-03 01:44:25
      stores/fds            ready                                  updatedAt 2026-08-01 07:53:02
    

    The substance of the report is fixed and has stayed fixed. The row that spent five days falsely
    claiming a healthy 18-entry git checkout was broken now reads unknown with a sentence that says who
    did not look and why. stores/fds still reads ready, so the monorepo-subfolder case has not
    regressed. clone_checked_at is NULL on every row, which is correct and is the column doing its job:
    no machine has looked since it was added, and inventing a time would be exactly the false precision it
    exists to remove.

    The one open box, and why it is not closable from here

    • The live row reads ready. Still blocked, and still not by this code. Both nodes are
      offline right now, so no verdict can be taken — unknown is the honest answer and the platform
      is giving it. Reading ready requires pags up on a machine holding ~/dev/stores/pas/platform.
      I am not starting a runner: the CLI is a user-facing global package and running it on the owner's
      behalf is not mine to do, and repointing the "Runs on" pin is the operational choice this ticket
      declined to make in the first place.

    So this is a verification waiting on one owner action, not unfinished work. Leaving it open on that
    basis, consistent with the previous comment.

    What the audit did find — filed as #461, not folded in here

    The recheck.reason above is correct today only because nothing is connected. When one machine
    is connected and it is not the pinned one — the exact state this ticket was diagnosed in — that same
    sentence comes out as "The machine is online but this agent isn't attached … Try: pags up --force",
    which is wrong in both halves for a pinned instance. describeFacts drops the two fields
    diagnoseAttachment's pinned-machine-offline branch needs, so the fourth item this ticket shipped
    ("say when a re-check did not happen") tells three surfaces the wrong story. That is #461 and it is
    being fixed in this batch.

  4. serge-ivo commented on Aug 12, 2026

    @serge-ivo
    ContributorAuthor

    Partial-delivery note, from an unrelated investigation that landed on the same file (#548).

    Defect 1 is delivered. 52d1561 ("a closed laptop no longer marks a healthy checkout broken") added the guard this ticket asked for — workers/api/src/lib/coding-session-open.ts:149-151:

    if (!isRunnerUnreachable(e)) {
        await updateRepoClone(env, repo.id, { cloneStatus: "error", cloneError: msg });
    }

    A transport failure now takes the same no-write path cloneStatusForVerdict gives unverified, exactly as proposed.

    Defect 2 (no way to re-take a verdict, no record of when one was taken) I did not re-verify, and the open question at the end of this ticket — why two list reads wrote nothing against a connected 0.4.44 runner — is still uninstrumented as far as I can see. So this stays live on that half.

    And the mirror defect, four lines up, was not in scope here and is still open. coding-session-open.ts:129 writes cloneStatus: "ready" on a successful POST /coding/start, with nothing having looked at the directory — the same invariant this ticket's own comment states two lines below it ("A REPO'S STATE IS ONLY EVER WRITTEN BY SOMETHING THAT LOOKED AT THE REPO"), in the false-ready direction rather than the false-error one. Measured in production 2026-08-12: ~/dev/aipa has no .git, the platform had written needs_attention with the correct sentence, and starting a run re-stamped it ready. Filed as #548 with the run-admission half; not folded into this ticket because the fix is a different line and a different direction.

  5. serge-ivo commented on Aug 15, 2026

    @serge-ivo
    ContributorAuthor

    The last box is now checked, measured in production — the row reads ready again

    No code was written for this. 52d1561 + 25cabf2 delivered every engineering item; the criterion
    left open was a live-state check that needed a connected runner, and a runner is connected now.

    GET /v1/instances/12ebf1f0-…/coding/repos ("Coder Home"), read 2026-08-15 21:18 UTC:

    recheck: { "ran": true, "checked": 7 }
    
      pas/platform          ready   cloneCheckedAt 2026-08-15 21:18:21   cloneError: null
      heartfull/platform    ready   cloneCheckedAt 2026-08-15 21:18:21
      fgs/platform          ready   cloneCheckedAt 2026-08-15 21:18:22
      apps/chess-academy    ready   cloneCheckedAt 2026-08-15 21:18:21
      stores/fds            ready   cloneCheckedAt 2026-08-15 21:18:22
      pags/platform         ready   cloneCheckedAt 2026-08-15 21:18:22
      fws/platform          ready   cloneCheckedAt 2026-08-15 21:18:22
    

    The row this ticket is about — five days of a false error / "No runner connected — run \pags up`", then an honest unknown — now reads **ready**, and it says **when** a machine last looked at the path rather than leaving that to be reconstructed from updated_at. stores/fdsstill readsready`, so the monorepo-subfolder case has not regressed.

    recheck.ran: true, checked: 7 is the fourth criterion doing its job from the other side: the list
    route is now saying it DID look, at seven paths, in this request.

    Acceptance criteria

    Nothing outstanding. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions