Repository navigation
[bug] A repo has been marked broken for five days by a dropped WebSocket — a transport failure is stored as the repo's state, and no verdict is ever re-taken #440
Description
A healthy repo has been marked broken for five days, and nothing has re-asked
pas/platform on the Coder Home instance says the checkout cannot be used. The checkout is fine.
The verdict was taken on 2026-08-03 by something that was not looking at the checkout at all, and
nothing has replaced it since.
Measured, production, twice, ten minutes apart
GET /v1/instances/12ebf1f0-…/coding/repos
pas/platform cloneStatus: "error"
cloneError: "No runner connected — run `pags up`"
workdir: ~/dev/stores/pas/platform
updatedAt: 2026-08-03 01:44:25 ← five days ago
On disk, right now: ~/dev/stores/pas/platform holds 18 entries and
git rev-parse --is-inside-work-tree → true. The machine is connected —
GET /v1/relay/12ebf1f0-…/status?node=Mac → {"connected":true}, runner 0.4.44 (which has
/coding/repo-check). Two consecutive reads of the repos list left the row byte-identical.
updateRepoClone bumps updated_at on every write (lib/coding-store.ts:155-171), so the
untouched timestamp is proof that no verdict has been written for this row in five days, not
merely that none changed it.
Meanwhile the chat prompt reads that row and tells the model the repository is unusable
(lib/repo-status-prompt.ts, attachedReposPrompt). #405 exists because a wrong ready makes an
agent invent code; a wrong error makes it refuse work it could do.
Two defects, and one correction to the report
1. A transport failure is stored as a property of the repo.
lib/coding-session-open.ts:112-120:
} catch (e) {
const msg = e instanceof Error ? e.message.slice(0, 300) : String(e);
await updateRepoClone(env, repo.id, { cloneStatus: "error", cloneError: msg });Any exception from POST /coding/start becomes the repo's clone status — including
No runner connected, which is a fact about a WebSocket, not about a directory. clone_status
answers "is there usable code at this path"; a closed laptop is not an answer to that question.
The platform already models this distinction correctly one file over: verdictFromCheck has a
dedicated unverified state for exactly "the machine could not tell us", and
cloneStatusForVerdict returns null for it — deliberately no write (lib/coding-workdir.ts:73-84, 122-133). runnerUnreachable-shaped failures should take that path, not the error path.
2. There is no way to re-take a verdict, and no record of when one was taken.
A correction first, because it changes what to build: the repo list DOES re-verify —
routes/coding-repos.ts:139-155 runs verifyLocalWorkdir over up to MAX_VERIFY_PER_LIST = 12
local repos on every GET …/coding/repos. So "the check only runs at add and at workdir change" is
not true of the code. What is true is that the re-check is:
- conditional —
if (!conn) return c.json({ repos }). No resolvable runner connection at that
instant, no re-check, no trace of the attempt; - incidental — the only trigger is someone opening that instance's repo list. A row belonging to
an instance nobody opens is never re-examined by anything; - one-directional in practice —
onUnverified: nullat list time is right (an offline laptop
must not condemn a path it verified yesterday), but it means the only way out of a wrong verdict is
a successful check, which requires the conditional path above to fire.
And an open question I could not close. With the relay reporting connected: true for that very
instance and a 0.4.44 runner, two list reads still wrote nothing — so either
getBoundRunnerConn returned null (a pin, a registration row under another node name, #379
alias resolution) or /coding/repo-check answered unverified. Both are invisible: the route
neither logs the attempt nor reports it in the response. Whoever picks this up should instrument
that first — it may be the whole bug, and it is unknowable from outside. For contrast, another
instance's row (bd43f4de… / apps/chess-academy) was written today at 06:21:55, so the write
path does work somewhere.
A second live row, reported as the same thing but genuinely different
stores/fds (same instance, workdir ~/dev/stores/fds) reads ready; the directory holds one
entry and no .git of its own. That is not a stale verdict — it is what today's check would
say: ~/dev/stores is itself a git work tree, so git rev-parse --is-inside-work-tree is true
for the subfolder, and #405 chose that test on purpose ("~/dev/monorepo/apps/thing is a
legitimate workdir and an existence test would have condemned every package in every monorepo
checkout"). Worth knowing while working here, not worth changing on this evidence.
What to do — cheapest first
- Never write
clone_status: "error"for a transport failure. Incoding-session-open.ts:117-120,
route an unreachable-runner exception (isRunnerUnreachable/relayFailureIsDisconnectalready
exist inlib/runner-availability.ts) to no write — the same answerunverifiedgets. A repo's
state should only be written by something that looked at the repo. - Add
clone_checked_at(migration + a write inverifyLocalWorkdir).repo-status-prompt.ts:113-129
already asks for it and explains whyupdated_atcannot stand in:"
updated_atis bumped by any edit to the row (rename, launch URLs, merge policy), so rendering
it as 'checked 3 minutes ago' would be a precise-looking claim the platform cannot actually
support … A realclone_checked_atcolumn would fix that."
With it,FRESHNESS_NOTEcan say when instead of only "the LAST RECORDED verdict", the
repo row can show it, and "this verdict is five days old" becomes visible rather than
reconstructed from a timestamp that means something else. - Make re-checking reachable on purpose. A
POST …/coding/repos/:id/recheck(or a "Re-check"
control beside the diagnosis bannerReposList.tsxalready renders) that runsverifyLocalWorkdir
withonUnverified: "unknown"and returns the verdict — so a user looking at a wrong verdict has
an action, and so the check can be exercised deliberately when diagnosing. - Say when a re-check did not happen. When the list route returns early for want of a
connection, that is information: the rows shown are last-known, not current. It is one field on
the response and one sentence in the UI, and it is what makes the difference between "the
platform checked and it is broken" and "nobody has looked since Monday".
Alternatives considered and rejected
- Re-check on every chat turn. Rejected, and [bug] Chat tells the agent a repo is "needs_attention" and drops the sentence saying why — #405's diagnosis never reaches the model #416 rejected it first: a relay round trip on the
latency path of every message, for a fact that changes rarely. - A cron that re-verifies every local repo. Rejected as the first move: it needs a live socket
per user per machine, so it would mostly writeunverifiedand spend relay traffic to learn
nothing. Revisit only if (3) proves insufficient. - Downgrade at list time when the runner is absent (
onUnverified: "unknown"). Rejected —
coding-repos.ts:103-112argues it correctly: it would flip every repo in the list every time a
laptop closed, and the offline banner already says the true thing. - Treat this as "[bug] A local repo is marked "ready" without anyone checking it exists — an empty checkout reports success, and the agent invents the code it cannot see #405 is unfinished". It is not. [bug] A local repo is marked "ready" without anyone checking it exists — an empty checkout reports success, and the agent invents the code it cannot see #405's fix is correct for a repo being added or
moved. This is the rows that already existed, plus a writer [bug] A local repo is marked "ready" without anyone checking it exists — an empty checkout reports success, and the agent invents the code it cannot see #405 never touched.
Acceptance criteria
- A
POST /coding/startthat fails because no runner is reachable leavesclone_status
unchanged (test: unreachable-runner error → noupdateRepoClonecall). -
coding_repos.clone_checked_atexists, is written by every verdict, and is surfaced in the
repo row and inattachedReposPrompt's freshness note. - An owner can re-check a repo on demand and see the verdict, with no session and no runner
restart. - The live row above reads
readyagain after a re-check, andstores/fdsstill readsready
(the monorepo-subfolder case must not regress). - When the list route cannot re-check, the response says so.
Regression risk
clone_statusis a plain string column with an additive union (ready/error/
needs_attention/unknown); readers arerepo-status-prompt.ts,repo-status.tsand
ReposList.tsx. Adding a column is safe; changing what a value means is not — leave the
existing four alone.- Not writing on a transport failure means a genuinely broken clone that also happens to fail at the
transport keeps its previous status. That is the correct trade (silence over a false claim) and it
is the same tradeonUnverified: nullalready makes, but it should be stated in the code, not
inferred. - A
recheckroute runs a relay command on user input: rate-limit it with the existing buckets and
scope it to the owner, like every other:instanceIdroute.
Files: workers/api/src/lib/coding-session-open.ts:112-120, workers/api/src/routes/coding-repos.ts:100-160,181,599,
workers/api/src/lib/coding-workdir.ts:73-133, workers/api/src/lib/coding-store.ts:155-171,
workers/api/src/lib/repo-status-prompt.ts:102-133, store/console/src/…/ReposList.tsx.
Related: #405 (which built the check; still open pending live confirmation — this is why reading
the list cannot confirm it), #416 (which decided against checking at chat time), #410/#411.
Activity
feat(coding): a repo declares the invariants it claims, and a finishe…
fix(coding): a closed laptop no longer marks a healthy checkout broken (
feat(coding): a repo verdict says when it was taken, and can be re-ta…
serge-ivo commentedon Aug 8, 2026
Shipped in
52d1561+25cabf2, deployed — and the open question is closed: the instance is pinned to an offline machineThe mechanism, which the issue explicitly left unresolved
getBoundRunnerConnreturns null on the repos-list route, soif (!conn) return c.json({ repos })fires and no verdict is ever re-taken. Measured live just now,
GET …/runtime/status:The instance carries a "Runs on" pin to a machine that is not running
pags up.getBoundRunnerConnhonours the pin and deliberately does not fall back (#379/#380), so null is thecorrect answer — the route was right and silent, which is the worst combination.
Why this looked impossible from outside, and why the report's own evidence misled: every surface
that reported this agent healthy bypasses the pin.
/coding/diagnosticsusesgetRunnerConnon theruntime row's node (it answered
reachable:true,health.ok:true),/coding/browseusesgetDefaultRunnerConn, and/v1/terminals/nodescallsrelayConnectedper node. I confirmed therunner itself is fine by probing it directly —
POST /coding/repo-checkfor that exact path returns{"checked":true,"exists":true,"isDirectory":true,"entryCount":18,"insideWorkTree":true}instantly. So suspects (b) relay transport and (c) a swallowed verdict are both excluded: the
transport works and the code never reaches the verdict.
One more piece of provenance: the stored
cloneError"No runner connected — run \pags up`"iscallRunner's **pre-#341 503 message, deleted inb347f68** — so it can only have been written bycoding-session-open.ts`'s catch, which is defect 1.Verified in production, after the deploy
The list now answers the question it used to duck:
And the on-demand re-check works, end to end, on the row from the report:
It persisted. That row's
updated_atis now2026-08-08 09:22:16— the first write in five days— and
clone_statushas gone from a falseerrorto an honestunknown. The platform hasstopped claiming a healthy 18-entry git checkout is broken, and
attachedReposPromptwill stoptelling the model the repository is unusable.
clone_checked_atis correctly still NULL: no machinelooked, and an absent time is exactly what that means.
Acceptance criteria
/coding/startleavesclone_statusunchanged —isRunnerUnreachableroutes it to the same no-write path
unverifiedtakes. Tests assertupdateRepoCloneisnot called, in both the typed-error and serialised-marker forms.
clone_checked_atexists (migration 0110), is written by every definite verdict includingan unchanged one, and is surfaced in the repo row and in
attachedReposPrompt's freshness note.NULL for existing rows with no backfill — inventing a time would be the false precision the
column exists to remove.
stores/fdsdoes not regress — the monorepo-subfolder case is pinned by a unit test(
insideWorkTree:true, 1 entry →ok).ready. Still blocked, and NOT by this code: nothing can verify thatpath until a runner the pin resolves to is attached. The remedy is one owner action —
PUT /v1/instances/12ebf1f0-…/runner-node {"runnerNode":"Mac"}(or clear it for Automatic),or start
pags uponSergeys-Mac-mini.local. Not applied here: repointing which machinean agent's coding sessions run on is an operational choice this ticket did not ask for.
Keeping the ticket open on that last box. What has changed is that the failure is now visible and
actionable instead of silent — which was the actual complaint.
Not written, deliberately
A resolved connection that answers nothing is the other invisible outcome, and no probe from outside
can see it: the check degrades to
unverified,onUnverified: nulldeclines to write, and theresponse looks identical to a successful re-check. That case now emits a
warnto the durable errorlog naming the node and the count, so it is evidence rather than a shrug.
Gates
pnpm typecheck·pnpm vitest run(412 files, 6849 tests, 0 failures) · biome--error-on-warnings(1046 files) · design-tokens · agents-allowlist · qa-config · file-size ·check-migrations --require-history(111 migrations, 110 distinct) · test-isolation ·docs:drift— all green, re-run after the final rebase onto
85a39d9.pnpm test:e2eran: 277 passed, chromium + webkit, including a new#440block at 320px and390px in WebKit. It caught a real defect before it shipped: the Re-check control measured
23.98px in WebKit, under WCAG 2.5.8's 24px floor — fixed
py-1→py-1.5. One earlier failure(
docs/browser-runtime/404) was the gitignored generatedstore/docstree missing from a freshworktree; I ran
pnpm docs:buildand re-ran rather than reporting it as pre-existing, and it passes.The
CodingTab.tsxsize pin moved 1302 → 1309 with the reason written at the pin: the tab nowcarries the server's own
recheckanswer instead of deriving one fromrunnerOnline— which wouldhave been the bug, since the runner status dot does not honour the "Runs on" pin and the repos route
does. A component guessing from the dot would report "checked" for a list that checked nothing.
fix(coder): the single-repo header is one line at 320px, because the …
serge-ivo commentedon Aug 8, 2026
Three of five criteria delivered and verified live. Staying open: one needs the owner, one is now #461.
Re-audited independently against the deployed API (
52d1561and25cabf2are ancestors ofb6f6c9f6, deployed 13:02 UTC today).Delivered, and confirmed in production
A transport failure no longer becomes the repo's state.
lib/coding-session-open.ts:151-153now guards the write with
if (!isRunnerUnreachable(e)), and the trade is written down ratherthan inferred. Live proof: the row this ticket is about no longer reads
error / "No runner connected — run \pags up`"`. It reads:clone_checked_atexists and is written by every verdict — migration0110, stamped evenwhen the verdict is unchanged (
routes/coding-repos.ts:118-126), which is the whole point: acorrect
readyand areadynobody has re-confirmed since Monday are now distinguishable.Observed live on three other instances, e.g.
cd489be0-…→recheck {"ran": true, "checked": 5}with
cloneCheckedAt: 2026-08-08 13:16:11on every local row.An owner can re-check on demand —
POST …/coding/repos/:repoId/recheck(
routes/coding-repos.ts:242) withonUnverified: "unknown", and the control atReposList.tsx:205-218.(the second half of criterion 4)
stores/fdsstill readsready— correctly. Themonorepo-subfolder case has not regressed:
git -C ~/dev/stores/fds rev-parse --is-inside-work-tree→true, toplevel~/dev/stores.Not met — and it needs the owner, not an agent
readyagain after a re-check."The list route now correctly reports why it did not re-check, and the reason is the root cause this
ticket's open question asked whoever picked it up to instrument. It was the pin:
getBoundRunnerConnhonours the pin and does not fall back — correctly, by #379/#380 design — so thelist route returns early,
POST …/recheckwill do the same, andpas/platformcannot leaveunknownwhile the pin points at a machine that is switched off.
This is one click and it is the owner's: on the "Coder Home" instance, set Runs on →
Mac (or Automatic). The re-check then runs on the next open of the Coding tab and the row
returns to
ready. Nothing in the codebase can do this for them — repointing a pin is a decision aboutwhere the owner's code runs, and an agent silently changing it would defeat the guarantee the pin
exists to provide.
Confirmed as of 2026-08-08 13:2x UTC: it has not been done.
Criterion 5 is met in form and wrong in content — split to #461
It does. But what it says, on this exact instance, is:
Both halves are false. Nothing else holds the agent, and
--forcecannot repoint a pin —pags upisalready running on
Macwith a live socket for this very instance.diagnoseAttachmenthas theright answer (
pinned-machine-offline,runtime-attachment.ts:63-73, which names both machines andsays to change "Runs on"), but
describeFacts(runner-availability.ts:110-116) never passespinnedNode/liveNodeExcludedByPin, so the branch is dead on every surface but/runtime/status.Filed as #461 rather than reopened here, because it defeats three call sites — this list route,
and both Workflow pause announcements — and is a defect in a shared adapter rather than in this
ticket's work. Worth noting the test that let it through:
coding-repos.test.ts:377asserts only thatthe reason is non-empty.
Summary
Everything that was an engineering task on this ticket is done. What remains is one owner action
(repoint "Runs on"), after which the last criterion verifies itself. Staying open until that is done,
because the criterion is a live-state check nobody has performed — not an assumption I am willing to
close on.
serge-ivo commentedon Aug 8, 2026
Re-measured in production today — the code is complete; the last box still needs a connected runner
Picked this up in the Coder/runner batch (#461 #460 #450 #449 #448 #440 #432 #454). Nothing further
was built here:
52d1561+25cabf2already deliver every code item. This is the fresh measurement,plus one thing the audit turned up that has its own ticket now.
Live,
2026-08-08 ~21:1x UTC, instance12ebf1f0-…("Coder Home")The substance of the report is fixed and has stayed fixed. The row that spent five days falsely
claiming a healthy 18-entry git checkout was broken now reads
unknownwith a sentence that says whodid not look and why.
stores/fdsstill readsready, so the monorepo-subfolder case has notregressed.
clone_checked_atis NULL on every row, which is correct and is the column doing its job:no machine has looked since it was added, and inventing a time would be exactly the false precision it
exists to remove.
The one open box, and why it is not closable from here
ready. Still blocked, and still not by this code. Both nodes areoffline right now, so no verdict can be taken —
unknownis the honest answer and the platformis giving it. Reading
readyrequirespags upon a machine holding~/dev/stores/pas/platform.I am not starting a runner: the CLI is a user-facing global package and running it on the owner's
behalf is not mine to do, and repointing the "Runs on" pin is the operational choice this ticket
declined to make in the first place.
So this is a verification waiting on one owner action, not unfinished work. Leaving it open on that
basis, consistent with the previous comment.
What the audit did find — filed as #461, not folded in here
The
recheck.reasonabove is correct today only because nothing is connected. When one machineis connected and it is not the pinned one — the exact state this ticket was diagnosed in — that same
sentence comes out as "The machine is online but this agent isn't attached … Try:
pags up --force",which is wrong in both halves for a pinned instance.
describeFactsdrops the two fieldsdiagnoseAttachment'spinned-machine-offlinebranch needs, so the fourth item this ticket shipped("say when a re-check did not happen") tells three surfaces the wrong story. That is #461 and it is
being fixed in this batch.
fix(runner): carry the "Runs on" pin on RuntimeFacts, so a pinned age…
feat(coding): a repo promoted to `act` is put back on its declared br…
serge-ivo commentedon Aug 12, 2026
Partial-delivery note, from an unrelated investigation that landed on the same file (#548).
Defect 1 is delivered.
52d1561("a closed laptop no longer marks a healthy checkout broken") added the guard this ticket asked for —workers/api/src/lib/coding-session-open.ts:149-151:A transport failure now takes the same no-write path
cloneStatusForVerdictgivesunverified, exactly as proposed.Defect 2 (no way to re-take a verdict, no record of when one was taken) I did not re-verify, and the open question at the end of this ticket — why two list reads wrote nothing against a connected 0.4.44 runner — is still uninstrumented as far as I can see. So this stays live on that half.
And the mirror defect, four lines up, was not in scope here and is still open.
coding-session-open.ts:129writescloneStatus: "ready"on a successfulPOST /coding/start, with nothing having looked at the directory — the same invariant this ticket's own comment states two lines below it ("A REPO'S STATE IS ONLY EVER WRITTEN BY SOMETHING THAT LOOKED AT THE REPO"), in the false-readydirection rather than the false-errorone. Measured in production 2026-08-12:~/dev/aipahas no.git, the platform had writtenneeds_attentionwith the correct sentence, and starting a run re-stamped itready. Filed as #548 with the run-admission half; not folded into this ticket because the fix is a different line and a different direction.fix(coding): a run refuses a folder measured as "not a git working tr…
fix(coding): the offline banner says WHICH machine, because a boolean…
serge-ivo commentedon Aug 15, 2026
The last box is now checked, measured in production — the row reads
readyagainNo code was written for this.
52d1561+25cabf2delivered every engineering item; the criterionleft open was a live-state check that needed a connected runner, and a runner is connected now.
GET /v1/instances/12ebf1f0-…/coding/repos("Coder Home"), read 2026-08-15 21:18 UTC:The row this ticket is about — five days of a false
error / "No runner connected — run \pags up`", then an honestunknown— now reads **ready**, and it says **when** a machine last looked at the path rather than leaving that to be reconstructed fromupdated_at.stores/fdsstill readsready`, so the monorepo-subfolder case has not regressed.recheck.ran: true, checked: 7is the fourth criterion doing its job from the other side: the listroute is now saying it DID look, at seven paths, in this request.
Acceptance criteria
/coding/startleavesclone_statusunchanged.coding_repos.clone_checked_atexists, is written by every verdict, and is surfaced — and itis now populated on every local row here, where it was NULL at the last two audits precisely
because nothing had looked.
readyafter a re-check — above. This was the only one outstanding.stores/fdsstill readsready.pags up --force— describeFacts drops the two fields the pinned-machine diagnosis needs, so the branch is dead on every surface but one #461 fixed what it was saying).Nothing outstanding. Closing.