instance_runtime_status.runtime publishes a raw status and blends two machines — #570's fix reached nodes[] only #587

Activity

1. serge-ivo

   added 3 commits that reference this issue on Aug 15, 2026

   fix(runtime): the default runtime row's status is derived from its he…

   ...

   35301fe

   fix(agents): publishing is the whole public gate, because the second …

   ...

   63ce16c

   fix(runtime): the probe echoes a D1-shaped timestamp, because an ISO …

   ...

   d8cc2bb
2. serge-ivo commented on Aug 15, 2026

   

   serge-ivo

   on Aug 15, 2026

   ContributorAuthor

   More actions

   Fixed and verified in production.

   Commits: 35301fe (the fix) and d8cc2bb (a regression the first one introduced — see below).

   What changed

   AC1 — the derivation moved into runtimeResponse, which runtimeNodeResponse now composes. There is one serialiser, so there is no bypass. A stored offline still wins, so it can only move the answer toward offline.

   AC2 — scoped to one node, at the write. updateRuntimeStatus's instance_runtimes UPDATE now carries the node filter its per-node sibling always had, so the shared row's liveness can only be refreshed by the machine it names. runner_node = '' (the pre-0030 default) still matches, or every registration older than that migration would go permanently unheard-from. Fixing the write rather than reconciling on read: the read-side repair leaves the corruption in the row and needs redoing at the next publisher.

   Also: last_seen_at stops advancing on an offline write. It means "when we last heard from this runner" and was being stamped at the moment we concluded we had not.

   AC3 — the probe passes the node it actually probed (both the success and the failure path), and echoes the timestamp the write left instead of now. An always-fresh stamp made heartbeatFresh true by construction, which is precisely how the more thorough call became the less truthful one. That also gives instance_runtime_nodes.status its first offline writer — #570's premise was that exactly one call site passed a node and it passed "online".

   AC4 — the guard is workers/api/src/lib/status-domain.ts + status-domain.test.ts, built once for this issue and #590. Denominator: 14 status columns, 52 declared values, 4 unwritable by application code, and it asserts nothing takes a decision on any of them. Allowlist ships empty. It would have caught #570, this, and #590 — including lib/runner-client.ts:73, which routes on WHERE status != 'offline', a value that had no writer at all until this change.

   AC5 — red demonstrated: reverting the one-line derivation fails 4 of the 9 new cases in lib/runtime-response.test.ts, including the measured 10h36m fixture.

   Verified live (post-deploy, 22 instances with a runtime)

   	before	after
   runtime.status == "online"	22 of 22	20 of 22
   runtime disagrees with its own named node	7 of 22	0 of 22

   The two now reporting offline are exactly the two this issue names — 12ebf1f0 (Coder Home) and 6d2d9401 (Facebook Friends) — both now offline for RLs-MacBook-Air.local, agreeing with that machine's own node row.

   Probe vs un-probed, 8 sampled: 8 agree, 0 disagree.

   A regression this fix introduced, caught in production and fixed forward (d8cc2bb)

   heartbeatFresh parses a stored stamp as `${s.replace(" ", "T")}Z`, because D1 writes datetime('now') without a zone. The probe echoed new Date().toISOString(), which already ends in Z — so it became …ZZ, Date.parse returned NaN, and the row read as never heard from. Harmless while the status was published raw; the moment 35301fe made the status derive from that stamp, the probe started reporting offline for machines it had just successfully reached. Measured: probe status=offline lastSeen=2026-08-15T10:54:51.917Z — an offline verdict carrying a four-second-old timestamp, self-contradictory on its face.

   d1Timestamp() now owns that shape next to the derivation that consumes it, and a guard walks the source for any last_seen_at: built with toISOString. Every fixture in the test file was already written in the D1 shape, which is exactly why 12 green tests could not see it.

   Not done

   routes/instances-tasks.ts has four updateRuntimeStatus call sites that pass no node. They resolve their runtime from the default row and write back to it, so they are self-consistent and were not part of the measured blend — but they cannot mark a node down. Left alone deliberately: outside this issue's files, and a separate change.
3. serge-ivo

   closed this as completedon Aug 15, 2026
4. serge-ivo

   added 2 commits that reference this issue on Aug 18, 2026

   fix(instances): cancelling one instance no longer retires the subscri…

   ...

   9c3d45e

   fix(runtime): every runtime-status writer can now mark a node down, a…

   ...

   299fe30