Repository navigation
Sync-check fix (#785) is not reliably working: runner /coding/sync 404s recur, checkouts still fall behind silently #801
Description
Issue #785 (commit 35fbf6d, closes #785) added staleness detection and a start/end-of-session sync check meant to stop a run from ever reading or working on a stale checkout without saying so. In practice this is not holding up.
Evidence — occurrence 1 (filed separately as #800): during a HeartFull-online/platform coding run (session csess_a73313bf-6537-4e8a-9f91-05b41b83d170, run starting 2026-09-11 03:13 UTC), both the start-of-run and end-of-run sync check logged:
"Whether this checkout is up to date could not be checked: Runner /coding/sync -> 404: {"error":"Not found"}"
The run proceeded anyway on the existing local checkout with no confirmed-fresh sync. It happened to land correctly this time, but only by luck.
Evidence — occurrence 2: this ProAgentStore/platform coder instance's own local checkout was found to be 6 commits behind origin/main (git status reporting "main...origin/main [behind 6]") during a live session, despite commit 35fbf6d (the #785 fix itself) being present in that same checkout's own history. So the mechanism exists in code but the checkout it is supposed to protect was still stale in practice, with no warning surfaced before it was noticed manually.
Conclusion: the #785 fix's intent (never silently work on a stale checkout) is not reliably achieved. The most likely shared root cause across both occurrences is the runner's /coding/sync endpoint itself returning 404 rather than a real up-to-date/behind answer, which the caller currently treats as a soft failure (log and continue) rather than a hard block or loud warning.
Requested: 1) investigate why runner /coding/sync 404s in practice (endpoint missing/misrouted on some runner nodes, or a version skew between runner and worker), 2) decide whether a failed sync check should hard-block a run from proceeding rather than logging and continuing silently, 3) add monitoring/alerting so repeated /coding/sync 404s are visible to an operator rather than only discoverable by reading a run's raw log after the fact.
Related: #785 (the original fix this regresses/fails to fully deliver), #800 (the HeartFull occurrence).
Activity
feat(coding): a run does not build on a base it could not confirm (#801)
serge-ivo commentedon Sep 11, 2026
Root cause, and the gate, in 0af538a
The 404 is version skew, and the sync code is not at fault
/coding/syncis served atpackages/browser-runner/src/server.ts:360and first shipped in CLI 0.4.58 — the releaseREPO_SYNC_MIN_CLI(workers/api/src/lib/repo-sync.ts:38) already names. A runner below that 404s it, exactly as the comment above the route says it will:Measured on this platform's own coder machine during the investigation: the live runner is
@proagentstore/cli@0.4.57. One patch below the minimum. So occurrence 1 (#800), occurrence 2, and a third found in this session's own run log are all the same cause, and none of them is intermittent — on a machine below 0.4.58 the endpoint 404s every time.That also means #785's design held up under both occurrences.
verdictFromSyncreturnsunverified, neverin_sync, for a runner that does not answer. The cloud never claimed the checkout was fresh.What was actually broken: the last mile of the sentence
readRepoSynccaught the 404, stringified it, andverdictFromSyncinterpolated it verbatim. The one sentence the owner saw was an HTTP artefact:Nobody reading that can tell it means "upgrade this machine's CLI". That is why two occurrences were logged and neither was acted on — the warning fired correctly and was unreadable.
And the platform already knew how to say it properly.
runner-upgrade.ts(#524) names the machine, the version it runs, the pin holding the agent there, and a capable alternative machine if one exists.repo-local.tsroutes/coding/search's 404 through it. Sync never did —REPO_SYNC_MIN_CLIwas consumed only on the two/coding/gitpaths, never on the/coding/syncpath where the 404 actually happens.What landed
New
workers/api/src/lib/repo-sync-gate.ts— pure decision half, impure half that names the machine and records. Wired intocoding-session.tsat the existingrepo-state-startread.Ask 1 — actionable message. The 404 now routes through
runnerUpgradeRefusal, so the refusal names the machine, its version, the pin, and a capable machine.`mbp` needs a newer runner to check whether this checkout is up to date (it has 0.4.57) — this needs CLI 0.4.58 or newer.instead of the raw 404.Ask 2 — hard block. Implemented.
behind,divergedandunverifiedstop the run before its first instruction.in_syncandaheadpass (unpushed work does not make a base stale).no_upstreamalso passes — there is no remote for a local-only checkout to be behind, and blocking it would protect it from a staleness that cannot exist.The gate sets
resultrather than throwing. A thrown gate reachescodingCrashReportand reads asrun error:— the exact confusion #523 had just closed — anddriverResumePlanwould replay it against the same unconfirmed base.Ask 3 — operator visibility. Every block writes to
error_logunder sourcecoding:sync-gate, read byGET /v1/errorsand thelist_errorsMCP tool.logErrorcollapses repeats, so "this machine has 404ed forty times since Tuesday" is one row with a count, not a grep. Deliberately not the[SYNC-404]log prefix the issue suggested: aconsoleline in a Worker reacheswrangler tailand nowhere else, and nobody was tailing at 03:13 UTC — which is exactly why #800 took a human reading a raw run log to find. 404s file aterror(a platform fault, the thing to chase); a checkout merely behind files atwarn, so it cannot drown them in the same query.Tests — 36 in
repo-sync-gate.test.ts: every verdict state, 404-vs-unreachable classification, the message naming the min CLI and the machine, repeated 404s landing in one collapse bucket, the disarm path, and six source-level assertions that the workflow actually obeys the gate (a Workflow can only be tested by running one, so the call shape is read off the file the waycoding-resume-note.test.tsdoes).Two things to know before this reaches production
1. This is armed by default, and that is the opposite of how this repo ships gates.
PAYWALL_ENFORCEandBUDGET_ENFORCEboth default to observe-only. This one blocks on deploy, per the explicit decision in ask 2. The blast radius is not hypothetical: every runner below CLI 0.4.58 stops running coding sessions the moment the API Worker deploys — including the machine that found this bug.CODING_SYNC_GATE=offstands the block down in one variable without switching off theerror_logobservation.2. The code makes the problem visible and blocking; it does not upgrade anyone's CLI. The operational fix for an affected machine is still
npm i -g @proagentstore/cliand apags uprestart — which the refusal now tells you, by name.One consequence worth flagging: the #785 instruction asking a run to make
git pull --ff-onlyits first step is now unreachable for abehindcheckout, because the run no longer starts. Self-healing is traded for never building on a stale base. If that turns out to be the wrong trade in practice, the narrow change is to passbehindwhen the tree is clean and on its configured branch, and keep blockingdivergedandunverified.Closes #801. Related: #785, #800, #524, #523.
serge-ivo commentedon Sep 11, 2026
Verified live 2026-09-11 05:22 UTC: both runner machines upgraded to CLI 0.4.58 (Mac.modem / RLs-MacBook-Air and Macmini.modem). A
repo_git statuson the PAGS platform coder instance through the 0.4.58 runner now reports(upstream: in sync with origin/main)instead of the 404. Root cause was runner version skew (0.4.57 < REPO_SYNC_MIN_CLI 0.4.58); the gate + readable refusal +coding:sync-gateerror_log rows landed in 0af538a. No new gate blocks since the upgrade.feat(coding,runner): a behind checkout is fast-forwarded before the r…
feat(coding,runner,mcp): a stale checkout heals itself past a dirty t…
fixes #917
test(#915): the Pilot's invariants are asserted by RUNNING the workfl…