Skip to content

[bug] A run is admitted onto a folder the platform has already judged "not a git working tree", and starting it overwrites that verdict with ready — three git pull runs died on it #548

Closed

Description

@serge-ivo

git pull failed three times overnight in a folder that has no .git

Investigated from the owner's report "Two git pull runs failed overnight — both timed out (stuck
not resolved in time)"
(the agent's own summary, instance_messages, 2026-08-12 22:38:11Z), plus a
third failure the same night.

Instance a1d3522f-6b91-4cd9-bf7b-380e470192b7 (Repo Coder), repo dev/aipa, workdir ~/dev/aipa.

On the machine, right now:

$ ls -d ~/dev/aipa          → /Users/serge-ivo/dev/aipa
$ ls -d ~/dev/aipa/.git     → ls: No such file or directory
$ ls -la ~/dev/aipa         → one entry: platform/

The platform already knows this. GET /v1/instances/a1d3522f…/coding/repos:

{"id":"repo_145254cf-…","name":"dev/aipa","provider":"local","workdir":"~/dev/aipa",
 "cloneStatus":"needs_attention",
 "cloneError":"The configured checkout `/Users/serge-ivo/dev/aipa` has files but is not inside a git working tree — it is a plain folder, not a clone of a repository.",
 "updatedAt":"2026-08-12 22:41:20"}

That sentence is lib/coding-workdir.ts:111, written by #405's machinery, working exactly as
designed. updated_at is 22:41:20 — twenty seconds after the run's last instruction and
fifteen minutes before it gave up. The verdict was on the row while the run was still going.

The run started anyway, and nothing consulted the verdict

codingDriver.start (workers/api/src/lib/loop-drivers.ts:150-284) is the path behind
start_work, the Loop button and delegate_goal. Its admission checks, in order:

  • pickLoopRepo (:156) — is there a repo, and is it this agent's;
  • runtimeConnectivity + classifySubordinateConnectivity (:163-175) — is a machine reachable;
  • claimSessionDriver (:192) — is somebody else already driving.

There is no fourth check. repo.cloneStatus is never read in this function — grepped across
routes/coding*.ts, lib/coding-store.ts and workflows/coding-session.ts: the only readers of
cloneStatus are the console (ReposList.tsx:150, CodingTab.tsx:405,814), the chat prompt
(lib/repo-status-prompt.ts), diagnostics (routes/coding-diagnostics.ts:233,265) and repo
selection for the local-repo connector (lib/connectors/repo-local.ts:128). Nothing on the run
path.

Worse: starting the run overwrites the verdict with ready

workers/api/src/lib/coding-session-open.ts:129, in startSessionOnRunner, immediately after a
successful POST /coding/start:

await updateRepoClone(env, repo.id, { cloneStatus: "ready", cloneError: null });

Nothing looked at the directory. /coding/start succeeding means the runner could chdir there and
was willing to run a command
— a plain folder satisfies that. So the run's own first act is to
erase the sentence that would have stopped it.

This is the same invariant the file states 4 lines below, in the comment #440 put there:

// A REPO'S STATE IS ONLY EVER WRITTEN BY SOMETHING THAT LOOKED AT THE REPO (#440).

#440 closed the false-error direction (a dropped WebSocket stored as the repo's state; the
isRunnerUnreachable guard at :149 is that fix, and it is in the code). The false-ready
direction, four lines above it, was not addressed — and it is ready, per #405, that makes an
agent invent code.

The third safety net is silent too

workflows/coding-session.ts:587-591 runs a repo-state-start step before the first instruction:

return (await readRepoWorkingState(conn, { repo, sessionId })) ?? null;

readRepoWorkingState (lib/repo-state.ts:120-136) runs git status over the relay and returns
null on any error — which is right for its own purpose (an absent report reads as "unknown", not
as a fabricated clean tree) but means fatal: not a git repository arrives as null, stateNote
is skipped, and the Pilot is told nothing.

The composition

Three individually-defensible decisions:

  1. codingDriver.start checks connectivity, not repo health;
  2. startSessionOnRunner treats a successful spawn as proof the checkout is good;
  3. readRepoWorkingState degrades a git error to "unknown".

Together: a run is admitted onto a non-repository, the evidence against it is deleted, and the one
component that could have said so says nothing. What the owner got was three git pull attempts
(22:40:52, 22:40:56, 22:41:00), an unanswered stuck-handoff, and at 22:56:23 the message
"Loop stopped (failed) — stuck not resolved in time". Nowhere in that chain does the phrase
"not a git repository" appear, though the platform had it in D1 at 22:41:20.

The engine-side half of the same run — every Codex turn exiting 1, unclassified — is filed
separately as #545; either fix alone would have shortened this failure, and neither subsumes
the other.

What to do, cheapest first

1. Refuse the run. In codingDriver.start, after pickLoopRepo and before opening a session:

if (repo.cloneStatus === "needs_attention") {
    return { ok: false, status: 409, error: repo.cloneError || `${repo.name}'s folder is not usable — fix it on the Coding tab.` };
}

cloneError is already written to be relayed verbatim (coding-workdir.ts:47-51 — "an agent can
say it to the owner"). A 409 with that sentence turns 15 minutes and three BYOK decisions into one
honest reply the owner can act on. Only needs_attention, which #405 defines as a definite
runner verdict — unknown and unverified must NOT block, or an offline laptop stops every run.

2. Stop writing ready from a spawn. coding-session-open.ts:129 should either drop the write
or take a real verdict. Since startSessionOnRunner already holds a live conn, the honest form is
checkWorkdirVia(conn, repo.workdir) + cloneStatusForVerdict — the pair routes/coding-repos.ts:105-110
already uses, which returns null (no write) for unverified. For a managed clone (no
workdir) the existing ready write is fine and should stay: something did look, and it cloned it.

3. Let the Pilot see it. readRepoWorkingState returning null for "git said no" and for "the
runner did not answer" is one value for two facts. Distinguish them, and let
describeRepoState/stateNote carry "this checkout is not a git repository" into the prompt — the
channel is already built and already injected at workflows/coding-session.ts:596-603.

Alternatives considered and rejected

  • Re-check the workdir at run start instead of reading the row. A relay round trip (5s timeout)
    on every start_work, to re-derive a verdict the list route already takes. Reading the row is
    free; step 2 is what keeps it trustworthy. Worth revisiting only if [bug] A repo has been marked broken for five days by a dropped WebSocket — a transport failure is stored as the repo's state, and no verdict is ever re-taken #440's "no verdict has been
    written in five days" case turns out to be common.
  • Auto-git init / auto-clone into the folder. No. ~/dev/aipa holds platform/ — quite
    possibly the user's real intent was ~/dev/aipa/platform. Guessing writes into a directory the
    owner did not ask us to touch, and the same argument the workflow already makes about not
    discarding an unexpected branch (coding-session.ts:582-586) applies here.
  • Refuse at repo-add time (400 instead of 201). Explicitly rejected by
    routes/coding-repos.ts:290-296, and correctly: the row is the handle the owner needs to fix the
    path. Nothing here changes that — the refusal belongs at run start, not at add.

Acceptance criteria

  1. start_work / Loop / delegate_goal on a repo whose cloneStatus is needs_attention returns
    409 carrying cloneError, and no session is opened, no loop-run row created, no budget pool
    opened.
  2. cloneStatus unknown or unverified still starts a run (a unit test for each).
  3. A successful /coding/start on a local workdir no longer writes ready without a
    checkWorkdirVia verdict; a managed clone still may.
  4. Replaying this repo's state, the failure the owner sees names the folder and the word "git",
    within seconds, instead of stuck not resolved in time fifteen minutes later.
  5. repo-state-start distinguishes "the runner could not answer" from "git said this is not a
    repository", and the second reaches the Pilot's prompt.

Regression risk

Not from feedback

This one was investigated from the owner's observation of repeated git pull failures, not from a
record_feedback row — there is no feedback row to stamp.

Activity

  1. serge-ivo commented on Aug 13, 2026

    @serge-ivo
    ContributorAuthor

    Fixed in 2d56156 — CI green, API Worker deployed, migration N/A.

    What shipped

    All three halves the issue asks for, and the acceptance criteria in order.

    1. Admission. lib/coding-repo-admission.ts (new, pure) + a gate in codingDriver.start, so start_work / the Loop button / delegate_goal refuse a repo whose stored verdict is needs_attention, relaying clone_error verbatim. Two deviations from the sketch in the issue, both deliberate:

    • The check sits after the connectivity gate, not immediately after pickLoopRepo. With the machine off, a needs_attention verdict can be days stale and the runner diagnosis is the truer, more actionable blocker; with the machine up, the folder verdict is the live one and is said first. A test pins both orderings.
    • The refusal is not routed through noSessionMessage, exactly as the Regression risk section warns. A test asserts the string pags up never appears in it, and that the message names Re-check — the remedy that lifts the block is the same probe that imposed it.

    unknown, cloning, missing_url and a stale error all still start (one test each; the error case cites #440's five-day false verdict).

    2. The overwrite. startSessionOnRunner no longer writes ready from a successful spawn. recordStartVerdict takes a real verdict via checkWorkdirVia + cloneStatusForVerdict and stamps clone_checked_at; unverified writes nothing, so a pags up older than /coding/repo-check cannot condemn a healthy repo. A managed clone (no workdir) keeps the unconditional ready and is not probed, per the issue. Framed in the commit as the mirror of #440: that stopped a disconnect demoting a good row, this stops a spawn promoting a bad one.

    3. The silent net. readRepoWorkingState stopped collapsing "the runner did not answer" and "git said this is not a repository" into one null. RepoWorkingState grows notAGitRepo, set only from a definite refusal (matched on the runner's own not a git repo and git's fatal: not a git repository), and describeRepoState emits a clause that replaces the other two — a path with no .git has no branch to be off and no diff to protect. That reaches the Pilot through the stateNote channel that already existed, so workflows/coding-session.ts needed no change (another agent was working in that file).

    One thing the issue did not anticipate, found while making that change: evaluateRepoPolicies scored the new state held for every policy, because both clauses return null for it — a claim of compliance about a repository that is not there. It now scores unknown. Guarded by its own test.

    Verified

    • pnpm typecheck, pnpm vitest run (8108 passed), biome, and all seven script gates including check-file-size.mjs and check-migrations.mjs --require-history — run in a throwaway worktree at origin/main, because the shared checkout had another agent's in-progress edit to workers/api/src/lib/usage.ts, which was failing typecheck and is not mine.
    • CI + Deploy API Worker both green on 2d56156.
    • Production, after the deploy — GET /v1/instances/a1d3522f…/coding/repos:
      dev/aipa | ~/dev/aipa | needs_attention | checkedAt = 2026-08-13 00:42:42
        The configured checkout `/Users/serge-ivo/dev/aipa` has files but is not inside a git working tree…
      recheck: {ran: true, checked: 1}
      
      The verdict is live and freshly re-taken (ran: true), not a stale row — so the new gate is acting on a measurement a minute old. A run on this repo is now refused with that sentence instead of admitted.

    Not verified

    I did not trigger a run to see the 409 in production — that would mean driving the owner's live agent. The refusal is covered by unit tests against the real repo row shape; the production check above is read-only.

    Left open, deliberately

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions