Skip to content

fix(lock): serialize writer and daemon ownership mutations - #3205

Open
bob-codedaptive wants to merge 1 commit into
colbymchenry:mainfrom
codedaptive:pr/lock-takeover-race
Open

bob-codedaptive wants to merge 1 commit into
colbymchenry:mainfrom
codedaptive:pr/lock-takeover-race

Conversation

@bob-codedaptive

@bob-codedaptive bob-codedaptive commented Nov 1, 2026 •

Copy link
Copy Markdown

Summary

#3056 serializes stale takeover of the database FileLock, but writer and daemon PID records still have separate check-then-unlink and check-then-rename paths. A process paused after its ownership check can delete or overwrite a live successor's record when it resumes.

Use FileLock's existing .break protocol as a short-lived mutation barrier for writer.pid, rebuild.pid, and daemon.pid. Initial publication, stale removal, readiness updates, handover, refresh, release, and authoritative daemon-artifact cleanup all participate. The FileLock implementation and #3056's database-lock protection are unchanged.

Preserve fresh empty exclusive-create publications for ten seconds and treat unreadable records conservatively. Owner mutations wait up to five seconds for brief contention; same-process acquisition also waits when contention reveals its own ownership, then revalidates under the barrier. The barrier is synchronous and is never held across an await.

The barrier file is removed with its own retried unlink (25, 50, 100 and 200 ms, for a Windows sharing violation) instead of FileLock.release(), which forgets its ownership even when the unlink failed and would leave a live pid in the barrier for the rest of the process. A removal that still fails after the retries is remembered and retried before the next use of that barrier. A project's owner sets caughtUp only after readiness was published, so a publication that timed out on the barrier is tried again by the next access or timer tick instead of leaving writer.pid at ready: false until the project is released.

Daemon shutdown sweeps its socket and registration before engine teardown can release writer ownership. Bind and refresh failures follow the same cleanup order, and cleanup checks daemon ownership before removing artifacts.

Regression evidence

The new __tests__/pid-lock-takeover.test.ts uses separate live processes and filesystem barriers at the final unlink or rename. It exercises writer/rebuild takeover, writer handover/readiness/release, daemon stale clearing and refresh, authoritative registry cleanup, fresh empty publication, brief owner contention, and shutdown socket ownership.

With src/mcp/daemon-registry.ts, daemon.ts and writer-lock.ts restored to unchanged upstream 5cd8a9a8, the new src/mcp/pid-lock.ts removed, and the tests kept, npx vitest run __tests__/pid-lock-takeover.test.ts fails all 16 cases:

  • 8: a competing writer, rebuild, swap, ready, release, daemon, daemon-refresh or daemon-cleanup mutation discards or overwrites a live successor's record.
  • 1: shutdown releases ownership before it has removed its socket.
  • 4: release, ready, swap or refresh does not wait for a briefly held barrier.
  • 3: a fresh empty writer.pid, rebuild.pid or daemon claim is removed during exclusive-create publication.

Two further test files cover the barrier removal and the readiness retry. Each fails with its source change reverted:

  • __tests__/pid-lock-barrier-release.test.ts (2 tests): one injected EBUSY on the barrier removal leaves the barrier on disk (expected true to be false); five injected refusals leave it holding the process's pid.
  • __tests__/project-lifecycle-ready.test.ts (1 test): after a failed readiness publication the next ready() does not publish again (expected "vi.fn()" to be called 2 times, but got 1 times).

With the change, on Node 26.12.0:

  • npx vitest run __tests__/pid-lock-takeover.test.ts: 16 passed.
  • npx vitest run __tests__/pid-lock-barrier-release.test.ts __tests__/project-lifecycle-ready.test.ts: 2 files, 3 passed.
  • npx vitest run __tests__/writer-lock.test.ts __tests__/mcp-writer-lock.test.ts: 2 files, 13 passed (7 and 6).
  • npx vitest run __tests__/filelock-stale-takeover.test.ts __tests__/daemon-older-version.test.ts __tests__/daemon-bind-failure.test.ts: 3 files, 39 passed, 1 skipped (7 passed and 1 skipped, 28 passed, 4 passed).
  • npx tsc --noEmit -p . completed without diagnostics.

Vitest's setup automatically rebuilds stale distribution assets. No full-suite, Linux, or Windows run was performed. The shutdown socket regression is POSIX-only.

  • Platform: tested on macOS only; not run on Linux or Windows.

Independence and limits

Based directly on upstream 5cd8a9a8; not stacked on another change and does not require copied-index identity stamps. The two changes edit some of the same lock functions, so applying both may require resolving textual conflicts.

Serialization requires participating processes to honor the mutation barrier; older versions do not. Discovery-registry writes and pruning are unchanged. A suspended barrier owner can exhaust the bounded wait: cleanup then preserves artifacts, acquisition reports contention, and readiness or refresh reports an error. Dead barrier owners use FileLock's existing stale-breaker recovery; live or reused PIDs can require inspection of the named .break record.

Changelog

  • Concurrent server startup, handover, and shutdown no longer let stale lock cleanup delete or overwrite another process's live ownership records.

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T03:14:58.728600Z c42a83f New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e4e2ecc513

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/mcp/pid-lock.ts Outdated
try {
return action();
} finally {
barrier.release();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Recover failed barrier removal before forgetting ownership

If removing a mutation barrier briefly fails, such as during a Windows sharing violation, FileLock.release() swallows the unlinkSync error and clears its held flag, leaving a .break record naming this still-live process. Subsequent mutations of that PID record are then blocked for the rest of the process lifetime because stale recovery preserves live holders. In a Linux source-level probe, injecting just one EBUSY on barrier removal allowed writer acquisition to succeed but made markWriterReady() time out and releaseWriterLock() leave ownership behind after the filesystem error had cleared. Retry or recover barrier removal before forgetting ownership, and verify this failure path on Windows under AGENTS.md's real-platform validation requirement.

AGENTS.md reference: AGENTS.md:L194-L194

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: FileLock.release() forgets its ownership even when the unlink fails, which leaves a live pid in the .break file and blocks every later mutation from this process. withPidLock now removes the barrier itself, checking ownership and retrying a refused unlink at 25, 50, 100 and 200 ms. A removal that still fails is retried before the barrier's next use. FileLock is unchanged. __tests__/pid-lock-barrier-release.test.ts injects one EBUSY and then five, and both tests fail without the change. The Windows sharing violation is simulated there, not run on Windows. The same push also makes pid-lock-takeover.test.ts publish each marker file only once it is whole, since on a loaded machine its waiter could read one while it was still empty.

Comment thread src/mcp/writer-lock.ts
} finally {
try { fs.unlinkSync(tmp); } catch { /* best-effort */ }
}
}, () => { throw new Error(`Timed out publishing writer readiness; inspect ${pidPath}.break for its current owner.`); }, { wait: true });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep readiness publication retryable after barrier timeouts

If a contending process holds writer.pid.break beyond the five-second deadline at the end of initial catch-up, this throws after activate() has already set project.caughtUp = true (src/mcp/project-lifecycle.ts:177). ready() then short-circuits for that owner, so neither its periodic timer nor later queries retry publication after the barrier becomes free. The PID record stays ready: false until the owner releases the project, causing other explicit-project readers of a direct/fallback owner to keep entering the 30-second readiness wait. A source-level probe confirmed that releasing the contending barrier and calling ready() again still leaves readiness false. Set caughtUp only after successful publication, or retain a separate publication retry.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed: caughtUp was set before readiness was published, and the PR makes publication throw on a barrier timeout, so ready() never tried again. activate() now publishes first and sets caughtUp only afterwards, so the next access or timer tick retries. __tests__/project-lifecycle-ready.test.ts makes the first publication throw and checks that the next ready() leaves writer.pid at ready: true. It fails without the change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant