Skip to content

ProjectOps.Create/Update race their queued closure when the caller's context is cancelled #85

Closed

Description

@barelyworkingcode

Problem

ProjectOps.Update and Create run their mutation through runQueued (Queue.Do). If the caller's context is cancelled after the command is admitted, Do returns ctx.Err() while the closure may still be running. The closure ignores that context, so it can still commit. Meanwhile Update reads updated, found and updateErr straight away. That is a data race, and the caller can be told a write failed when it happened.

Root cause

These ops use runQueued where runCommitted, which waits for the closure to finish, exists for exactly this case.

Fix

Switch both to runCommitted, or wait for the closure before reading its results. Test under -race with a context cancelled after admission.

Found in the review of #46. It predates that change.

Activity

  1. added
    needs-reviewAwaiting owner review before work starts
    and removed
    needs-reviewAwaiting owner review before work starts
    on Sep 26, 2026
  2. barelyworkingcode commented on Sep 26, 2026

    @barelyworkingcode
    OwnerAuthor

    Approved 2026-09-26 by the coordinator under the owner's delegation: a real data race that can report a failed write that actually committed, with an existing helper (runCommitted) as the fix. Queued right after #46, which touches the same code.

  3. barelyworkingcode commented on Sep 26, 2026

    @barelyworkingcode
    OwnerAuthor

    Contract

    Approved by the coordinator.

    Scope. Four ProjectOps methods (cmd/relay/project_ops.go) run a cancellable ctx through runQueued and read the closure's outputs after it returns: Create, Update, RotateToken and NarrowForEnrolment. RotateToken is the worst of them: a cancelled caller is told the rotation failed while the new token is already stored, and its plaintext is gone. Each switches o.runQueued(ctx, …) to o.runCommitted(ctx, …). That is the existing helper (Queue.DoCommitted), which never abandons an admitted step on caller cancellation. Remove already uses context.Background() and is unchanged.

    Nothing else changes: signatures, return tuples, gate, audit and wire shapes stay the same. The structural queue test already accepts runCommitted.

    Test (red first). New file cmd/relay/project_ops_commit_race_test.go, run with -race. For each of the four methods:

    1. Build a ProjectOps with a real config.CommandQueue (or whatever Queue field type production uses) and a store whose write blocks on a channel once the closure is admitted.
    2. Cancel the caller's context.
    3. Release the write.
    4. Assert that the method returned the committed result (not ctx.Err()), that the stored state matches it, and that -race reports nothing.

    Where blocking inside the store is impractical, a hook store in the style of odwHookStore is fine. Tests must be red, or race-flagged, on the current code.

    Files. Dev: cmd/relay/project_ops.go. Test writer: cmd/relay/project_ops_commit_race_test.go (new).

  4. barelyworkingcode commented on Sep 26, 2026

    @barelyworkingcode
    OwnerAuthor

    Contract amendment (coordinator): the widening to four methods is approved. RotateToken's test must pin the invariant:

    • When the call reports success, the returned token is the stored one.
    • When the context is cancelled, the caller either receives the new token or storage is unchanged. There must never be a stored new token the caller did not receive.
  5. barelyworkingcode commented on Sep 26, 2026

    @barelyworkingcode
    OwnerAuthor

    Contract amendment 2 (coordinator): the invariant excludes the deliberate audit-failure path. When recordProjectTokenRotated fails, RotateToken commits the rotation and withholds the token (errProjectTokenUnrecorded), so issuance is never unrecorded. Outside that path: on success the returned token is the stored one, and on cancellation the caller receives the new token or storage is unchanged. A rotation cancelled while it is still queued, but not yet started, now runs (accepted; the cancel-until-started queue mode is filed separately at low priority).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions