Skip to content

fix(server): bound the provisioning-deadline scan's lock wait #4547

Open

Description

@EmilienM

User Story

I run multi-replica OpenShell gateways with PostgreSQL. When someone changes a provider or a workspace profile, start, stop and delete on a provisioning sandbox shouldn't stall behind the gateway's background deadline scan.

Problem Statement

reconcile_provisioning_deadlines (crates/openshell-server/src/compute/provisioning_deadline.rs) takes the local S(global) S(workspace) X(sandbox) keys of each provisioning sandbox through lock_sandbox_local_in_workspace, and waits for them with no deadline. Keys are taken in ascending key order. When a sandbox's key sorts before its workspace's key, the scan holds X(sandbox) while it waits for S(workspace) behind a provider or profile change on that workspace. That change holds X(workspace) through its PostgreSQL wait (up to 10 seconds) and its guarded work.

Start, stop and delete for that sandbox wait on X(sandbox) the whole time. The scan is sequential, so every sandbox after it in the pass waits too.

Impact / Why This Matters

A slow provider or profile change delays lifecycle operations on a sandbox it doesn't touch, and delays deadline expiry for the rest of the scan.

Proposed Design

The scan waits for a sandbox's keys for a short bound, well under the 10-second lock wait. If they aren't free by then, it skips that sandbox and picks it up on the next pass a second later. The local acquisition is already cancel-safe, so giving up releases the keys it took. Expiry can run a few passes late, against a 300-second repair window.

Acceptance Criteria

  • Provisioning-deadline expiry bounds its lock wait, with a test using a sandbox whose key sorts before its workspace's key while a provider change holds the workspace.

Alternatives Considered

  • Taking the workspace key before the sandbox key breaks the ascending key order that keeps the locks deadlock-free.
  • Dropping S(workspace) would let the scan re-derive configuration while a provider or profile write is in flight.
  • A try-lock with no wait would skip a sandbox on routine contention, like a provider write that finishes in a few milliseconds.

Part of #4307.

Activity

  1. added
    state:acceptedA maintainer decided OpenShell should pursue this issue
    and removed
    state:triage-neededOpened without agent diagnostics and needs triage
    on Oct 14, 2026
  2. added a commit that references this issue on Oct 16, 2026
    098b69c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:acceptedA maintainer decided OpenShell should pursue this issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions