You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(server): bound the provisioning-deadline scan's lock wait #4547
I run multi-replica OpenShell gateways with PostgreSQL. When someone changes a provider or a workspace profile, start, stop and delete on a provisioning sandbox shouldn't stall behind the gateway's background deadline scan.
Problem Statement
reconcile_provisioning_deadlines (crates/openshell-server/src/compute/provisioning_deadline.rs) takes the local S(global) S(workspace) X(sandbox) keys of each provisioning sandbox through lock_sandbox_local_in_workspace, and waits for them with no deadline. Keys are taken in ascending key order. When a sandbox's key sorts before its workspace's key, the scan holds X(sandbox) while it waits for S(workspace) behind a provider or profile change on that workspace. That change holds X(workspace) through its PostgreSQL wait (up to 10 seconds) and its guarded work.
Start, stop and delete for that sandbox wait on X(sandbox) the whole time. The scan is sequential, so every sandbox after it in the pass waits too.
Impact / Why This Matters
A slow provider or profile change delays lifecycle operations on a sandbox it doesn't touch, and delays deadline expiry for the rest of the scan.
Proposed Design
The scan waits for a sandbox's keys for a short bound, well under the 10-second lock wait. If they aren't free by then, it skips that sandbox and picks it up on the next pass a second later. The local acquisition is already cancel-safe, so giving up releases the keys it took. Expiry can run a few passes late, against a 300-second repair window.
Acceptance Criteria
Provisioning-deadline expiry bounds its lock wait, with a test using a sandbox whose key sorts before its workspace's key while a provider change holds the workspace.
Alternatives Considered
Taking the workspace key before the sandbox key breaks the ascending key order that keeps the locks deadlock-free.
Dropping S(workspace) would let the scan re-derive configuration while a provider or profile write is in flight.
A try-lock with no wait would skip a sandbox on routine contention, like a provider write that finishes in a few milliseconds.
Activity
fix(server): bound the provisioning-deadline scan's lock wait