feat(server): scope mutation locks and bound lock waits #4501

User Story

I run multi-replica OpenShell gateways on Kubernetes with an external PostgreSQL database. When replicas roll, scale down or redirect sessions, many supervisors reconnect at once, and I need those reconnects and other sandbox changes to keep going instead of queuing behind one lock.

Problem Statement

Every guarded mutation takes the same process-wide lock and the same fleet-wide PostgreSQL advisory key, so changes to unrelated sandboxes and workspaces wait on each other across all replicas. Each guard also opens a fresh connection from the 10-connection data pool, competing with its own queries, and a lock wait that hits PostgreSQL's 10-second lock_timeout surfaces as INTERNAL.

Impact / Why This Matters

A reconnect burst during a rollout or an HPA scale-down queues every sandbox operation in the fleet behind it, and clients get INTERNAL errors they can't tell apart from real failures. Operators can't size around it, because the pool ceiling is fixed in the binary.

Proposed Design

• Mutations lock at global, workspace or sandbox scope, so changes to different sandboxes and workspaces run concurrently on every replica. Replicas still on the previous release keep excluding new ones during a rolling upgrade.
• A lock wait gives up after 10 seconds with UNAVAILABLE and reason MUTATION_LOCK_TIMEOUT, so clients know to retry.
• Operators can size the database pool and the lock pool per replica, and see lock waits, timeouts, errors and hold times in the gateway metrics.

Suggested UX

gatewayConfig:
  openshell.gateway:
    database_max_connections: 20
    database_lock_max_connections: 8

Acceptance Criteria

• Mutations of different sandboxes, on the same or different replicas, don't wait on each other.
• A lock wait past 10 seconds returns UNAVAILABLE with MUTATION_LOCK_TIMEOUT instead of INTERNAL.
• Old and new replicas still serialize their mutations during an upgrade.
• Both pool sizes are configurable, and lock metrics are exported.

Alternatives Considered

Raising the data pool while keeping the fleet-wide lock only lengthens the queue: every mutation still waits for every other one. Routing all mutations through one leader replica would also remove the contention, but it needs a leader election and failover path the gateway doesn't have today.

Part of #3528. #4147 implements this, and follow-ups are tracked in #4307.