Repository navigation
fix(gateway): make the renew and rotation locks atomic within a node - #1357
Merged
Merged
Conversation
kvinwang
added this pull request to stack #1361
September 24, 2026 08:06
kvinwang
force-pushed
the
fix/gateway-cert-lock-atomic
branch
from
September 24, 2026 08:07
8a27fa1 to
2aca705
Compare
Both lease locks read under a read guard, dropped it, then wrote under a write guard, so two tasks on one node could both acquire: with eight threads racing on one KvStore, several acquired the same cert lock. The background renew loop and an operator's RenewZtDomainCert reach that concurrently, as do rotation, CAA reconciliation and first-use registration for the rotation lock. Both locks now go through one try_acquire_lock / release_lock pair that holds a single write guard across read, expiry check and write. Release only deletes the lock it acquired, which the rotation lock already did and the cert lock did not: a renewal that outlived the timeout deleted its successor's lock. Admin.ForceReleaseCertLock keeps an unconditional delete as force_release_cert_lock. Across nodes this stays best-effort: WaveKV is last-writer-wins without compare-and-swap, so two nodes can both acquire during a replication gap.
kvinwang
force-pushed
the
fix/gateway-cert-lock-atomic
branch
from
September 24, 2026 08:16
2aca705 to
ee333d8
Compare
kvinwang
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split from #1235.
Both lease locks (per-domain cert renew lock, global ACME rotation lock) read under a read guard, dropped it, then wrote under a write guard, so two tasks on one node could both acquire. That is reachable: the renew loop races an operator's
RenewZtDomainCert, and rotation, CAA reconciliation and first-use registration share the rotation lock. Concurrent renewals delete each other's_acme-challengeTXT records.try_acquire_lock/release_lockpair holding a single write guard across read, expiry check and write.Admin.ForceReleaseCertLockkeeps an unconditional delete asforce_release_cert_lock.Across nodes this stays best-effort: WaveKV is last-writer-wins without CAS.
two_tasks_on_one_node_cannot_both_acquireanda_holder_that_was_taken_over_does_not_release_the_new_holder_lockfail againstnext.