You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(cuda.core): pick one owner when IPC pool registrations of one UUID race - #3524
Follow-up to #3511, which closed#3510 and deferred two review items so it could merge without another CI round while #3516 waited on it. This PR addresses both.
Pick one owner when registrations of one UUID race in MP_register. Repeat imports of one IPC pool share a single owner keyed by the pool's UUID. Selecting that owner was not atomic against same-thread reentry: inserting into the registry runs Python code (the WeakValueDictionary methods and uuid.UUID.__hash__), where a garbage collection can start and its callbacks can register the same UUID on the same thread (the lock is reentrant). The re-entrant call inserted its resource, then the outer call overwrote it, leaving two owners of one driver pool with the registry holding only one of them. The overwrite predates #3511; main registered with no lock at all. MP_register now records the resource it is inserting, under _registry_lock, before it inserts. A re-entrant registration of the same UUID returns that resource instead of inserting its own, and one that completed before the record was set is found by a second lookup. Both calls then return the same owner, and the registry holds it.
Check every supported GPU after the re-import race.test_reimport_racing_the_last_release_keeps_access checked read/write access and exercised a buffer only on the default GPU. The final release of a pinned pool revokes access one GPU at a time, so a re-import could in principle be left without access on a non-default GPU and the test would still pass. It now asserts read/write access on every GPU the pool grants access to.
Notes for reviewers
Please review the combined diff. The first commit replaced the registry with a plain dict of weak references whose removal callback took _registry_lock. That made every drop of a registered resource wait for registrations in progress on other threads, and a thread that dropped one while holding an application lock deadlocked with a registration whose garbage-collection finalizer took the same lock. The second commit restores the WeakValueDictionary and fixes the overwrite inside MP_register instead.
Testing
New test_reentrant_registration_of_one_uuid_picks_one_winner drives a deterministic regression for the overwrite through a one-shot, test-only hook in MP_register (_set_register_reentry_hook_for_testing) that registers the same UUID re-entrantly right before the outer call's insertion, where a finalizer could. With the code on main plus the hook, it fails on assert outer is winner for both PinnedMemoryResource and DeviceMemoryResource; with the fix it passes.
cuda.core only. The registry and MP_register have no counterpart in cuda_bindings_12/ or cuda_bindings_13/, and the touched files contain no BEGIN SYNCED groups. No release-note entry is added: the winner-selection fix hardens a latent, essentially unobservable condition already covered in spirit by the #3510 note on concurrent imports sharing one resource, and the GPU check is test-only.
…D race
Follow-up to NVIDIA#3511 (NVIDIA#3510). Two deferred review items:
- MP_register could overwrite a re-entrant registration of the same UUID.
A garbage-collection finalizer that registered the same pool between the
outer call's lookup and its insert left two owners of one driver pool, with
the registry pointing at the loser. Build the weak reference before the
insert and claim the slot with dict.setdefault so a re-entrant winner is
kept, not overwritten; both calls then return that one owner. A plain dict
of weak references replaces the WeakValueDictionary so the wrapper
allocation happens outside the check-and-set. A one-shot test hook injects
the reentry at the vulnerable point for a deterministic regression test.
- The re-import race test now checks read/write access on every GPU the pool
grants access to, not only the default one, since the final release revokes
access one GPU at a time.
Improved reliability when importing shared memory pools, including cases where multiple registrations occur concurrently or are nested. Imports now consistently use the same registered pool, helping prevent conflicts and resource leaks.
Expanded memory-pool access checks to cover every device supported by a pool, improving confidence that access remains available across supported devices.
The IPC registry now tracks UUIDs with registrations in progress and handles re-entrant registration without replacing the resource already registered. Tests cover the nested registration case and check pool access on supported devices.
MP_register checks for in-progress registrations, marks the UUID before insertion, and clears the marker after registration. The stub declares the test hook used before insertion.
Registration and pool access tests cuda_core/tests/helpers/memory.py, cuda_core/tests/memory_ipc/test_ipc_pool_duplicate_import.py
A helper selects devices supported by each pool type. Tests trigger nested registration for the same UUID and verify that both calls return the same resource. The access test checks all supported devices.
Linked issue assessment
Objective
Addressed
Explanation
Concurrent same-UUID registration returns the registered resource rather than failing. [#3510]
✅
The registered resource retains device access after a competing import is dropped. [#3510]
❓
The tests check the registration winner and owned-pool count, but do not check access after dropping a competing import.
Repeated raw-handle imports share ownership and avoid premature pool destruction with outstanding buffers. [#3510]
❌
The diff does not change raw-handle imports or native pool ownership and destruction.
Possibly related PRs
NVIDIA/cuda-python#3511: It changes the same IPC registration path and ownership contract. This PR follows up on re-entrant registration behavior.
Suggested reviewers:andy-jost, mdboom
Change: Bug fix · Severity of issue fixed: Medium
Merge Risk: Low · up to e04e2
This change makes nested registrations of the same IPC pool UUID return a single owner. A narrow window remains where the nested caller receives that owner before it is fully registered. The impact is limited, so the change is mergeable with owner awareness.
Security architecture risk: Low · up to e04e2
Normal reentrant registrations now converge on one resource, but a nested caller receives that resource before its registration and identity are committed. This leaves a process-local ownership consistency gap. No expansion of external access or privileges was identified.
Retained concerns
Low · reliability · inferred: The in-progress branch returns the outer owner before committing its UUID or registry entry. For a raw-descriptor import whose UUID is initially None, a reentrant caller can register that returned owner under another UUID; the outer call then assigns its original UUID, leaving two registry keys for one object with a changed identity. If the outer insertion raises instead, the nested caller can retain an owner that was never registered. This weakens identity and recovery guarantees used by UUID-based serialization and reconstruction. At the base, the corresponding nested registration completed its own identity assignment before returning.
Security review details
Security Blast Radius
inferred — The changed ownership state is process-local and UUID-keyed. Affected resources can represent device pools or pinned pools; pinned-pool imports already grant read/write access to every supporting GPU in the process. The identified consistency concern does not establish additional GPU permissions or access to another process's pool.
Trust Boundaries and Controls
inferred — The new in-progress return requires same-thread reentry while registration holds its lock, such as an executing finalizer or the local testing hook. The inspected change does not introduce a network entrypoint or a mechanism by which an IPC peer alone can install that callback.
Resilience and Maintainability Implications
inferred — The unchanged C++ import layer reuses shared ownership for repeated imports of the same driver pool. Its final shared-owner deleter revokes access and destroys the counted imports; closing a Python resource releases its handle reference. This limits the identified Python registry inconsistency and is counterevidence against inferring double destruction or premature access revocation from duplicate Python owners alone.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
The previous commit replaced the registry with a plain dict of weak
references whose removal callback took _registry_lock. That callback runs
wherever the last reference to a registered resource is dropped, including
in garbage collection on any thread, so dropping one now waited for any
registration in progress on another thread. A thread holding an application
lock while dropping a registered resource deadlocked with a registration
whose garbage-collection finalizer took the same lock. WeakValueDictionary
removes entries without a lock, through an atomic helper, for exactly this
reason.
Restore the WeakValueDictionary and its lookups unchanged. MP_register now
marks the resource it is inserting, under _registry_lock, before it inserts:
a registration of the same UUID re-entered from a garbage-collection
callback returns the marked resource instead of inserting its own, and one
that completed before the mark is found by a second lookup.
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 8 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
Minor · Data Integrity & Integration · Quick win
important: Complete registration before returning the marked owner.
With the supplied raw-fd imports, nested second.register(U) returns first before Line 665 inserts it and Line 671 sets its UUID. Inside that callback, mr_type.from_registry(U) therefore raises, although register() promises a registered resource. first.uuid also remains None until the outer call resumes. (raw.githubusercontent.com)
Make the registry entry and UUID usable before the nested call returns. Add both assertions inside the re-entry callback; assertions after the outer call cannot detect this failure.
Preview will be ready when the GitHub Pages deployment is complete.
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bugSomething isn't workingcuda.coreEverything related to the cuda.core module
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #3511, which closed #3510 and deferred two review items so it could merge without another CI round while #3516 waited on it. This PR addresses both.
Pick one owner when registrations of one UUID race in
MP_register. Repeat imports of one IPC pool share a single owner keyed by the pool's UUID. Selecting that owner was not atomic against same-thread reentry: inserting into the registry runs Python code (theWeakValueDictionarymethods anduuid.UUID.__hash__), where a garbage collection can start and its callbacks can register the same UUID on the same thread (the lock is reentrant). The re-entrant call inserted its resource, then the outer call overwrote it, leaving two owners of one driver pool with the registry holding only one of them. The overwrite predates #3511;mainregistered with no lock at all.MP_registernow records the resource it is inserting, under_registry_lock, before it inserts. A re-entrant registration of the same UUID returns that resource instead of inserting its own, and one that completed before the record was set is found by a second lookup. Both calls then return the same owner, and the registry holds it.Check every supported GPU after the re-import race.
test_reimport_racing_the_last_release_keeps_accesschecked read/write access and exercised a buffer only on the default GPU. The final release of a pinned pool revokes access one GPU at a time, so a re-import could in principle be left without access on a non-default GPU and the test would still pass. It now asserts read/write access on every GPU the pool grants access to.Notes for reviewers
Please review the combined diff. The first commit replaced the registry with a plain dict of weak references whose removal callback took
_registry_lock. That made every drop of a registered resource wait for registrations in progress on other threads, and a thread that dropped one while holding an application lock deadlocked with a registration whose garbage-collection finalizer took the same lock. The second commit restores theWeakValueDictionaryand fixes the overwrite insideMP_registerinstead.Testing
test_reentrant_registration_of_one_uuid_picks_one_winnerdrives a deterministic regression for the overwrite through a one-shot, test-only hook inMP_register(_set_register_reentry_hook_for_testing) that registers the same UUID re-entrantly right before the outer call's insertion, where a finalizer could. With the code onmainplus the hook, it fails onassert outer is winnerfor bothPinnedMemoryResourceandDeviceMemoryResource; with the fix it passes.tests/memory_ipcpasses locally on driver 595.99.02 (single GPU): 171 passed, 46 skipped.Scope
cuda.coreonly. The registry andMP_registerhave no counterpart incuda_bindings_12/orcuda_bindings_13/, and the touched files contain noBEGIN SYNCEDgroups. No release-note entry is added: the winner-selection fix hardens a latent, essentially unobservable condition already covered in spirit by the #3510 note on concurrent imports sharing one resource, and the GPU check is test-only.Checklist