Repository navigation
Make weakref thread-safe without the GIL #111926
Description
Feature or enhancement
The current weakref implementation relies on the GIL for thread-safety.
The nogil-3.12 fork substantially modifies the weakref implementation. I think we can implement a simpler change in CPython 3.13 (main) now that all PyObject's have their own mutex (in --disable-gil builds).
Basic idea
Protect access to the weakrefs linked list using the mutex in the weakly referenced object. Use the critical section API.
Prior implementation: colesbury/nogil-3.12@0dddcb6f9d
Linked PRs
- gh-111926: Update _weakref to be threadsafe in --disable-gil build #112189
- gh-111926: Update _PyWeakref_IS_DEAD to be thread-safe #112267
- gh-111926: Set up basic sementics of weakref API for freethreading #113621
- gh-111926: Simplify weakref/proxy creation #116825
- gh-111926: Simplify weakref creation logic #116843
- gh-111926: Simplify proxy creation logic #116844
- gh-111926: Make weakrefs thread-safe in free-threaded builds #117168
- gh-111926: Avoid locking in PyType_IsSubtype #117275
Activity
corona10 commentedon Nov 10, 2023
Maybe we should check the performance impact the default build first.
I will try to investigate it.
colesbury commentedon Nov 10, 2023
The critical section APIs are no-ops on the default build:
cpython/Include/internal/pycore_critical_section.h
Lines 107 to 113 in 289af86
corona10 commentedon Nov 10, 2023
I know that there is no op for the default build but the commit you suggested modify some internal structure, I just want to know ovehead from the structure changed, even I actually do not expect real overhead from some refactoring. (I didn't look at deeply now please let me know if I miss something)
colesbury commentedon Nov 10, 2023
@corona10 the commit I linked to (in nogil-3.12) is not how I intend to make weakref thread-safe in CPython 3.13. I wouldn't spend too much time investigating it because it's not going to make it's way into 3.13. I linked to it because it's sometimes helpful for me to look back at the nogil-3.12 implementation.
corona10 commentedon Nov 16, 2023
By the way, I am working on it :)
pythongh-111926 Update _weakref to be threadsafe in --disable-gil build
pythongh-111926 Update _weakref to be threadsafe in --disable-gil build
gh-111926: Update _weakref to be threadsafe in --disable-gil build (g…
pythongh-111926 Update _PyWeakref_IS_DEAD to be thread-safe
gh-111926: Update _PyWeakref_IS_DEAD to be thread-safe (gh-112267)
pythongh-111926: Set up basic sementics of weakref API for freethreading
34 remaining items
Squashed commit of the following:
Squashed commit of the following:
Merge branch 'pythongh-111926-thread-safe-weakref' into nogil-integra…
Merge branch 'pythongh-111926-thread-safe-weakref' into nogil-integra…
gh-111926: Make weakrefs thread-safe in free-threaded builds (#117168)
pythongh-111926: Simplify weakref creation logic (python#116843)
pythongh-111926: Simplify proxy creation logic (python#116844)
pythongh-111926: Avoid locking in PyType_IsSubtype (python#117275)
pythongh-111926: Make weakrefs thread-safe in free-threaded builds (p…
Don't raise an error if tuple allocation fails when clearing weakrefs
Don't raise an error if tuple allocation fails when clearing weakrefs
pythongh-111926: Update _weakref to be threadsafe in --disable-gil bu…
pythongh-111926: Update _PyWeakref_IS_DEAD to be thread-safe (pythong…
pythongh-111926: Set up basic sementics of weakref API for freethread…