Repository navigation
nogil set clear causes concurrent __str__ to print as empty dict #129967
Description
Bug report
Bug description:
Hi,
We're a research group focused on testing concurrent runtimes. Our work-in-progress prototype found that the current nogil build __str__ can return "{}" (empty dict) instead of the expected "set()" when there is a concurrent clear() operation. The program below shows the wrong behavior:
import threading
import sys
def t0(b1,s,res):
b1.wait()
s.clear()
def t1(b1,s,res):
b1.wait()
res.append(s.__str__())
def Test():
s = {17, 18, 'a', 'b', 'c', 'd', 'e'}
threads=[]
barrier = threading.Barrier(2)
res = []
threads.append(threading.Thread(target= t0, args=(barrier, s,res)))
threads.append(threading.Thread(target= t1, args=(barrier, s,res)))
for i in range(0, len(threads)):
threads[i].start()
for i in range(0, len(threads)):
threads[i].join()
if res[0] == "{}":
print("found bug: " + res[0])
print("test begin...")
for i in range(0,50000):
threads = []
if i % 1000 == 0:
print(i)
for i in range(0,100):
threads.append(threading.Thread(target= Test))
for t in threads:
t.start()
for t in threads:
t.join()
print("test Done")Sample output:
test begin...
0
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
found bug: {}
This behavior can be observed quite readily. We tested it on a number of x86_64 and one ARM machine.
@flypoodles and @overlorde are part of the team, adding them so they get notified about further discussion.
CPython versions tested on:
3.14, CPython main branch
Operating systems tested on:
Linux
Activity
clearcauses concurrent__str__to print as empty dict #129968 as a duplicate of this issuecolesbury commentedon Feb 10, 2025
Thanks - this is a bit of a weird one:
setbefore callingstr()orrepr()PySequence_List, which internally tries to lock both the newly created list and the set a second time. This leads to the unlocking + relocking behavior, which allows another thread to clear the set in the meantime. @Yhg1s's recursive critical section optimization doesn't apply here because it's a two-object critical section, not a simple one object critical section.This can happen in the GIL-enabled build too, at least in Python 3.11 and earlier, where
PySequence_List()can trigger arbitrary code by the GC running.I think we should do two things:
set_repr_lock_held()more robust and check the size of the list that results fromPySequence_List.set_repr_lock_held.pythongh-129967: Fix race condition in `repr(set)`
Merge branch 'main' into pythongh-129967-set-repr-clear
gh-129967: Fix race condition in `repr(set)` (gh-129978)
[3.13] pythongh-129967: Fix race condition in `repr(set)` (pythongh-1…
[3.13] gh-129967: Fix race condition in `repr(set)` (gh-129978) (gh-1…
colesbury commentedon Feb 11, 2025
Thanks @luisggpina - this is fixed now in the main and 3.13 branches.