Repository navigation
collections.deque is not atomic during copy #144809
Description
[Edit: This bug was originally reported as "not thread safe", and updated to "not atomic"]
Bug report
Bug description:
In free-threaded mode, calling copy on collections.deque is not thread safe atomic and can raise RuntimeError: deque mutated during iteration errors.
Example code:
import sys
import threading
from collections import deque
from copy import copy
print(f"Python {sys.version}")
print(f"GIL enabled: {sys._is_gil_enabled()}")
d = deque(range(100))
stop = threading.Event()
def mutate():
i = 0
while not stop.is_set():
d.append(i)
if len(d) > 200:
d.popleft()
i += 1
def copy_loop():
while not stop.is_set():
copy(d)
barrier = threading.Barrier(5)
def run(fn):
barrier.wait()
fn()
threads = [
threading.Thread(target=run, args=(f,), daemon=True)
for f in [mutate, mutate, copy_loop, copy_loop]
]
for t in threads:
t.start()
barrier.wait()
stop.wait(timeout=5)
stop.set()
for t in threads:
t.join(timeout=5)
print("Done")Running under free-threaded build:
❯ ./python /tmp/deque.py
Python 3.15.0a6+ free-threading build (heads/main:945bf8ce1bf, Feb 13 2026, 22:04:08) [GCC 11.4.0]
GIL enabled: False
Exception in thread Thread-4 (run):
Exception in thread Thread-3 (run):
Traceback (most recent call last):
Traceback (most recent call last):
File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "cpython/Lib/threading.py", line 1017, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/deque.py", line 32, in run
fn()
~~^^
File "/tmp/deque.py", line 24, in copy_loop
copy(d)
~~~~^^^
File "cpython/Lib/copy.py", line 82, in copy
return copier(x)
RuntimeError: deque mutated during iteration
File "cpython/Lib/threading.py", line 1075, in _bootstrap_inner
self._context.run(self.run)
~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "cpython/Lib/threading.py", line 1017, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/deque.py", line 32, in run
fn()
~~^^
File "/tmp/deque.py", line 24, in copy_loop
copy(d)
~~~~^^^
File "cpython/Lib/copy.py", line 82, in copy
return copier(x)
RuntimeError: deque mutated during iteration
DoneCPython versions tested on:
CPython main branch, 3.14
Operating systems tested on:
Linux
Activity
serhiy-storchaka commentedon Feb 15, 2026
I do not think it is not thread safe. Copying does not crash Python, does not corrupt memory, and does not left a deque in an inconsistent state. It is not atomic -- but you cannot in general expect that an operation with non-constant complexity on collection is atomic.
cc @rhettinger
yilei commentedon Feb 16, 2026
Thanks for taking a look at this!
Hmm, my mental model of thread safety is that I'd expect
copy(d)would reasonably give me a copy when there are other concurrent mutations in parallel, without using e.g. a lock to guard the copy and mutations. And perhaps that's a wrong interpretation? And non-atomic is the better term. My assumption was largely based on:dictandlistsucceeds without errorsBtw, additional context for this bug report is getsentry/sentry-python#5465 where we encountered this issue.
I'm fine if we couldn't make
copy(d)work here without users adding guards around calls to deque methods (though IMO it hurts usability), it would be great if we could document this expectation somewhere.colesbury commentedon Feb 17, 2026
I think we should make this behave atomically because:
The problem is a combination of:
__copy__is called (good!)That triggers the deadlock avoidance routine, which unlocks the deque before re-locking both the iterator and the deque. The time in between the unlock and re-locking allows another thread to modify the deque (which would not be allowed in the GIL-enabled build).
Yhg1s commentedon Feb 18, 2026
Given that we already have an optimization where we skip locking (directly) nested critical sections of the same object, and we order CRITICAL_SECTION2 locks by mutex address to avoid deadlocks (so the order of it/deque locking is already opaque to the caller) -- couldn't we just skip locking whichever lock in a CRITICAL_SECTION2 is currently locked in the (previous) top-most critical section?
yilei commentedon Feb 18, 2026
I think this works, and it's a good optimization to have anyway. I'll clean up my draft code, add some tests, and send a PR.
colesbury commentedon Feb 18, 2026
That would work for the specific code in deque, but it's not safe in general because if the thread state is detached and then attached, only the (new) top-most critical section is locked. For example:
I think we could probably get something to work safely along the lines of "only lock b", but I think it'll be complicated.
pythongh-144809: Make deque copy atomic in free-threaded build
colesbury commentedon Feb 18, 2026
#144966 is a small change that I'd be comfortable backporting to 3.14.
It'd be great to add an optimization for CRITICAL_SECTION2, but I think that will be a lot more complicated.
yilei commentedon Feb 18, 2026
Makes sense, thanks @colesbury!
[-]`collections.deque` is not thread safe during copy[/-][+]`collections.deque` is not atomic during copy[/+]gh-144809: Make deque copy atomic in free-threaded build (gh-144966)
pythongh-144809: Make deque copy atomic in free-threaded build (pytho…
[3.14] gh-144809: Make deque copy atomic in free-threaded build (gh-1…
pythongh-144809: Make deque copy atomic in free-threaded build (pytho…
pythongh-144809: Make deque copy atomic in free-threaded build (pytho…