Repository navigation
Free-threaded CPython 3.14: concurrent functools.partial.__setstate__ crashes with repr(), calls, or another __setstate__ #157841
Description
Prior report
gh-157099 (2026-09-07) reported the same missing synchronization in partial_setstate, reaching it through partial_new (a crash in tuple_concat under ASan, 3.14.7 @ 823f032). It was closed by its reporter after a request to consolidate a batch of reports (gh-157124), and the consolidated issues gh-157196 / gh-157197 / gh-157198 were then withdrawn by the reporter on 2026-09-11 to be resubmitted under the security policy. As far as I can see nothing tracks it now, and main @ 89c67a9 still crashes. This report was found independently (static analysis of partial_repr on 3.14.0rc1, then runtime confirmation) and adds a sanitizer-free 25-line reproducer covering repr(), calls and writer/writer, with a GIL control on the same binary. If you would rather reopen gh-157099, please close this one as a duplicate. Filed on the public tracker because triggering it requires running Python code in the process (the assessment given on gh-157196).
What happened?
On the free-threaded build, calling partial.__setstate__ on a functools.partial object while another thread uses the same object (repr(p) or p()), or while a second thread also calls __setstate__, crashes the interpreter within seconds. The read-only control completed all five four-second runs without a crash. The same binary with PYTHON_GIL=1 completed every run without a crash.
Standalone reproducer (repro_partial.py <mode>):
import functools, itertools, os, sys, threading
if os.environ.get("ALLOW_GIL") != "1":
assert not sys._is_gil_enabled()
mode = sys.argv[1]
counter = itertools.count(10**12) # fresh heap ints, never cached
def fresh(): return next(counter)
def f(*a, **k): return None
p = functools.partial(f, fresh(), fresh())
stop = threading.Event()
def reader_repr(): repr(p)
def reader_call(): p()
def writer(): p.__setstate__((f, (fresh(), fresh()), {}, None))
reader, nr, nw = {"ro": (reader_repr, 4, 0), "rw": (reader_repr, 4, 1),
"ww": (None, 0, 2), "cw": (reader_call, 4, 1)}[mode]
def loop(fn):
while not stop.is_set():
fn()
ts = [threading.Thread(target=loop, args=(reader,)) for _ in range(nr)] + \
[threading.Thread(target=loop, args=(writer,)) for _ in range(nw)]
for t in ts: t.start()
stop.wait(4); stop.set()
for t in ts: t.join()
print("ok")Results, 5 runs × 4 s per mode, run with -X faulthandler ("hang" = the process had not exited 30 s after start and was killed):
| mode | threads | 3.14.6t | 3.14.6t, PYTHONMALLOC=debug |
same binary, PYTHON_GIL=1 |
main @ 89c67a9 (--disable-gil) |
main, PYTHONMALLOC=debug |
|---|---|---|---|---|---|---|
ro |
4 × repr(p) |
5/5 ok | 5/5 ok | — | 5/5 ok | 5/5 ok |
rw |
4 × repr(p) + 1 × __setstate__ |
0/5 (SIGILL, SIGABRT, 3 hangs) | 0/5 (SIGSEGV ×5) | 3/3 ok | 0/5 (SIGABRT ×2, SIGILL, 2 hangs) | 0/5 (SIGSEGV ×5) |
ww |
2 × __setstate__ |
4/5 (SIGSEGV ×1) | 1/5 (SIGSEGV ×4) | 3/3 ok | 2/5 (SIGSEGV ×3) | 4/5 (SIGSEGV ×1) |
cw |
4 × p() + 1 × __setstate__ |
0/5 (SIGILL ×2, SIGTRAP ×2, SIGSEGV) | 0/5 (SIGSEGV ×5) | 3/3 ok | 0/5 (SIGSEGV ×3, SIGILL, 1 hang) | 0/5 (SIGSEGV ×5) |
ww is intermittent: over several runs of this script today it crashed in 1 to 4 of 5 runs per configuration. rw and cw failed in every run of every free-threaded configuration. The SIGABRT cases are
Fatal Python error: PyMutex_Unlock: unlocking mutex that is not locked
raised from the repr thread (faulthandler shows reader_repr). The repr/__setstate__ shape also crashes on 3.14.0rc1 free-threaded (5/5, PYTHONMALLOC=debug). Note that main already carries the partial_vectorcall snapshot from gh-154189, and cw still crashes there.
Analysis
partial_setstate replaces pto->fn, pto->args and pto->kw with Py_SETREF and holds no critical section on the partial object (3.13 branch Modules/_functoolsmodule.c:514; 3.14 branch :762; main :857). The readers take strong references from the fields — partial_repr since gh-144475 (3.14 branch :630), partial_vectorcall since gh-154189 (main only; not in 3.14). The source suggests an unsynchronized reference-acquisition gap: Py_NewRef(pto->args) does not protect the interval between loading the field and acquiring a strong reference while another thread replaces it. The observed crashes have not yet been attributed to a particular native access. Two concurrent __setstate__ writers also crash without a reader thread; the failing access in that case has not yet been identified. gh-144475 and gh-154189 addressed the re-entrant, same-thread shape of this; gh-157099 and this report are the cross-thread shape.
Expected behaviour
I expect concurrent use to avoid interpreter crashes; no particular ordering or consistent result is required by this reproducer. The free-threading HOWTO says the build "aims to provide similar thread-safety behavior at the Python level to the default GIL-enabled build" while noting that this "should be treated as a description of the current implementation, not a guarantee"; the same binary with the GIL enabled runs the reproducer without a crash.
Possible approach
A possible approach is to synchronize state publication and acquisition of coherent reader snapshots on the partial object, including associated placeholder and vectorcall state; the complete access and reentrancy audit remains to be done. Happy to test a patch.
CPython versions tested on
3.14.6 (free-threaded, python-build-standalone via uv), 3.14.0rc1 (free-threaded), CPython main branch @ 89c67a9 (2026-09-20, local --disable-gil build; Modules/_functoolsmodule.c identical to upstream at that commit)
Operating systems tested on
macOS (arm64, Darwin 25.6)
Output from running 'python -VV' on the command line
Python 3.14.6 free-threading build (main, Jul 18 2026, 17:07:52) [Clang 22.1.3 ]
Python 3.16.0a0 free-threading build (remotes/origin/HEAD-dirty:89c67a9, Sep 20 2026, 14:12:46) [Clang 21.0.0 (clang-2100.3.34.2)]
Related
gh-157099 (same root cause, closed not_planned), gh-144475, gh-154189, gh-140590
Activity
Adarsh-Me commentedon Sep 21, 2026
I'd like to work on this, subject to one question.
The analysis looks right:
partial_setstatemutatesp->args/p->keywords/p->funcwithout holding the per-object mutex that guards the readers inpartial_reprandpartial_call, so the fix is to takepartial->lockinpartial_setstate(Modules/_functoolsmodule.c) with the same ordering the readers already use, and to make the swap atomic under free-threading rather than reassigning the fields in place.Two things before I write it: (1) the report notes gh-157196 was withdrawn to be resubmitted under the security policy — is this issue intended to be fixed in the open, or does it belong on the security tracker? (2)
partial_newmay need the same treatment if__setstate__can race with a fresh construction. Happy to take whichever path you prefer, with a free-threaded regression test and aMisc/NEWS.dentry.espressolee commentedon Sep 21, 2026
Thanks for picking this up — and yes, please do.
(1) Tracker. I filed this publicly on the assessment quoted in the report: triggering it requires already running Python code in the process, which was the assessment given on gh-157196. gh-157099, gh-157124 and gh-157196 are all closed as
not plannedby their author, and as far as I can see from the public trackers nothing follows them — but I can't see any non-public tracker, so whether this should move to the security process is for the maintainers to say, not me.(2) One correction to the premise, and it widens the patch. You wrote that
partial_setstatemutates the fields "without holding the per-object mutex that guards the readers inpartial_reprandpartial_call". In the code I audited at89c67a9, those readers do not take a critical section on the partial either — so there is no existing reader-side ordering for the writer to join:89c67a9partial_setstate:801fn,args,kw,phcount,dictat:856-860, thenpartial_setvectorcallat:861partial_repr:705Py_NewRef(pto->fn/args/kw)at:724-726. ThePy_BEGIN_CRITICAL_SECTIONat:745is onkw, entered after that reference is acquired, and protects the dict iterationpartial_vectorcall:369pto->phcountat:377, thenPy_NewRef(pto->fn/args/kw)at:386-388partial_call:592pto->phcountat:600andPyDict_GET_SIZE(pto->kw)at:610without taking a reference firstI read your proposal as two parts, and I think the second is the load-bearing one. Making the swap atomic in the writer is necessary, but on its own it does not give a reader a coherent, live state: with no shared protocol, a reader still loads each field independently. So whatever the writer does,
partial_repr/partial_vectorcall/partial_callneed a matching acquisition path. Worth stating explicitly in the PR how readers get their snapshot.Two invariants that I think the fix has to establish, since they are separate:
Py_NewRef(pto->args)loads the field and then increments. A concurrent replacement can drop the last reference in between, so the acquisition itself needs to be safe, not just the use afterwards.partial_vectorcallsamplesphcountat:377andargsat:387and combines them at:419(tot_nargs = pto_nargs + nargs - pto_phcount). Nothing currently makes those two come from the same state. Thenargs < pto_phcountguard at:378does keeptot_nargs >= pto_nargs, so a stale count does not shrink that total below the sampled tuple — but it does not make the pair consistent either, and protecting object lifetimes alone would not fix it.I have not attributed the observed crashes to a particular native access, so please treat the second point as an invariant that needs to hold rather than as my diagnosis of the signal you'll see.
Two directions I can see, without a view on which suits this object — that's a call for the core devs:
selfin the writer and in each reader, acquiring the strong references and copyingphcountout of one coherent state, then doing the formatting or the call after leaving it; orphcountcannot be sampled apart fromargs.Either way, two things are easy to miss: the displaced values' decrefs can run arbitrary code, so the new state should be fully installed before they run; and the vectorcall function pointer is part of the state too, not just the four fields.
On
partial_new: gh-157099 reached the same crash through it, so it belongs in the audit. My reproducer doesn't exercise that shape, so I'd rather see it covered by the access audit than guessed at.What I can offer. Both reader/writer modes —
repr+__setstate__, and call +__setstate__— crashed in all five runs of every free-threaded configuration I tried, while the writer/writer mode was intermittent (one to four crashes in five) and the read-only control never crashed. I haven't tested a writer-only patch, so I can't tell you from measurement what it would leave behind. Point me at a branch and I'll run all four modes, 5 runs × 4 s each, with and withoutPYTHONMALLOC=debug, on 3.14.6t and on a local--disable-gilbuild ofmain, plus thePYTHON_GIL=1control on the same binary.Adarsh-Me commentedon Sep 23, 2026
Opened as #157979.
@espressolee your correction of my premise was right and it changed the shape of the patch, so thank you for it. I had written "without holding the per-object mutex that guards the readers in
partial_reprandpartial_call", and as you pointed out there is no reader-side ordering there to join — so a writer-only atomic swap would have been the wrong fix. The PR takes the first of the two directions you sketched (a critical section onselfin the writer and in each reader) because it is the one that reuses the protocol the object already has:phcountandargsare copied out of one state and the formatting or the call happens after leaving the section, sotot_nargscannot mix two states;partial_new()'s flatten path is treated as a reader too, which is thegh-157099shape you asked to be covered rather than guessed at.On measurement: your offer is exactly what this needs, because everything I have is one platform. The build I have is a Windows
amd64tRelease (MSC v.1929,3.16.0a0). Againstmain's_functoolsmodule.cwith the new tests present,test_concurrent_setstate_and_readsdies withFatal Python error: PyMutex_Unlock: unlocking mutex that is not locked; against the PR branch all 7 tests in that file pass. The tests do not only watch for a crash — every state they install carries the same value twice, once positionally and once as a keyword, so a torn read fails an assertion. If you run your four modes against#157979's head, that would tell us whether the reader-side behaviour you measured (both reader/writer modes crashing in 5/5 runs) actually goes away, which is the claim I cannot make from a Windows box.On the tracker question, agreed that it is a call for the core devs rather than either of us — raising it explicitly in #157979 so whoever picks it up sees it: this is triggerable from already-running Python code, which is the assessment
gh-157196was closed under, but the earlier closures were by the author and there may be a non-public tracker nobody here can see.