Repository navigation
Crash in PyGILState_Ensure during _Py_Finalize #142135
Description
Crash report
What happened?
I have CPython embedded in another process, which is then loading the equivalent of this python:
import onxruntime as ort
import numpy as np
...
state['session' ] = ort.InferenceSession(...)
state['session'].run(...)
Then, later I shutdown the process, which drops all the references from C (including the state object), and call Py_Finalize. Which dumps core.
This has given me this stack trace:
#12 0x00007f9cece94d23 in _Py_FatalErrorFunc (func=func@entry=0x7f9cecfa4730 <__func__.5> "_PyThreadState_Attach", msg=msg@entry=0x7f9cecfa4610 "non-NULL old thread state") at Python/pylifecycle.c:3391
#13 0x00007f9cece98451 in _PyThreadState_Attach (tstate=0x7f9ce257a480) at Python/pystate.c:2084
#14 _PyThreadState_Attach (tstate=0x7f9ce257a480) at Python/pystate.c:2074
#15 0x00007f9cece6b93a in PyEval_RestoreThread (tstate=<optimized out>) at Python/ceval_gil.c:659
#16 0x00007f9cece99819 in PyGILState_Ensure () at Python/pystate.c:2785
#17 0x00007f9ceceb4793 in PyTraceMalloc_Untrack (domain=389047, ptr=140311524497640) at Python/tracemalloc.c:1354
#18 0x00007f9cc874a0fe in PyDataMem_UserFREE () from /work/mjj29/c8y-analytics-blocks/blocks/ONNX/venv/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so
#19 0x00007f9cc874ad0a in array_dealloc () from /work/mjj29/c8y-analytics-blocks/blocks/ONNX/venv/lib/python3.13/site-packages/numpy/_core/_multiarray_umath.cpython-313-x86_64-linux-gnu.so
#20 0x00007f9cecd3a4e8 in Py_DECREF (op=<optimized out>) at ./Include/object.h:949
#21 Py_XDECREF (op=<optimized out>) at ./Include/object.h:1042
#22 dict_dealloc (self=0x7f9cc6dc3d00) at Objects/dictobject.c:3214
#23 dict_dealloc (self=0x7f9cc6dc3d00) at Objects/dictobject.c:3195
#24 0x00007f9cecd42878 in Py_DECREF (op=<optimized out>) at ./Include/object.h:949
#25 Py_XDECREF (op=<optimized out>) at ./Include/object.h:1042
#26 _PyObject_SetManagedDict (obj=obj@entry=0x7f9cc67c2a50, new_dict=new_dict@entry=0x0) at Objects/dictobject.c:7198
#27 0x00007f9cecd428fb in PyObject_ClearManagedDict (obj=obj@entry=0x7f9cc67c2a50) at Objects/dictobject.c:7222
#28 0x00007f9cecd8524d in subtype_dealloc (self=0x7f9cc67c2a50) at Objects/typeobject.c:2359
#29 0x00007f9cecd3a927 in Py_DECREF (op=<optimized out>) at ./Include/object.h:949
#30 Py_XDECREF (op=<optimized out>) at ./Include/object.h:1042
#31 dictkeys_decref (use_qsbr=false, interp=<optimized out>, dk=0x7f9c26b4d470) at Objects/dictobject.c:504
#32 dictkeys_decref (use_qsbr=false, dk=0x7f9c26b4d470, interp=<optimized out>) at Objects/dictobject.c:481
#33 dict_dealloc (self=0x7f9cc6df0400) at Objects/dictobject.c:3222
#34 dict_dealloc (self=0x7f9cc6df0400) at Objects/dictobject.c:3195
#35 0x00007f9cecd3cc98 in Py_DECREF (op=0x7f9cc6df0400) at ./Include/object.h:949
#36 Py_XDECREF (op=0x7f9cc6df0400) at ./Include/object.h:1042
#37 insertdict (interp=<optimized out>, mp=0x7f9cc6dc36c0, key=0x7f9cc6ddae70, hash=<optimized out>, value=0x7f9ced10c7c0 <_Py_NoneStruct>) at Objects/dictobject.c:1885
#38 0x00007f9cecd3f201 in setitem_take2_lock_held (value=<optimized out>, key=<optimized out>, mp=<optimized out>) at Objects/dictobject.c:2549
#39 0x00007f9cecd3f294 in PyDict_SetItem (op=op@entry=0x7f9cc6dc36c0, key=<optimized out>, value=value@entry=0x7f9ced10c7c0 <_Py_NoneStruct>) at ./Include/object.h:1059
#40 0x00007f9cecd4fc65 in _PyModule_ClearDict (d=0x7f9cc6dc36c0) at Objects/moduleobject.c:701
#41 0x00007f9cecd4ff5e in _PyModule_Clear (m=m@entry=0x7f9cc6ddca90) at Objects/moduleobject.c:670
#42 0x00007f9cece913c7 in finalize_modules_clear_weaklist (verbose=0, weaklist=0x7f9cecb95d40, interp=0x7f9ced146d50 <_PyRuntime+88400>) at Python/pylifecycle.c:1682
#43 finalize_modules (tstate=tstate@entry=0x7f9ced176590 <_PyRuntime+283024>) at Python/pylifecycle.c:1778
#44 0x00007f9cece94f59 in _Py_Finalize (runtime=0x7f9ced131400 <_PyRuntime>) at Python/pylifecycle.c:2134
I've tried a bunch of things (like forcing some GC before finalize etc), but it still crashes.
I thought this would be the same as #129185 since it seems almost exactly the same, but I'm trying this with 3.13.9, which allegedly has this fixed, and it's still crashing.
I don't have a small standalone repro just yet, but you could in principle reproduce it using this test: https://github.com/mjj29/c8y-analytics-blocks/tree/aiblock/tests/ONNXBlock on this branch, using the latest docker image from https://gallery.ecr.aws/apama/apama-builder.
This is using 3.13.3 without the fix, but I have the same repro using a local build with the latest 3.13.9.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.13.9 (main, Dec 1 2025, 11:59:57) [GCC 12.2.0]
Activity
ZeroIntensity commentedon Dec 1, 2025
This is a duplicate of #124619. The ideal solution here is PEP-788.
matj-sag commentedon Dec 1, 2025
OK, that's nice, but is there anything I can do right now to avoid this issue?
ZeroIntensity commentedon Dec 1, 2025
You can replicate PEP 788 using a countdown in an
atexitcallback. Some pseudocode:matj-sag commentedon Dec 1, 2025
Hmm, I don't believe I'm doing any calling into python myself during finalization, I've finishad all other C threads by that point. The stacktrace has finalize calling into numpy which then calls ensure. None of this is my code in the server