Repository navigation
Random segfaults on Python 3.12.10 during CI testing #134193
Description
Crash report
What happened?
We see segfaults randomly (not frequently enough to get more detailed info) when running integration tests for PySR. The following segfault was seen on Python 3.12.10.
Full stack trace of the segfault (click to expand)
[2413] signal 11 (1): Segmentation fault
in expression starting at none:0
_PyObject_IS_GC at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_object.h:343 [inlined]
visit_decref at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:465
dict_traverse at /home/runner/work/_temp/SourceCode/Objects/dictobject.c:3549
subtract_refs at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:491 [inlined]
deduce_unreachable at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:1116
gc_collect_main at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:1[242](https://github.com/MilesCranmer/PySR/actions/runs/15098605107/job/42436577017#step:9:243)
gc_collect_with_callback at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:1426
gc_collect_generations at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:1481 [inlined]
_Py_RunGC at /home/runner/work/_temp/SourceCode/Modules/gcmodule.c:2318
_Py_HandlePending at /home/runner/work/_temp/SourceCode/Python/ceval_gil.c:1045
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/ceval.c:836
_PyEval_EvalFrame at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_ceval.h:89 [inlined]
gen_send_ex2 at /home/runner/work/_temp/SourceCode/Objects/genobject.c:230 [inlined]
gen_iternext at /home/runner/work/_temp/SourceCode/Objects/genobject.c:603
builtin_all at /home/runner/work/_temp/SourceCode/Python/bltinmodule.c:322
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:2913
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
PyObject_Vectorcall at /home/runner/work/_temp/SourceCode/Objects/call.c:325
property_descr_set at /home/runner/work/_temp/SourceCode/Objects/descrobject.c:1678
_PyObject_GenericSetAttrWithDict at /home/runner/work/_temp/SourceCode/Objects/object.c:1569 [inlined]
PyObject_GenericSetAttr at /home/runner/work/_temp/SourceCode/Objects/object.c:1636
wrap_setattr at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:8050
wrapperdescr_raw_call at /home/runner/work/_temp/SourceCode/Objects/descrobject.c:537 [inlined]
wrapperdescr_call at /home/runner/work/_temp/SourceCode/Objects/descrobject.c:574
_PyObject_MakeTpCall at /home/runner/work/_temp/SourceCode/Objects/call.c:240
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:2715
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
vectorcall_unbound at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:2236 [inlined]
vectorcall_method at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:2267 [inlined]
slot_tp_setattro at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:8905
PyObject_SetAttr at /home/runner/work/_temp/SourceCode/Objects/object.c:1192
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:1135
_PyFunction_Vectorcall at /home/runner/work/_temp/SourceCode/Objects/call.c:419 [inlined]
_PyObject_FastCallDictTstate at /home/runner/work/_temp/SourceCode/Objects/call.c:144 [inlined]
_PyObject_Call_Prepend at /home/runner/work/_temp/SourceCode/Objects/call.c:508
slot_tp_init at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:9035
type_call at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:1679
_PyObject_MakeTpCall at /home/runner/work/_temp/SourceCode/Objects/call.c:240
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:2715
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
method_vectorcall at /home/runner/work/_temp/SourceCode/Objects/classobject.c:61
unknown function (ip: 0x7f18b800f0e4)
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:3[263](https://github.com/MilesCranmer/PySR/actions/runs/15098605107/job/42436577017#step:9:264)
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
PyObject_CallOneArg at /home/runner/work/_temp/SourceCode/Objects/call.c:401
slot_tp_repr at /home/runner/work/_temp/SourceCode/Objects/typeobject.c:8720
PyObject_Str at /home/runner/work/_temp/SourceCode/Objects/object.c:630
PyFile_WriteObject at /home/runner/work/_temp/SourceCode/Objects/fileobject.c:124
builtin_print_impl at /home/runner/work/_temp/SourceCode/Python/bltinmodule.c:2057 [inlined]
builtin_print at /home/runner/work/_temp/SourceCode/Python/clinic/bltinmodule.c.h:962
cfunction_vectorcall_FASTCALL_KEYWORDS at /home/runner/work/_temp/SourceCode/Objects/methodobject.c:438
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
PyObject_Vectorcall at /home/runner/work/_temp/SourceCode/Objects/call.c:325
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:[271](https://github.com/MilesCranmer/PySR/actions/runs/15098605107/job/42436577017#step:9:272)5
_PyObject_VectorcallTstate at /home/runner/work/_temp/SourceCode/./Include/internal/pycore_call.h:92 [inlined]
method_vectorcall at /home/runner/work/_temp/SourceCode/Objects/classobject.c:91
_PyEval_EvalFrameDefault at /home/runner/work/_temp/SourceCode/Python/bytecodes.c:3263
pyrun_file at /home/runner/work/_temp/SourceCode/Python/pythonrun.c:1674
_PyRun_SimpleFileObject at /home/runner/work/_temp/SourceCode/Python/pythonrun.c:459
_PyRun_AnyFileObject at /home/runner/work/_temp/SourceCode/Python/pythonrun.c:78
pymain_run_file_obj at /home/runner/work/_temp/SourceCode/Modules/main.c:361 [inlined]
pymain_run_file at /home/runner/work/_temp/SourceCode/Modules/main.c:380 [inlined]
pymain_run_python at /home/runner/work/_temp/SourceCode/Modules/main.c:634 [inlined]
Py_RunMain at /home/runner/work/_temp/SourceCode/Modules/main.c:714
Py_BytesMain at /home/runner/work/_temp/SourceCode/Modules/main.c:768
unknown function (ip: 0x7f18b7a2a1c9)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /opt/hostedtoolcache/Python/3.12.10/x64/bin/python (unknown line)
Allocations: 379668601 (Pool: 379663673; Big: 4928); GC: 303
/home/runner/work/_temp/b8a8bd4b-faae-4481-94e0-330e30094622.sh: line 1: 2413 Segmentation fault (core dumped) coverage run -m pysr test main,cli,startup
PySR relies on PythonCall.jl to interface with Julia backend libraries.
This stacktrace was seen from the following GitHub action: https://github.com/MilesCranmer/PySR/blob/057e3ec5a9d87c9495f7d36965ce4cbee5ef452a/.github/workflows/CI.yml#L21-L88. I have uploaded the full logs for this action to the following gist: https://gist.github.com/MilesCranmer/1df76efecf606123dae2146e461676b1 which have further details on the system and environment.
Unfortunately I don't think this will be easily reproducible as we see it randomly, and not very frequently. We thought it might be worth reporting anyway, to see if there were any ideas for where the issue is coming from.
The CI runs on 3.8, 3.10, and 3.12. I have not seen this segfault on 3.8 nor on 3.10.
CPython versions tested on:
3.12
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
N/A (GitHub action)
Activity
JelleZijlstra commentedon May 18, 2025
The segfault is here:
Others may have more insights but there's a good chance this is a bug in some extension module you're using, not in Python itself.
MilesCranmer commentedon May 18, 2025
I don't doubt it, but I don't know where. Is it from the CPython API of juliacall, numpy, sklearn, or pandas, or is it some interaction between permutations of these. I truly don't know. Any tips in getting more info out would be much appreciated. Or if you have any insight based on the stacktrace itself.
MilesCranmer commentedon May 18, 2025
Potentially related to my similar issue here: #113591. There are two differences:
EXCEPTION_ACCESS_VIOLATIONrather than a segmentation fault, andThe similarities are:
picnixz commentedon May 18, 2025
Access violations on Windows may translate into "equivalent" SIGSEV on Linux so this could be the reason.
Concerning the Julia stacktrace, it could be because of how the stack trace is actually printed, and may depend on the OS (maybe Linux is able to recover more than on Windows, or vice-versa).
MilesCranmer commentedon May 18, 2025
Thanks, that is interesting. Indeed perhaps they have a similar root cause.
One other clarification is that #113591 was observed in 2023 when PySR used pyjulia as its interface, whereas the error reported in this thread was observed in the present day, when PySR runs on juliacall. pyjulia and juliacall are entirely different packages that interface Python and Julia. They have separate codebases. But in both cases, the GC-related crash started appearing on Python 3.12.
picnixz commentedon Jun 29, 2025
3.12 won't get fixes but are 3.13+ affected as well? Another possibility may be that the bindings are incorrectly done or that some objects become corrupted. Now:
Would it be possible to isolate which test is actually the culprit? at least to know which JL functions were involved. If not, I'm sorry but I don't think we can help on our side (could be a use-after-free due double visiting but that would be hard to fix, cc @ZeroIntensity).
ZeroIntensity commentedon Jun 29, 2025
Yeah, there are a ton of ways to cause crashes during GC through C API misuse. The backtrace doesn't tell us much other than "the crash happened while calling Python", which isn't very useful.
I'd try putting this through some more debuggers. If you have a core dump available, PyStack will be really helpful for tracking this down.
MilesCranmer commentedon Jul 2, 2025
Thanks. Quick update is that on Python 3.13, it is now in the form of a Julia stack trace stemming from the Python stack trace. I can't figure out why but only the Python version increment is the trigger for it. Python 3.12 (earlier versions) are more stable, and I have never seen it on Python 3.11 and earlier. But Python 3.13 regularly triggers a segfault at import time. Almost like the GC's of each language are competing. Again, I fully understand the issue is likely in some C API misuse. Just trying to diagnose it.
Here's a MWE, using the
juliacallpackage:On Python 3.12, there is no error. But on Python 3.13, I see this:
(Click to expand)
Same Julia version; same juliacall version.
I see this with some randomness. On Linux I never see this locally; only in CI. On macOS (even locally) and windows, I see it much more frequently.
I completely understand this is likely an issue in Julia and/or one of the CPython API calls. But, just to help solve it, were there any changes from Python 3.12 to Python 3.13 that might have triggered this? Or perhaps a new optimization in 3.12 (i.e., the original issue posting) that was turned up to be more aggressive in 3.13? That might help point me in the right direction.
ZeroIntensity commentedon Jul 2, 2025
My best guess would be that something is raising on 3.13, and then the C code that has a failure path makes a
Py_DECREFto a reference it doesn't own. But again, that's total speculation. If you can't use a debugger, I guess you could just try intentionally leaking references and seeing if it works again.MilesCranmer commentedon Jul 15, 2025
Ok it's understood now. This bug was quite something.
So, the cause of this was Python adding a field to PyTypeObject in 3.13 (#114900). This change was not documented (#132331). Julia needs to manually mirror these structs internally, so this broke pointer casts and caused it to overwrite its size.
This is the relevant struct in Julia: https://github.com/JuliaPy/PythonCall.jl/blob/815c0de27a5a175411447dea250e82d52e40f7c5/src/C/consts.jl#L252
Thread on discovering this by debugging corruption of a sysimage, using rr: https://discourse.julialang.org/t/plea-for-segfault-assistance/130697/19?u=milescranmer
ZeroIntensity commentedon Jul 15, 2025
Ok, thanks. I'll post to the thread to see what we can do. In the meantime, I'll keep this open in case we need to do anything upstream e.g. to improve debugging.
vstinner commentedon Sep 9, 2025
Since this issue is a bug in
PythonCall.jl, can we close this issue?