Repository navigation
Segmentation fault during the test__xxsubinterpreters test #100981
Description
Crash report
When running the testsuite one test is intermittently failing with a segmentation fault. The test in question is test=test__xxsubinterpreters and the crash seems to be GIL related. I am able to consistently reproduce the crash on my environment where I am building Python inside a chroot (/proc, /sys, and /dev are "mount --bind"ed from the host system).
On a freshly built with GCC 12.2.0 Python 3.11 with the following configure options:
CFLAGS='-pipe -O2 -ggdb -fomit-frame-pointer'
export CFLAGS
CXXFLAGS='-pipe -ggdb -O0'
export CXXFLAGS
FFLAGS='-pipe -ggdb -O0 '
export FFLAGS
FCFLAGS='-pipe -ggdb -O0 '
export FCFLAGS
LDFLAGS=
export LDFLAGS
/home/galaxy/rpm-work/BUILD/Python-3.11.1/configure \
--enable-ipv6 \
--with-computed-gotos \
--with-dbmliborder=gdbm:ndbm:bdb \
--with-system-expat \
--without-system-ffi \
--without-system-libmpdec \
--enable-loadable-sqlite-extensions \
--without-dtrace \
--with-lto \
--without-ensurepip \
--with-pkg-config=yes \
--without-static-libpython \
--with-tzpath=/usr/share/zoneinfo \
--with-openssl-rpath=no \
--with-ssl-default-suites=openssl \
--disable-optimizations \
--with-pydebug
This is the testsuite output with just the test in question:
galaxy@apollo:~/rpm-work/BUILD/Python-3.11.1 $ LD_LIBRARY_PATH=/home/galaxy/rpm-work/BUILD/Python-3.11.1/build/debug build/debug/python -m test -j4 --slowest --timeout=1800 -W -F test__xxsubinterpreters
0:00:00 load avg: 0.05 Run tests in parallel using 4 child processes (timeout: 30 min, worker timeout: 35 min)
0:00:08 load avg: 0.65 [ 1/1] test__xxsubinterpreters crashed (Exit code -11)
Fatal Python error: Segmentation fault
Thread 0x00007effcf9a2468 (most recent call first):
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/test__xxsubinterpreters.py", line 283 in clean_up_interpreters
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/test__xxsubinterpreters.py", line 299 in tearDown
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/case.py", line 584 in _callTearDown
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/case.py", line 626 in run
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/case.py", line 678 in __call__
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 122 in run
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 84 in __call__
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 122 in run
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 84 in __call__
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 122 in run
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/suite.py", line 84 in __call__
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/unittest/runner.py", line 217 in run
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/support/__init__.py", line 1095 in _run_suite
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/support/__init__.py", line 1221 in run_unittest
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest.py", line 276 in _test_module
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest.py", line 312 in _runtest_inner2
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest.py", line 355 in _runtest_inner
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest.py", line 214 in _runtest
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest.py", line 260 in runtest
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/runtest_mp.py", line 90 in run_tests_worker
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/main.py", line 722 in _main
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/main.py", line 701 in main
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/libregrtest/main.py", line 763 in main
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/regrtest.py", line 43 in _main
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/test/regrtest.py", line 47 in <module>
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/runpy.py", line 88 in _run_code
File "/home/galaxy/rpm-work/BUILD/Python-3.11.1/Lib/runpy.py", line 198 in _run_module_as_main
Extension modules: _testcapi, _xxsubinterpreters (total: 2)
Kill <TestWorkerProcess #2 running test=test__xxsubinterpreters pid=1282561 time=8.8 sec> process group
Kill <TestWorkerProcess #3 running test=test__xxsubinterpreters pid=1282560 time=8.8 sec> process group
Kill <TestWorkerProcess #4 running test=test__xxsubinterpreters pid=1282564 time=8.8 sec> process group
== Tests result: FAILURE ==
10 slowest tests:
1 test failed:
test__xxsubinterpreters
Total duration: 9.0 sec
Tests result: FAILURE
galaxy@apollo:~/rpm-work/BUILD/Python-3.11.1 $
Error messages
I acquired a core dump and under gdb it looks as follows:
Reading symbols from python...
[New LWP 1282606]
[New LWP 1282563]
[New LWP 1282568]
Core was generated by `/home/galaxy/rpm-work/BUILD/Python-3.11.1/build/debug/python -u -m test.regrtes'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007effcf93b6a5 in __syscall4 (n=14, a1=2, a2=139637136696304, a3=0, a4=8)
at ./arch/x86_64/syscall_arch.h:38
38 __asm__ __volatile__ ("syscall" : "=a"(ret) : "a"(n), "D"(a1), "S"(a2),
[Current thread is 1 (LWP 1282606)]
(gdb) info threads
Id Target Id Frame
* 1 LWP 1282606 0x00007effcf93b6a5 in __syscall4 (n=14, a1=2, a2=139637136696304, a3=0, a4=8)
at ./arch/x86_64/syscall_arch.h:38
2 LWP 1282563 0x00007effcf545221 in validate_list (head=0x7effce211d10,
flags=collecting_clear_unreachable_clear)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Modules/gcmodule.c:397
3 LWP 1282568 __cp_end () at src/thread/x86_64/syscall_cp.s:29
(gdb) bt
#0 0x00007effcf93b6a5 in __syscall4 (n=14, a1=2, a2=139637136696304, a3=0, a4=8)
at ./arch/x86_64/syscall_arch.h:38
#1 0x00007effcf93b74c in __restore_sigs (set=0x7effcdee23f0) at src/signal/block.c:43
#2 0x00007effcf93ba59 in raise (sig=11) at src/signal/raise.c:11
#3 0x00007effcf54a6fd in faulthandler_fatal_error (signum=11)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Modules/faulthandler.c:385
#4 <signal handler called>
#5 0x00007effcf47e945 in drop_gil (ceval=0x7effcf855060 <_PyRuntime+352>, ceval2=0x7effce07a1c0,
tstate=0x7effce3ddaa0) at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Python/ceval_gil.h:169
#6 0x00007effcf47f3a3 in _PyEval_ReleaseLock (tstate=0x7effce3ddaa0)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Python/ceval.c:448
#7 0x00007effcf50d4ef in _PyThreadState_DeleteCurrent (tstate=0x7effce3ddaa0)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Python/pystate.c:1126
#8 0x00007effcf5c95df in thread_run (boot_raw=0x7effcdf8e750)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Modules/_threadmodule.c:1098
#9 0x00007effcf52cee7 in pythread_wrapper (arg=0x7effce3dd380)
at /home/galaxy/rpm-work/BUILD/Python-3.11.1/Python/thread_pthread.h:241
#10 0x00007effcf957277 in start (p=0x7effcdee2b00) at src/thread/pthread_create.c:207
#11 0x00007effcf95bffe in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC
(gdb)
Your environment
- CPython versions tested on: 3.11.1 (3.10 works and I cannot trigger the crash doing the exactly same steps)
- Operating system and architecture: ALT Linux x86_64 (kernel 5.15.50, glibc 2.35.0.6) -> chroot with musl 1.2.3
- The host machine is Intel(R) Core(TM) i3 CPU 530 @ 2.93GHz with 8GB of RAM
Activity
galaxy4public commentedon Jan 12, 2023
BTW, I have to specify "-F" to run the test indefinitely otherwise it's really hard to catch with a single invocation of the test. Also, "-j4" seems to be accelerate the crash, since with "-j0" it runs quite a few successful passes before crashing:
I also noticed that load average is climbing up despite that noting else is occupying the machine (OK, this is expected, it seems, the number should be close to the number of children).
If I run the test with "-j1" (just one child process) then either I am not patient enough or it seems that it does not crash in that scenario:
AlexWaygood commentedon Jan 12, 2023
Cc. @ericsnowcurrently
Fidget-Spinner commentedon Jan 13, 2023
Seems similar to or at least related to #100711.
ericsnowcurrently commentedon Jan 30, 2023
@galaxy4public, do you still get the failure with gh-101431 applied?
galaxy4public commentedon Feb 27, 2023
@ericsnowcurrently, unfortunately, I don't have enough knowledge for backporting the mentioned commit to the latest released version of Python (which is 3.11.2) and I cannot proceed with the bleeding edge, so it seems I cannot test the fix :(.
ericsnowcurrently commentedon Jun 5, 2023
Can you verify if this is fixed now? This may have been a case of gh-104341.
picnixz commentedon Dec 2, 2024
Closing as
completedsince it appears it has been fixed. Please open a separate issue if the problem persists.