Regression in Python 3.14.0a7.

Crash report

What happened?

I noticed a regression in Python 3.14.0 alpha 7. One of tests from Django test suite crashes test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test:

$ ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
Fatal Python error: Segmentation fault

Current thread 0x00007858cbcc2b80 [python] (most recent call first):
  File "/django/django/test/runner.py", line 185 in _confirm_picklable
  File "/django/django/test/runner.py", line 215 in check_picklable
  File "/django/django/test/runner.py", line 294 in addError
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 246 in _addClassOrModuleLevelException
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 236 in _createClassOrModuleLevelException
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 176 in _handleClassSetUp
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 114 in run
  File "/django/tests/test_runner/test_parallel.py", line 225 in test_handle_add_error_before_first_test
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 615 in _callTestMethod
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 669 in run
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 725 in __call__
  File "/django/django/test/testcases.py", line 357 in _setup_and_call
  File "/django/django/test/testcases.py", line 302 in __call__
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 122 in run
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 84 in __call__
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/runner.py", line 259 in run
  File "/django/django/test/runner.py", line 1020 in run_suite
  File "/django/django/test/runner.py", line 1093 in run_tests
  File "/django/tests/./runtests.py", line 427 in django_tests
  File "/django/tests/./runtests.py", line 788 in <module>

Current thread's C stack trace (most recent call first):
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _Py_DumpStack+0x45 [0x7858cb8e3145]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2f85d4 [0x7858cb8f85d4]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x42520 [0x7858cb242520]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyObject_GC_UnTrack+0x19 [0x7858cb887269]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2e08a3 [0x7858cb8e08a3]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10cb15 [0x7858cb70cb15]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10d235 [0x7858cb70d235]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x18e862 [0x7858cb78e862]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x8ec9 [0x7858cb699059]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyEval_EvalCode+0x1dd [0x7858cb854ced]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c57ad [0x7858cb8c57ad]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7996 [0x7858cb8c7996]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7f3f [0x7858cb8c7f3f]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at Py_RunMain+0x996 [0x7858cb8f64e6]
  <truncated rest of calls>

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Segmentation fault (core dumped)

It also crashes on the current main branch of cpython (reproduced at 1550c30 compiled with debug):

$ runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
./Include/refcount.h:526: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6598b3930 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Traceback (most recent call last):
  File "/cpython/Lib/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
    ~~~~~~~~~~^^
  File "/django/tests/test_runner/test_parallel.py", line 74, in setUpClass
    raise ValueError("woops")
ValueError: woops

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/django/tests/.test-python-dev/lib/python3.14/site-packages/tblib/pickling_support.py", line 30, in unpickle_exception
    inst.__traceback__ = tb
    ^^^^^^^^^^^^^^^^^^
TypeError: __traceback__ must be a traceback or None

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Aborted (core dumped)

I bisected this regression to the 053c285. Previously it passed with an assertion warning

./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/workspace/repo/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
Objects/typeobject.c:6226: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7cd1aa2aa1c0
object refcount : 4294967295
object type     : 0x63d9c0393140
object type name: type
object repr     : <class 'bserobj_tuple'>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x000063d9c0404558)

Current thread 0x00007cd1ab58b740 (most recent call first):
  <no Python frame>
Aborted (core dumped)

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a7+ (heads/main:6a9bfee4fa1, Apr 25 2025, 10:31:59) [GCC 11.4.0]

Can you please link the test code?

Can you please link the test code?

Added link to the issue description.

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

I uninstalled both packages (pywatchman and markupsafe) but the crash still occurs.

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

Will try 🕵

@mpage

cc @mpage

Will take a look on monday

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

You're right, thanks 👍 It still reports Objects/typeobject.c:6216: _Py_NegativeRefcount: Assertion failed: object has negative ref count, but no longer crashes.

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

(django_a6) > ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/data/users/mpage/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
(django_a6) > python -VV
Python 3.14.0a6 (tags/v3.14.0a6:77b2c933cab, May  5 2025, 14:12:02) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)]
(django_a6) >

Are you able to bisect the assertion failure?

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

Maybe because I configure python with --with-pydebug 🤔

./configure --with-pydebug && make -j

Are you able to bisect the assertion failure?

Should be doable, give me 1-2 days.

Maybe because I configure python with --with-pydebug

Hrm, I'm also compiling with --with-pydebug...

I compiled Python from 3a64844 (main ~ 3.15) and 03457ca (3.14 branch) and I can't replicate either.

I ran tests in a Python venv after installing django/django@6aa05fd

Ok, sounds like everything is done here.

Navigation Menu

Repository navigation

Regression in Python 3.14.0a7. #133352

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Regression in Python 3.14.0a7. #133352

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Activity

StanFromIreland commented on May 3, 2025

felixxm commented on May 3, 2025

ZeroIntensity commented on May 3, 2025

felixxm commented on May 3, 2025

sobolevn commented on May 3, 2025

mpage commented on May 4, 2025

mpage commented on May 5, 2025

felixxm commented on May 5, 2025

mpage commented on May 5, 2025

felixxm commented on May 5, 2025

mpage commented on May 5, 2025

ilovelinux commented on Jul 19, 2025

ZeroIntensity commented on Jul 19, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions