test_json case test_highly_nested_objects_decoding fails under PGO on Clang 20

Building 3.15.0a1 on aarch64 macOS with Clang 20 in python-build-standalone failed during the PGO test run in test_highly_nested_objects_decoding:

cpython-3.15> FAIL: test_highly_nested_objects_decoding (test.test_json.test_recursion.TestCRecursion.test_highly_nested_objects_decoding)
cpython-3.15> ----------------------------------------------------------------------
cpython-3.15> Traceback (most recent call last):
cpython-3.15>   File "/private/var/folders/6p/k5sd5z7j31b31pq4lhn0l8d80000gn/T/tmp7fxaapg7/Python-3.15.0a1/Lib/test/test_json/test_recursion.py", line 77, in test_highly_nested_objects_decoding
cpython-3.15>     with self.assertRaises(RecursionError):
cpython-3.15>          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
cpython-3.15> AssertionError: RecursionError not raised

This failure is specific to macOS, and also occurs on x86-64. I haven't reproduced it outside of python-build-standalone yet. Jump to #140125 (comment) for the root cause.

Linked PRs

I cannot reproduce this issue on x86-64 Linux with clang 20. I used these commands:

./configure --enable-optimizations CC=clang LD=clang
time make
./python -m test -v test_json

My clang version: clang version 20.1.8 (Fedora 20.1.8-4.fc42).

Thanks for trying to reproduce!

Just to be clear, my failure occurred while running the profile task, e.g., here, rather than during the test run after a successful build — so I'd expect it to fail during make.

We're specifically using clang 20.1.4 over in python-build-standalone. I tried to reproduce this on HEAD (2ca3c85) on my macOS machine with LLVM clang 20.1.3 from homebrew and could not — the build succeeded. I suspect there may be an interaction with another facet of our build system.

I've opened astral-sh/python-build-standalone#822 to reproduce in our CI matrix and determine if this is specific to aarch64 macOS (I encountered this locally) or occurs on other platforms.

astral-sh/python-build-standalone#822 confirms that this is only an issue on aarch64 and x86-64 macOS.

I tried to reproduce in cpython with build flags that more closely resemble python-build-standalone's

export CFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export CPPFLAGS='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export CFLAGS_JIT='-arch arm64 -mmacosx-version-min=11.0 -Wno-nullability-completeness -Wno-expansion-to-defined -Wno-undef-prefix  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -fPIC  -Werror=unguarded-availability-new'
export LDFLAGS='-arch arm64 -mmacosx-version-min=11.0 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -LModules/_hacl -Wl,-headerpad,40'
./configure --build=aarch64-apple-darwin --host=aarch64-apple-darwin --prefix=/install --with-system-expat --with-system-libmpdec --without-ensurepip --with-tail-call-interp --enable-shared --with-mimalloc --enable-optimizations --enable-experimental-jit=yes-off ac_cv_lib_intl_textdomain=no ac_cv_func_ptsname_r=no ac_cv_func_explicit_bzero=no --with-dbmliborder=ndbm

which requires the following patch to allow LLVM 20 in the JIT (we are currently using this in 3.14 without problem)

diff --git a/Tools/jit/_llvm.py b/Tools/jit/_llvm.py
index bc3b50ffe61..c90abad38da 100644
--- a/Tools/jit/_llvm.py
+++ b/Tools/jit/_llvm.py
@@ -11,7 +11,7 @@
 import _targets
 
 
-_LLVM_VERSION = "19"
+_LLVM_VERSION = "20"
 _EXTERNALS_LLVM_TAG = "llvm-19.1.7.0"
 
 _P = typing.ParamSpec("_P")

Unfortunately, this failed with a segfault

./Programs/_freeze_module getpath ./Modules/getpath.py Python/frozen_modules/getpath.h
make[2]: *** [Python/frozen_modules/getpath.h] Segmentation fault: 11
make[2]: *** Deleting file `Python/frozen_modules/getpath.h'
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2

If I use --enable-experimental-jit=no instead, the build succeeds. Unfortunately, that means our configure flags are not the sole cause of the test_json failure. I suspect the segfault is a JIT interaction with LLVM 20. I'm not sure why I get this segfault in cpython but not in python-build-standalone. It's worth noting my clang versions do not match exactly.

However, the test_json failure cannot be caused by the JIT because it is occurring on our x86-64 macOS and free-threaded builds, which have the JIT disabled.

I cannot reproduce the segfault after running make clean, so I think we can disregard that for now.

Looking at the test case... the failure is at

cpython/Lib/test/test_json/test_recursion.py

Lines 74 to 79 in 77eade3

    
           very_deep = 200000 
        
           # test that loading highly-nested objects doesn't segfault when C 
        
           # accelerations are used. See #12017 
        
           with self.assertRaises(RecursionError): 
        
               with support.infinite_recursion(): 
        
                   self.loads('{"a":' * very_deep + '1' + '}' * very_deep)

which is checking that there are not segfaults when parsing a deep object.

The recursion limit should be set to 20,000 (+ current depth)

cpython/Lib/test/support/__init__.py

Lines 2398 to 2409 in 41712c4

    
           def infinite_recursion(max_depth=None): 
        
               if max_depth is None: 
        
                   # Pick a number large enough to cause problems 
        
                   # but not take too long for code that can handle 
        
                   # very deep recursion. 
        
                   max_depth = 20_000 
        
               elif max_depth < 3: 
        
                   raise ValueError(f"max_depth must be at least 3, got {max_depth}") 
        
               depth = get_recursion_depth() 
        
               depth = max(depth - 1, 1)  # Ignore infinite_recursion() frame. 
        
               limit = depth + max_depth 
        
               return set_recursion_limit(limit)

The object is an order of magnitude deeper, so we should be hitting the recursion limit.

Specifically, we're failing in the cjson test, so it's just the C-accelerated JSON implementation that has different behavior

cpython/Lib/test/test_json/__init__.py

Lines 23 to 29 in 872eafd

    
           @unittest.skipUnless(cjson, 'requires _json') 
        
           class CTest(unittest.TestCase): 
        
               if cjson is not None: 
        
                   json = cjson 
        
                   loads = staticmethod(cjson.loads) 
        
                   dumps = staticmethod(cjson.dumps) 
        
                   JSONDecodeError = staticmethod(cjson.JSONDecodeError)

@rokm

Applying the following patch to revert 7016044 from #139232 (cc @rokm and @markshannon) resolves the issue

diff --git a/Python/ceval.c b/Python/ceval.c
index f48f412fab8..3bf1c098003 100644
--- a/Python/ceval.c
+++ b/Python/ceval.c
@@ -448,12 +448,6 @@ hardware_stack_limits(uintptr_t *top, uintptr_t *base)
     ULONG guarantee = 0;
     SetThreadStackGuarantee(&guarantee);
     *base = (uintptr_t)low + guarantee;
-#elif defined(__APPLE__)
-    pthread_t this_thread = pthread_self();
-    void *stack_addr = pthread_get_stackaddr_np(this_thread); // top of the stack
-    size_t stack_size = pthread_get_stacksize_np(this_thread);
-    *top = (uintptr_t)stack_addr;
-    *base = ((uintptr_t)stack_addr) - stack_size;
 #else
     /// XXX musl supports HAVE_PTHRED_GETATTR_NP, but the resulting stack size
     /// (on alpine at least) is much smaller than expected and imposes undue limits

You can see the CI pass in astral-sh/python-build-standalone#823

@zanieb

@zanieb Do you perhaps have a problematic build with PGO+LTO available for testing? I imagine you would need to disable the cjson test that triggers segfault to get past the profiling stage for PGO...

But it would be interesting to see if a) such a build also causes segfault in my test case from #139231, b) if it does, what the stack depth increments were between recursions. Could be that LTO causes a problem similar to #137573, and the cost of (the last) recursion step ends up exceeding the margin.

Indeed! I've disabled this test in order to release 3.15.0a1 in python-build-standalone so uv python install 3.15.0a1 will get you what you've requested. I'm not sure if the failure will occur when run outside of the profiling step, I haven't checked yet.

edit: I tried the questionable invocation

PYTHONPATH=Lib uvx python@3.15 -m test test_json

and the tests passed

edit: I tried the questionable invocation

Can you try the example from #139231? (The recorded stack depth increments between recursion steps there are what is most interesting - i.e., whether they are suspiciously large).

Although if the test_json passes, it may be that the final PGO+LTO build is OK, and it is the intermediate (presumably just LTO-enabled) build that is problematic.

❯ uvx python@3.13 test.py | tail
Recursion level: 4990, stack pointer: 0x16F3272B0, depth: 4451.33 kB
Recursion level: 4991, stack pointer: 0x16F326F20, depth: 4452.22 kB
Recursion level: 4992, stack pointer: 0x16F326B90, depth: 4453.11 kB
Recursion level: 4993, stack pointer: 0x16F326800, depth: 4454.00 kB
Recursion level: 4994, stack pointer: 0x16F326470, depth: 4454.89 kB
Recursion level: 4995, stack pointer: 0x16F3260E0, depth: 4455.78 kB
Recursion level: 4996, stack pointer: 0x16F325D50, depth: 4456.67 kB
Recursion level: 4997, stack pointer: 0x16F3259C0, depth: 4457.56 kB
Recursion level: 4998, stack pointer: 0x16F325630, depth: 4458.45 kB
Recursion Limit ok (reached level 4999)
❯ uvx python@3.15 test.py | tail
Recursion level: 10007, stack pointer: 0x16C9D19F0, depth: 8137.52 kB
Recursion level: 10008, stack pointer: 0x16C9D16B0, depth: 8138.33 kB
Recursion level: 10009, stack pointer: 0x16C9D1370, depth: 8139.14 kB
Recursion level: 10010, stack pointer: 0x16C9D1030, depth: 8139.95 kB
Recursion level: 10011, stack pointer: 0x16C9D0CF0, depth: 8140.77 kB
Recursion level: 10012, stack pointer: 0x16C9D09B0, depth: 8141.58 kB
Recursion level: 10013, stack pointer: 0x16C9D0670, depth: 8142.39 kB
Recursion level: 10014, stack pointer: 0x16C9D0330, depth: 8143.20 kB
Recursion level: 10015, stack pointer: 0x16C9CFFF0, depth: 8144.02 kB
Recursion Limit ok (reached level 10016)

(note I compiled the stackpointer.dylib with gcc as suggested there)

It's also plausible that the instrumentation from PGO is causing the problem.

Those increments are in line what I would expect (aside from the fact that stack size is 8 MB and not 16 MB - I assume you are overriding the setting in your builds)?

It's also plausible that the instrumentation from PGO is causing the problem.

That's true; the instrumentation is likely to result in increased stack consumption per recursion. It would be interesting to see how much (i.e., if it exceeds the value of the margin). Do you have an easy way to run test.py with the intermediate (instrumented) build?

Do you have an easy way to run test.py with the intermediate (instrumented) build?

I don't unfortunately. I presume it's doable via a patch but it doesn't sound trivial.

It looks like this affects 3.14.1 as well now.

Have you tried increasing the depth from 200k to, say, 500k? Is it much slower?
If that doesn't fail, it implies that PGO is thinking that deep recursion is a hot path, and maybe turning the calls into tailcalls.

We don't need the test to raise a recursion error, just not crash. So we could remove the assertion.
The test for the recursion error is just to check that the test is hitting the stack limit.

@markshannon

@markshannon that's an interesting idea. It looks like that resolves the issue astral-sh/python-build-standalone#901

pythongh-140125: Increase recursion depth for `test_json` from 200k t…

gh-140125: Increase object recursion depth for `test_json` from 200k …

pythongh-140125: Increase object recursion depth for `test_json` from…

[3.14] gh-140125: Increase object recursion depth for `test_json` fro…

Navigation Menu

Repository navigation

`test_json` case `test_highly_nested_objects_decoding` fails under PGO on Clang 20 #140125

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	very_deep = 200000
	# test that loading highly-nested objects doesn't segfault when C
	# accelerations are used. See #12017
	with self.assertRaises(RecursionError):
	with support.infinite_recursion():
	self.loads('{"a":' * very_deep + '1' + '}' * very_deep)

	def infinite_recursion(max_depth=None):
	if max_depth is None:
	# Pick a number large enough to cause problems
	# but not take too long for code that can handle
	# very deep recursion.
	max_depth = 20_000
	elif max_depth < 3:
	raise ValueError(f"max_depth must be at least 3, got {max_depth}")
	depth = get_recursion_depth()
	depth = max(depth - 1, 1) # Ignore infinite_recursion() frame.
	limit = depth + max_depth
	return set_recursion_limit(limit)

	@unittest.skipUnless(cjson, 'requires _json')
	class CTest(unittest.TestCase):
	if cjson is not None:
	json = cjson
	loads = staticmethod(cjson.loads)
	dumps = staticmethod(cjson.dumps)
	JSONDecodeError = staticmethod(cjson.JSONDecodeError)

test_json case test_highly_nested_objects_decoding fails under PGO on Clang 20 #140125

Description

Linked PRs

Activity

vstinner commented on Oct 14, 2025

zanieb commented on Oct 14, 2025

zanieb commented on Oct 14, 2025

zanieb commented on Oct 14, 2025

zanieb commented on Oct 14, 2025

zanieb commented on Oct 15, 2025

rokm commented on Oct 18, 2025

zanieb commented on Oct 18, 2025

rokm commented on Oct 19, 2025

zanieb commented on Oct 19, 2025

rokm commented on Oct 19, 2025

zanieb commented on Dec 2, 2025

markshannon commented on Dec 3, 2025

zanieb commented on Dec 3, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`test_json` case `test_highly_nested_objects_decoding` fails under PGO on Clang 20 #140125