test_super_deep() for test_call triggers a stack overflow protection under a debug build for WASI

Tested using WASI-SDK 20 and wasmtime 14.

Linked PRs

Lines 992 to 1026 in 804575b

    
           @skip_on_s390x 
        
           def test_super_deep(self): 
        
               def recurse(n): 
        
                   if n: 
        
                       recurse(n-1) 
        
               def py_recurse(n, m): 
        
                   if n: 
        
                       py_recurse(n-1, m) 
        
                   else: 
        
                       c_py_recurse(m-1) 
        
               def c_recurse(n): 
        
                   if n: 
        
                       _testcapi.pyobject_vectorcall(c_recurse, (n-1,), ()) 
        
               def c_py_recurse(m): 
        
                   if m: 
        
                       _testcapi.pyobject_vectorcall(py_recurse, (1000, m), ()) 
        
               depth = sys.getrecursionlimit() 
        
               sys.setrecursionlimit(100_000) 
        
               try: 
        
                   recurse(90_000) 
        
                   with self.assertRaises(RecursionError): 
        
                       recurse(101_000) 
        
                   c_recurse(100) 
        
                   with self.assertRaises(RecursionError): 
        
                       c_recurse(90_000) 
        
                   c_py_recurse(90) 
        
                   with self.assertRaises(RecursionError): 
        
                       c_py_recurse(100_000) 
        
               finally: 
        
                   sys.setrecursionlimit(depth)

Looks like the C stack blows up past a depth of 85.

@markshannon

@markshannon what's the best way to update this test so it passes with a more limited stack size? Should I update Py_C_RECURSION_LIMIT in Include/cpython/pystate.h to something like 75 when under a debug build for WASI based on the "85" value I found to work?

I can also simply skip it much like on x360. I tried setting Py_C_RECURSION_LIMIT to pass various tests that were exhsusting their stacks (e.g., #111803 and #111802), but the limit was getting ridiculously low, to the point that test_pickle was failing on non-deep tests.

@hroncok

I also saw this issue on Linux (Fedora). But I failed to reproduce it in a reliable way.

cc @hroncok

When Python is built in debug mode, -Og or event -O0 optimization level is used. It's possible that Python consumes more stack memory and so crash earlier in test_super_deep(): with less deep recursive call.

We already had issues with other tests which rely on how much stack memory is available: https://pythondev.readthedocs.io/unstable_tests.html#unlimited-recursion

Well, I suppose that here the purpose of the test is to prove that Python no longer consumes any stack memory. But maybe this assumption is wrong when Python is built in debug mode?

I propose to reduce the maximum depth is test.support.Py_DEBUG is true. How much? I don't know.

I can reproduce the issue by building Python with gcc -O0 (disable all compiler optimizations):

./configure --cache-file=../python-config.cache --with-pydebug CFLAGS=-O0
make
./python -m test -v test_call -m test_super_deep

Output:

vstinner@mona$ ./python -m test -v test_call -m test_super_deep 
== CPython 3.13.0a1+ (heads/main:e5dfcc2b6e1, Nov 15 2023, 19:17:40) [GCC 13.2.1 20231011 (Red Hat 13.2.1-4)]
== Linux-6.5.11-300.fc39.x86_64-x86_64-with-glibc2.38 little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_674715æ
== CPU count: 12
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1935379936
0:00:00 load avg: 1.79 Run 1 test sequentially
0:00:00 load avg: 1.79 [1/1] test_call
test_super_deep (test.test_call.TestRecursion.test_super_deep) ... Fatal Python error: Segmentation fault

Current thread 0x00007fcd6b199740 (most recent call first):
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  File "/home/vstinner/python/main/Lib/test/test_call.py", line 1007 in c_recurse
  (...)

I tested commit e5dfcc2b6e1c8450e47acbc6e7d3fd3380ef0f03.

Using gcc -O1, I cannot reproduce the issue: so it seems like Python uses more and less stack memory depending on the compiler optimization level.

Measure with c_limit=90 py_limit=1,000 recursion_limit=100,000 on Fedora 39 (gcc 13.2.1 on x86-64):

gcc -O0 + --with-pydebug: c_py_recurse: 13 618.7 bytes/call
gcc -O1 + --with-pydebug: c_py_recurse: 689.8 bytes/call
gcc -Og + --with-pydebug: c_py_recurse: 705.6 bytes/call
gcc -O3 + --with-pydebug: c_py_recurse: 609.6 bytes/call
gcc -O0: c_py_recurse: 24 690.7 bytes/call
gcc -Og: c_py_recurse: 577.1 bytes/call
gcc -O3: c_py_recurse: 464.9 bytes/call

I suppose that the fact that -O0 disable function inlining has a high cost on stack memory consumption.

Script to measure c_py_recurse() stack memory consumption:

import _testcapi
import sys

sys.setrecursionlimit(100_000)
py_limit = 1000
c_limit = 90

lowest = None

def py_recurse(n, m):
    global lowest
    sp = _testcapi.stack_pointer()
    if lowest is None or sp < lowest:
        lowest = sp

    if n >= 1:
        py_recurse(n-1, m)
    else:
        c_py_recurse(m-1)


def c_py_recurse(m):
    global lowest
    sp = _testcapi.stack_pointer()
    if lowest is None or sp < lowest:
        lowest = sp

    if m >= 1:
        _testcapi.pyobject_vectorcall(py_recurse, (py_limit, m), ())


def measure(func, depth):
    global lowest
    lowest = None

    high = _testcapi.stack_pointer()
    func(depth)
    low = lowest
    per_call = (high - low) / depth
    print(f"{func.__name__}: {per_call:.1f} bytes/call "
          f"-- measured with {c_limit=:,} {py_limit=:,} "
          f"recursion_limit={sys.getrecursionlimit():,}")

measure(c_py_recurse, c_limit)

Latest major change related to recursion:

This change added test_super_deep() and skips it on s390x architecture.

Alright, i'm not surprised to see a different behavior on this specific recursion stress test depending on the compiler version and flags. I'm always using the same container, i don't know which versions are used inside.

could you shed some light as to the purpose of the failing test so we can update it appropriately?

It is to test that the interpreter doesn't crash when the recursion limit is set to some silly number.
Rather than disable the test we should probably set the C recursion limit to different values for different builds and platforms.

I'm still seeing a failure under WASI SDK 20 and wasmtime 14 after merging up to 762eb58 .

Tools/wasm/README.md says:

WASI builds require the WASI SDK 16.0+

I'm only using quay.io/tiran/cpythonbuild:emsdk3 container to build Python on WASI. I have no idea what is the WASI SDK version of this container. It would be nice to document supported WASI SDK versions and wasmtime versions.

So the problem is that the test_call fails if Python is built in debug mode with WASI SDK 20 and wasmtime 14?

I'm only using quay.io/tiran/cpythonbuild:emsdk3 container to build Python on WASI. I have no idea what is the WASI SDK version of this container.

Neither do I, hence why we install the requisite tools into the devcontainer.

It would be nice to document supported WASI SDK versions and wasmtime versions.

I'm working on it (and it's technically WASI-SDK 16+ and any version of wasmtime that fully implements WASI preview1, but in general just some recent version).

So the problem is that the test_call fails if Python is built in debug mode with WASI SDK 20 and wasmtime 14?

Correct as that's how I discovered the failure, but I don't know how strict those versions are in terms of causing the issue. But since those are the common/preferred tools for working on WASI and (nearly) the latest versions it's at least worth my time to try and make sure they work (and it's what the WASI buildbots use today, so fixing this means I can make them do pydebug builds).

pythonGH-111798: skip `test_super_deep()` from `test_call` under pyde…

GH-111798: skip `test_super_deep()` from `test_call` under pydebug bu…

I went ahead and turned the test off under WASI w/ a pydebug build as the numbers required to pass are no longer "some silly number", negating the point of the test.

pythonGH-111798: skip `test_super_deep()` from `test_call` under pyde…

pythongh-111798: Use lower Py_C_RECURSION_LIMIT in debug mode (python…

pythonGH-111798: skip `test_super_deep()` from `test_call` under pyde…

pythongh-111798: Use lower Py_C_RECURSION_LIMIT in debug mode (python…

pythonGH-111798: skip `test_super_deep()` from `test_call` under pyde…

Navigation Menu

Repository navigation

`test_super_deep()` for `test_call` triggers a stack overflow protection under a debug build for WASI #111798

Linked PRs

17 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	@skip_on_s390x
	def test_super_deep(self):

	def recurse(n):
	if n:
	recurse(n-1)

	def py_recurse(n, m):
	if n:
	py_recurse(n-1, m)
	else:
	c_py_recurse(m-1)

	def c_recurse(n):
	if n:
	_testcapi.pyobject_vectorcall(c_recurse, (n-1,), ())

	def c_py_recurse(m):
	if m:
	_testcapi.pyobject_vectorcall(py_recurse, (1000, m), ())

	depth = sys.getrecursionlimit()
	sys.setrecursionlimit(100_000)
	try:
	recurse(90_000)
	with self.assertRaises(RecursionError):
	recurse(101_000)
	c_recurse(100)
	with self.assertRaises(RecursionError):
	c_recurse(90_000)
	c_py_recurse(90)
	with self.assertRaises(RecursionError):
	c_py_recurse(100_000)
	finally:
	sys.setrecursionlimit(depth)

test_super_deep() for test_call triggers a stack overflow protection under a debug build for WASI #111798

Description

Linked PRs

Activity

brettcannon commented on Nov 8, 2023

brettcannon commented on Nov 8, 2023

brettcannon commented on Nov 8, 2023

vstinner commented on Nov 15, 2023

vstinner commented on Nov 15, 2023

vstinner commented on Nov 15, 2023

vstinner commented on Nov 15, 2023

vstinner commented on Nov 15, 2023

vstinner commented on Nov 15, 2023

17 remaining items

vstinner commented on Nov 16, 2023

markshannon commented on Nov 20, 2023

vstinner commented on Nov 28, 2023

brettcannon commented on Nov 29, 2023

brettcannon commented on Jan 13, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`test_super_deep()` for `test_call` triggers a stack overflow protection under a debug build for WASI #111798