Parser stack overflow on WASI with --with-pydebug

Bug report

Bug description:

$ python Tools/wasm/wasi.py configure-build-python -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-build-python
...
$ python Tools/wasm/wasi.py configure-host -- --with-pydebug
...
$ python Tools/wasm/wasi.py make-host
...
$ wasmtime run --dir .::/ cross-build/wasm32-wasip1/python.wasm -m test.test_compile
Traceback (most recent call last):
  File "/Lib/runpy.py", line 189, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
                               ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Lib/runpy.py", line 159, in _get_module_details
    code = loader.get_code(mod_name)
  File "<frozen importlib._bootstrap_external>", line 896, in get_code
  File "<frozen importlib._bootstrap_external>", line 826, in source_to_code
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
MemoryError: Parser stack overflowed - Python source too complex to parse

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

NOTE

The failing tests have shifted; see #131770 (comment)

Linked PRs

GH-131770: increase assumed WASI stack size to 131072 (wasi-sdk default) #131844

@markshannon

I bisected this to 0142236, specifically, the following call, which uses the new _Py_ReachedRecursionLimitWithMargin function.

cpython/Tools/peg_generator/pegen/c_generator.py

Lines 382 to 386 in 4b3d5b6

    
           def add_level(self) -> None: 
        
               self.print("if (p->level++ == MAXSTACK || _Py_ReachedRecursionLimitWithMargin(PyThreadState_Get(), 1)) {") 
        
               with self.indent(): 
        
                   self.print("_Pypegen_stack_overflow(p);") 
        
               self.print("}")

PYOS_STACK_MARGIN_BYTES seems a bit low, causing it to fail. In my last run, for example, I am seeing p->level at 209, which is quite far from the MAXSTACK limit. I don't know exactly how the WASM double stack model works, but I suspect that our PYOS_STACK_MARGIN value might be a bit low. I think expecting WASI to have a big enough stack for the parser to be able to parse all regular Python source files in the repo should be a fairly reasonable expectation.

cc @markshannon

pythonGH-131770: increase assumed WASI stack size to 131072 (wasi-sdk…

GH-131770: increase assumed WASI stack size to 131072 (wasi-sdk defau…

@hoodmane

So what are our options at this point?

Play with numbers until we find one that passes the test suite (that's normally what I do when our stack protection starts failing)?
Turn off the stack protection under WASI for what was changed (Implement stack overflow protection for webassembly #130397)?
Rip it out as I believe @hoodmane has suggested it might not actually work as intended?
Give up in debug builds in WASI as it's the most constant pain point and I don't know how useful it is?

I just worry the longer we don't fix #131769 the more pain it will take to get debug builds working again. Plus I don't want to go too far into the betas w/ this not working.

@markshannon

In my opinion, I think we should disable the stack protection until WASM provides a mechanism that allow us to implement it correctly.

Due to WASM being sandboxed, stack overflows are not as big of a concern there compared to native runtimes, so I don't think we are losing a particularly large amount of value by disabling this functionality.

That said, I do understand it might be frustrating for @markshannon, as the one who implemented it, so I'd appreciate his thoughts on this.

@hugovk

@hugovk I made this a deferred blocker as I don't think it matters for a beta since it's a behind-the-scenes mechanism, but I don't want to reach final w/o a solution as not being able to a debug build in WASI doesn't seem great.

I tried playing with various values to get a debug build to work to no avail.

cpython/Include/pythonrun.h

Lines 32 to 34 in 2b67db7

    
           #elif defined(__wasi__) 
        
              /* Web assembly has two stacks, so this isn't really a size */ 
        
           #  define PYOS_LOG2_STACK_MARGIN 9

With a value of 29 I get:

Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "/Lib/traceback.py", line 7, in <module>
    import textwrap
  File "/Lib/textwrap.py", line 8, in <module>
    import re
  File "/Lib/re/__init__.py", line 127, in <module>
    import functools
  File "/Lib/functools.py", line 517, in <module>
    _CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
  File "/Lib/collections/__init__.py", line 447, in namedtuple
    __new__ = eval(code, namespace)
MemoryError: Parser stack overflowed - Python source too complex to parse
Traceback (most recent call last):
  File "/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/Lib/test/__main__.py", line 1, in <module>
    from test.libregrtest.main import main
  File "/Lib/test/libregrtest/main.py", line 3, in <module>
    import re
  File "/Lib/re/__init__.py", line 127, in <module>
    import functools
  File "/Lib/functools.py", line 517, in <module>
    _CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])
  File "/Lib/collections/__init__.py", line 447, in namedtuple
    __new__ = eval(code, namespace)
object address  : 0x190ff20
object refcount : 6
object type     : 0x12e7698
object type name: MemoryError
object repr     : MemoryError('Parser stack overflowed - Python source too complex to parse')
lost sys.stderr

And with a value of 30 I get (truncated):

Fatal Python error: _Py_CheckRecursiveCall: Unrecoverable stack overflow (used 128 kB) while calling a Python object

You can use #133219 to get a pydebug of WASI again.

Now that b1 is out I have made this a release blocker for b2.

@hoodmane

This looks like a WASI configuration problem. configure.ac contains the line AS_VAR_APPEND([LDFLAGS_NODIST], [" -z stack-size=16777216 -Wl,--stack-first -Wl,--initial-memory=41943040"]) which looks like it sets the wasm stack to 16M, but LLVM sets the C stack size to only 130k. We should probably reduce the wasm stack size and increase the C stack size.

@hoodmane

I am not able to reproduce. I ran:

$ python3.12 ./Tools/wasm/wasi.py build -- --with-pydebug --config-cache
$ ./cross-build/wasm32-wasip1/python.sh -m test.test_compile
..........................................................................s................s...................................s....................s...............................
----------------------------------------------------------------------
Ran 180 tests in 3.435s

OK (skipped=4)

@markshannon

This looks like a WASI configuration problem. configure.ac contains the line AS_VAR_APPEND([LDFLAGS_NODIST], [" -z stack-size=16777216 -Wl,--stack-first -Wl,--initial-memory=41943040"]) which looks like it sets the wasm stack to 16M, but LLVM sets the C stack size to only 130k. We should probably reduce the wasm stack size and increase the C stack size.

@markshannon Is this the appropriate way to set the C stack size via Clang?

cpython/configure.ac

Line 3615 in 91e6a58

LINKFORSHARED="-Wl,-stack_size,$stack_size $LINKFORSHARED"

I am not able to reproduce

@hoodmane Was that with my patch to fix the breakage for detecting a debug build? Otherwise the WASI build ends up being a non-debug build. And you have to test with e.g. test_ast (see #131770 (comment) for the consistent failing tests).

No it's not with your patch to fix debug builds that explains why I can't reproduce.

And yes, -z stack-size=... is the right way to set the stack size in clang.

And yes, -z stack-size=... is the right way to set the stack size in clang.

So then if the stack size is already being set appropriately to 16 MiB, what flag is missing to make the stack not be 130K?

I've opened a question on the Bytecode Alliance Zulip instance to see if anyone over there has any ideas.

@markshannon

I've verified that -z stack-size for WASI SDK/clang and --wasm maxi-wasm-stack for wasmtime are the ways to increase the stack sizes on the BA's Zulip instance in the question I linked to yesterday.

@markshannon any other ideas about what to do here?

FYI I opened #134469 to fix this by just dropping the special-casing for WASI. That does cause some deep call stack tests to fail, but they were already turned off for Emscripten so I just followed suit. 😁

This is now fixed in main and 3.14.

Navigation Menu

Repository navigation

Parser stack overflow on WASI with `--with-pydebug` #131770

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

5 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	def add_level(self) -> None:
	self.print("if (p->level++ == MAXSTACK \|\| _Py_ReachedRecursionLimitWithMargin(PyThreadState_Get(), 1)) {")
	with self.indent():
	self.print("_Pypegen_stack_overflow(p);")
	self.print("}")

	#elif defined(__wasi__)
	/* Web assembly has two stacks, so this isn't really a size */
	# define PYOS_LOG2_STACK_MARGIN 9

Parser stack overflow on WASI with --with-pydebug #131770

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

FFY00 commented on Mar 26, 2025

brettcannon commented on Apr 21, 2025

FFY00 commented on Apr 22, 2025

brettcannon commented on Apr 24, 2025

brettcannon commented on Apr 30, 2025

5 remaining items

brettcannon commented on May 8, 2025

markshannon commented on May 19, 2025

hoodmane commented on May 19, 2025

brettcannon commented on May 20, 2025

hoodmane commented on May 20, 2025

hoodmane commented on May 20, 2025

brettcannon commented on May 20, 2025

brettcannon commented on May 21, 2025

brettcannon commented on May 21, 2025

brettcannon commented on May 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Parser stack overflow on WASI with `--with-pydebug` #131770