Repository navigation
string length overflows to negative when executing #107450
Description
Bug report
Checklist
- I am confident this is a bug in CPython,
not a bug in a third-party project - I have searched the CPython issue tracker,
and am confident this bug has not been reported before
A clear and concise description of the bug
The following code causes an error to be raised in all tested versions of Python.
exec(f"if True:\n {' ' * 2**31}print('hello world')")Which error is raised varies by version, and as such the specific error message is listed with its corresponding version in the below section.
Some (hopefully) helpful information:
- I have confirmed that the creation of the string is not causing a problem (by creating it separately, storing it in a variable, and then
exec'ing it), the problem occurs when attempting toexecit. - I have confirmed that the issue is does not occur solely because of the large string. (The same issue does not occur when I use a value of, for example,
2**33).
NOTE:
I believe that this error will also occur when running real file rather than building a string and using exec, but I have not yet confirmed that.
Your environment
I tested this in three environments, with as many versions of Python as I could conveniently use.
Arch GNU/Linux
- Operating system and architecture:
- Operating system:
Arch GNU/Linux - CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
- Operating system:
- CPython versions:
Python 3.10.10 (main, Mar 5 2023, 22:56:53) [GCC 12.2.1 20230201]IndentationError: unindent does not match any outer indentation level
Python 3.11.3 (main, Jun 5 2023, 09:32:32) [GCC 13.1.1 20230429]SystemError: Negative size passed to PyUnicode_New
Microsoft Windows 10
- Operating system and architecture:
- Operating system:
Microsoft Windows 10 (version 22H2 build 19045.3208)inside a VirtualBox virtual machine - CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
- Operating system:
- CPython versions:
Python 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 16:30:00) [MSC v.1900 64 bit (AMD64)]IndentationError: None
Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]IndentationError: None
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]IndentationError: unindent does not match any outer indentation level
Python 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]IndentationError: unindent does not match any outer indentation level
Python 3.11.4 (tags/v3.11.4:d2340ef, Jun 7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]SystemError: Negative size passed to PyUnicode_New
Ubuntu GNU/Linux
- Operating system and architecture:
- Operating system:
Ubuntu 22.04 GNU/Linuxinside a VirtualBox virtual machine - CPU: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz 2.80 GHz
- Operating system:
- CPython versions:
Python 3.7.17 (default, Jun 6 2023, 20:10:09) [GCC 11.3.0]IndentationError: None
Python 3.8.17 (default, Jun 6 2023, 20:10:50) [GCC 11.3.0]IndentationError: None
Python 3.9.17 (main, Jun 6 2023, 20:11:21) [GCC 11.3.0]IndentationError: unindent does not match any outer indentation level
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]IndentationError: unindent does not match any outer indentation level
Python 3.11.4 (main, Jun 7 2023, 12:45:48) [GCC 11.3.0]SystemError: Negative size passed to PyUnicode_New
Should any further information be required about the environments that I used, please ask.
What I believe causes the problem
I suspect very strongly that the error occurs as a result of an integer overflow. (A value larger than 2**31 - 1 will overflow and become negative when using 2's complement with 32 bit signed integers).
Possible solution
I think that this can be fixed by using a 64 bit integer to hold the length of the string while 64 bit platforms. This, while still leaving the error theoretically possible, would in practice avoid the error entirely because one would encounter a MemoryError first.
Linked PRs
- gh-107450: Raise OverflowError when parser column offset overflows #110754
- [3.12] gh-107450: Raise OverflowError when parser column offset overflows (GH-110754) #110762
- [3.11] gh-107450: Raise OverflowError when parser column offset overflows (GH-110754) #110763
- gh-107450: Fix parser column offset overflow test on Windows #110768
- GH-107450: fix unexpected OverflowError in testMemoryErrorBigSource #110801
- [3.12] gh-107450: Fix parser column offset overflow test on Windows (GH-110768) #110808
- [3.11] gh-107450: Fix parser column offset overflow test on Windows (GH-110768) #110809
- gh-107450: Fix testMemoryErrorBigSource using the bigmemtest decorator #110810
- gh-107450: Check for overflow in the tokenizer and fix overflow test #110832
- gh-107450: Fix testMemoryErrorBigSource in test_exceptions #110871
- [3.12] gh-107450: Check for overflow in the tokenizer and fix overflow test (GH-110832) #110931
- [3.11] gh-107450: Check for overflow in the tokenizer and fix overflow test (GH-110832) #110939
- gh-107450: Remove unnecessary overflow check in parser error handler #110940
Activity
[-]Indentation string length overflow[/-][+]string length overflows to negative when executing[/+]terryjreedy commentedon Jul 30, 2023
There have been multiple reports that extreme artificial code produces compile problems or runtime problems, either exceptions or crashes. The answer for some is "don't do that; all compilers have limits".
'exec' compiles and runs. IndentationErrors are compile errors. Installed 3.12.0b4 Win 10, running
compileon IDLE gives the same SystemError. Fresh 3.13.0a0 debug REPL crashes after perhaps a minute with C Runtime debug error:abort() has been called.serhiy-storchaka commentedon Jul 30, 2023
I agree that "don't do that" is the answer, but SystemError should never be triggered by the user Python code. It signals about improper use of the C API or some issues in the core. Although the former IndentationError was also not very useful. An error should explicitly report about the compiler limitation, not rely on the undefined behavior of integer overflow in C.
On main in debug build it crashes after long delay.
mqyhlkahu commentedon Jul 30, 2023
Thank you both for your response!
Yes, I am aware that compilers have limits, and I don't think that's a bad thing! It would take a lot more work, and quite probably be a lot slower, were these limitations to be removed.
However, I am of the opinion that:
SystemErrors shouldn't be raised because of user code (except forraise SystemErrorand the like, of course).IndentationErrors are also bad, because they don't really explain the problem (and, in the case of 3.9 / 3.10, actively mislead the user).While I would certainly prefer this to be fixed, I understand that it probably won't be. However, I feel that this should at least provide a useful error message (and not provide a
SystemError), perhaps something along the lines of the exception that results fromIt would hopefully utilise
BaseExceptionso that the error is less likely to be caught unintentionally by someone. (Ideally, there would be an exception-type derived fromBaseExceptioncalledCompilerErroror something like that, but introducing a new exception-type is FAR beyond the scope of this issue).pythongh-107450: Raise MemoryError when parser column offset overflows
gh-107450: Raise OverflowError when parser column offset overflows (#…
pythongh-107450: Raise OverflowError when parser column offset overfl…
pythongh-107450: Raise OverflowError when parser column offset overfl…
[3.11] gh-107450: Raise OverflowError when parser column offset overf…
16 remaining items
pythongh-107450: Fix testMemoryErrorBigSource in test_exceptions
gh-107450: Check for overflow in the tokenizer and fix overflow test (#…
[3.12] pythongh-107450: Check for overflow in the tokenizer and fix o…
[3.11] pythongh-107450: Check for overflow in the tokenizer and fix o…
pythongh-107450: Remove unnecessary overflow check in parser error ha…
[3.12] gh-107450: Check for overflow in the tokenizer and fix overflo…
gh-107450: Remove unnecessary overflow check in parser error handler (#…
[3.11] gh-107450: Check for overflow in the tokenizer and fix overflo…
pythongh-107450: Check for overflow in the tokenizer and fix overflow…
pythongh-107450: Remove unnecessary overflow check in parser error ha…
pythongh-107450: Raise OverflowError when parser column offset overfl…
pythongh-107450: Fix parser column offset overflow test on Windows (p…
pythongh-107450: Fix testMemoryErrorBigSource using the bigmemtest de…
pythongh-107450: Check for overflow in the tokenizer and fix overflow…
pythongh-107450: Remove unnecessary overflow check in parser error ha…