Skip to content

Exception raised in traceback.StackSummary._should_show_carets exits interpreter: IndexError on tree.body[0] #122145

Closed
Closed
Exception raised in traceback.StackSummary._should_show_carets exits interpreter: IndexError on tree.body[0]#122145
Labels
type-bugAn unexpected behavior, bug, or error

Description

@devdanzin
devdanzin
opened on Jul 22, 2024

Bug report

Bug description:

Running the code sample from #122071 in 3.13.0b4 or main exits the interpreter due to traceback.StackSummary._should_show_carets raising an exception:

>>> exec(compile("tuple()[0]", "s", "exec"))
Traceback (most recent call last):
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 139, in _print_exception_bltin
    return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 130, in print_exception
    te.print(file=file, chain=chain, colorize=colorize)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 1448, in print
    for line in self.format(chain=chain, colorize=colorize):
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 1384, in format
    yield from _ctx.emit(exc.stack.format(colorize=colorize))
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 747, in format
    formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 583, in format_frame_summary
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 701, in _should_show_carets
    statement = tree.body[0]
IndexError: list index out of range
Traceback (most recent call last):
  File "~\PycharmProjects\cpython\Lib\code.py", line 91, in runcode
    exec(code, self.locals)
  File "<python-input-2>", line 1, in <module>
  File "s", line 1, in <module>
IndexError: tuple index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~\PycharmProjects\cpython\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "~\PycharmProjects\cpython\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "~\PycharmProjects\cpython\Lib\_pyrepl\__main__.py", line 6, in <module>
    __pyrepl_interactive_console()
  File "~\PycharmProjects\cpython\Lib\_pyrepl\main.py", line 59, in interactive_console
    run_multiline_interactive_console(console)
  File "~\PycharmProjects\cpython\Lib\_pyrepl\simple_interact.py", line 156, in run_multiline_interactive_console
    more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
  File "~\PycharmProjects\cpython\Lib\code.py", line 303, in push
    more = self.runsource(source, filename, symbol=_symbol)
  File "~\PycharmProjects\cpython\Lib\_pyrepl\console.py", line 200, in runsource
    self.runcode(code)
  File "~\PycharmProjects\cpython\Lib\code.py", line 95, in runcode
    self.showtraceback()
  File "~\PycharmProjects\cpython\Lib\_pyrepl\console.py", line 168, in showtraceback
    super().showtraceback(colorize=self.can_colorize)
  File "~\PycharmProjects\cpython\Lib\code.py", line 147, in showtraceback
    lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next, colorize=colorize)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 155, in format_exception
    return list(te.format(chain=chain, colorize=colorize))
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 1384, in format
    yield from _ctx.emit(exc.stack.format(colorize=colorize))
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 747, in format
    formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 583, in format_frame_summary
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
  File "~\PycharmProjects\cpython\Lib\traceback.py", line 701, in _should_show_carets
    statement = tree.body[0]
IndexError: list index out of range
[Thread 31424.0x8ec0 exited with code 1]
[Thread 31424.0x15dc exited with code 1]
[Thread 31424.0x8b8c exited with code 1]
[Inferior 1 (process 31424) exited with code 01]

We can either protect the _should_show_carets call with suppress(Exception) here:

cpython/Lib/traceback.py

Lines 581 to 583 in 2762c6c

with suppress(Exception):
anchors = _extract_caret_anchors_from_line_segment(segment)
show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)

Or guard against tree.body being empty here:

cpython/Lib/traceback.py

Lines 700 to 701 in 2762c6c

tree = ast.parse('\n'.join(all_lines))
statement = tree.body[0]

(or both?)

Should I submit a PR with one of the fixes above? Any preferences?

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Windows, Linux

Linked PRs

Activity

  1. devdanzin
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 22, 2024
  2. devdanzin
    changed the title [-]Exception raised in traceback.StackSummary._should_show_carets exits interpreter on Windows[/-] [+]Exception raised in traceback.StackSummary._should_show_carets exits interpreter[/+] on Jul 23, 2024

  3. gaogaotiantian commented on Jul 23, 2024

    @gaogaotiantian
    gaogaotiantian
    on Jul 23, 2024
    Member

    In my opinion, we should figure out that issue first. This is probably something that should not happen - it's kind of equivalent to a comment only file generates an exception, that's just impossible.

  4. devdanzin commented on Jul 23, 2024

    @devdanzin
    devdanzin
    on Jul 23, 2024
    MemberAuthor

    In my opinion, we should figure out that issue first. This is probably something that should not happen - it's kind of equivalent to a comment only file generates an exception, that's just impossible.

    Indeed, ISTM that due to the linecache bug we try to parse "lines" consisting of only a comment then access the tree's body first node. So fixing that issue will fix this specific failure.

    Maybe it would still be interesting to make the code more robust against exceptions in _should_show_carets?

  5. sobolevn commented on Jul 23, 2024

    @sobolevn
    sobolevn
    on Jul 23, 2024
    Member

    Looks like there are two bugs:

    1. With incorrect module detection, exception is not in Lib/_pyrepl/__main__.py (this file is used for sources)
    2. But, tree.body can be empty in other cases as well, so why not guard it?
    diff --git Lib/traceback.py Lib/traceback.py
index 6ee1a50ca68..eba6f03a3c6 100644
--- Lib/traceback.py
+++ Lib/traceback.py
@@ -698,6 +698,8 @@ def _should_show_carets(self, start_offset, end_offset, all_lines, anchors):
         with suppress(SyntaxError, ImportError):
             import ast
             tree = ast.parse('\n'.join(all_lines))
+            if not tree.body:
+                return False
             statement = tree.body[0]
             value = None
             def _spawns_full_line(value):

    After:

    » ./python.exe
Python 3.14.0a0 (heads/main-dirty:3de092b82f5, Jul 20 2024, 09:23:52) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exec(compile("tuple()[0]", "s", "exec"))
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    exec(compile("tuple()[0]", "s", "exec"))
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "s", line 1, in <module>
    # Important: don't add things to this module, as they will end up in the REPL's
IndexError: tuple index out of range

  6. picnixz commented on Jul 23, 2024

    @picnixz
    picnixz
    on Jul 23, 2024
    Member

    I've fixed the second issue and added a test for that one. I can move to fixing the first issue as well.

  7. devdanzin commented on Jul 23, 2024

    @devdanzin
    devdanzin
    on Jul 23, 2024
    MemberAuthor
    2. But, `tree.body` can be empty in other cases as well, so why not guard it?

    Exactly. Another way to get an empty tree.body is to run:

    exec(compile("tuple()[0]", "x.py", "exec"))

    Where you have a x.py file that consists of:

    # Just a comment.

  8. devdanzin commented on Jul 23, 2024

    @devdanzin
    devdanzin
    on Jul 23, 2024
    MemberAuthor

    Yet another way to get an empty tree.body:

    exec(compile("print(2**100000)", "s", "exec"))

    EDIT: never mind, I had reverted the fix.

  9. picnixz commented on Jul 23, 2024

    @picnixz
    picnixz
    on Jul 23, 2024
    Member

    Errr... ok. I'll amend the NEWS entry in the PR since it only talks about comments.

    EDIT: Actually, I think you can only have an empty AST body if you are either an empty string or if you are a comment.

  10. devdanzin commented on Jul 30, 2024

    @devdanzin
    devdanzin
    on Jul 30, 2024
    MemberAuthor

    In StackSummary.format_frame_summary, StackSummary._should_show_carets is called without error suppression (like is done for _extract_caret_anchors_from_line_segment). This causes any errors in that method to exit the interpreter. I suggest we also ignore errors from _should_show_carets here:

    cpython/Lib/traceback.py

    Lines 581 to 583 in d27a53f

    with suppress(Exception):
    anchors = _extract_caret_anchors_from_line_segment(segment)
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)

    This time, an error in parsing a too deeply nested structure exits the interpreter:

    >>> with open("lambda.txt", "w") as n:
...     n.write("lambda: lambda: " * 1000 + "None")
...
16004
>>> exec(compile("str(2**100000)", "lambda.txt", "exec"))
Traceback (most recent call last):
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 139, in _print_exception_bltin
    return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 130, in print_exception
    te.print(file=file, chain=chain, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 1449, in print
    for line in self.format(chain=chain, colorize=colorize):
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 1385, in format
    yield from _ctx.emit(exc.stack.format(colorize=colorize))
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 748, in format
    formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 583, in format_frame_summary
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 701, in _should_show_carets
    tree = ast.parse('\n'.join(all_lines))
  File "/home/danzin/projects/cpython/Lib/ast.py", line 53, in parse
    return compile(source, filename, mode, flags,
RecursionError: maximum recursion depth exceeded during compilation
Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<python-input-1>", line 1, in <module>
  File "lambda.txt", line 1, in <module>
    lambda: lambda: lambda: lambda: lambda: [...] lambda: None
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/danzin/projects/cpython/Lib/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/__main__.py", line 6, in <module>
    __pyrepl_interactive_console()
  File "/home/danzin/projects/cpython/Lib/_pyrepl/main.py", line 59, in interactive_console
    run_multiline_interactive_console(console)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/simple_interact.py", line 156, in run_multiline_interactive_console
    more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
  File "/home/danzin/projects/cpython/Lib/code.py", line 303, in push
    more = self.runsource(source, filename, symbol=_symbol)
  File "/home/danzin/projects/cpython/Lib/_pyrepl/console.py", line 200, in runsource
    self.runcode(code)
  File "/home/danzin/projects/cpython/Lib/code.py", line 95, in runcode
    self.showtraceback()
  File "/home/danzin/projects/cpython/Lib/_pyrepl/console.py", line 168, in showtraceback
    super().showtraceback(colorize=self.can_colorize)
  File "/home/danzin/projects/cpython/Lib/code.py", line 147, in showtraceback
    lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 155, in format_exception
    return list(te.format(chain=chain, colorize=colorize))
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 1385, in format
    yield from _ctx.emit(exc.stack.format(colorize=colorize))
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 748, in format
    formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 583, in format_frame_summary
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
  File "/home/danzin/projects/cpython/Lib/traceback.py", line 701, in _should_show_carets
    tree = ast.parse('\n'.join(all_lines))
  File "/home/danzin/projects/cpython/Lib/ast.py", line 53, in parse
    return compile(source, filename, mode, flags,
RecursionError: maximum recursion depth exceeded during compilation

    If an even deeper nested structure is used, MemoryError is raised, but somehow doesn't exit the interpreter.

    Doesn't seem to affect 3.13. Should a new issue be created for this one?

  11. vstinner commented on Sep 17, 2024

    @vstinner
    vstinner
    on Sep 17, 2024
    Member

    IMO #122071 is a duplicate of this issue.

  12. hroncok commented on Sep 17, 2024

    @hroncok
    hroncok
    on Sep 17, 2024
    Contributor

    IMO #122071 is a duplicate of this issue.

    I think #122071 affects all the Python versions:

    # Hi, I'm in the error message
import traceback
try:
    exec(compile("tuple()[0]", "s", "exec"))
except:
    traceback.print_exc()
    $ python3.12 repro.py 
Traceback (most recent call last):
  File "/home/churchyard/tmp/repro.py", line 4, in <module>
    exec(compile("tuple()[0]", "s", "exec"))
  File "s", line 1, in <module>
    # Hi, I'm in the error message
    ^^^^^^^^^^
IndexError: tuple index out of range

    While this one only affects Python 3.13, e.g.:

    >>> code = "#boom\nraise RuntimeError('see the line above')"
>>> eval(compile(code, "str_eval", "exec"))
Traceback (most recent call last):
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
  File "str_eval", line 2, in <module>
RuntimeError: see the line above

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib64/python3.13/_pyrepl/__main__.py", line 6, in <module>
    __pyrepl_interactive_console()
  File "/usr/lib64/python3.13/_pyrepl/main.py", line 59, in interactive_console
    run_multiline_interactive_console(console)
  File "/usr/lib64/python3.13/_pyrepl/simple_interact.py", line 157, in run_multiline_interactive_console
    more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
  File "/usr/lib64/python3.13/code.py", line 313, in push
    more = self.runsource(source, filename, symbol=_symbol)
  File "/usr/lib64/python3.13/_pyrepl/console.py", line 205, in runsource
    self.runcode(code)
  File "/usr/lib64/python3.13/code.py", line 96, in runcode
    self.showtraceback()
  File "/usr/lib64/python3.13/code.py", line 129, in showtraceback
    self._showtraceback(typ, value, tb.tb_next, '')
  File "/usr/lib64/python3.13/code.py", line 144, in _showtraceback
    self._excepthook(typ, value, tb)
  File "/usr/lib64/python3.13/_pyrepl/console.py", line 169, in _excepthook
    lines = traceback.format_exception(
  File "/usr/lib64/python3.13/traceback.py", line 155, in format_exception
    return list(te.format(chain=chain, colorize=colorize))
  File "/usr/lib64/python3.13/traceback.py", line 1388, in format
    yield from _ctx.emit(exc.stack.format(colorize=colorize))
  File "/usr/lib64/python3.13/traceback.py", line 747, in format
    formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
  File "/usr/lib64/python3.13/traceback.py", line 583, in format_frame_summary
    show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
  File "/usr/lib64/python3.13/traceback.py", line 701, in _should_show_carets
    statement = tree.body[0]
IndexError: list index out of range

    I think they build on top of each other -- when #122071 is fixed, this one will likely also get fixed, but I am not quite sure.

  13. vstinner
    changed the title [-]Exception raised in traceback.StackSummary._should_show_carets exits interpreter[/-] [+]Exception raised in traceback.StackSummary._should_show_carets exits interpreter: IndexError on tree.body[0][/+] on Sep 17, 2024
  14. vstinner
    added
    release-blocker
    on Sep 18, 2024

  15. vstinner commented on Sep 18, 2024

    @vstinner
    vstinner
    on Sep 18, 2024
    Member

    This bug is impacting Fedora Installer (Anaconda) and was marked as a Fedora 41 blocker issue: https://bugzilla.redhat.com/show_bug.cgi?id=2311907 So I also mark this bug as a blocker issue.

  16. ezio-melotti
    added this to Release and Deferred blockers 🚫on Sep 18, 2024
  17. github-project-automation
    moved this to Todo in Release and Deferred blockers 🚫on Sep 18, 2024
  18. vstinner
    added a commit that references this issue on Sep 18, 2024

    gh-122145: Handle an empty AST body when reporting tracebacks (#122161)

    5cd50cb
  19. miss-islington
    added a commit that references this issue on Sep 18, 2024

    pythongh-122145: Handle an empty AST body when reporting tracebacks (p…

    d474ae3
  20. vstinner
    removed
    release-blocker
    on Sep 18, 2024

  21. vstinner commented on Sep 18, 2024

    @vstinner
    vstinner
    on Sep 18, 2024
    Member

    @Yhg1s explained me that if we add a bugfix after 3.13rc2, another rc3 is needed. I don't think that this bug should require a whole new rc3. So I removed the release-blocker label. The fix can wait for 3.13.1 even if it's bad.

    For Fedora, we can have a downstream patch to unblock Fedora 41, until the fix is merged into the 3.13 branch.

  22. vstinner commented on Sep 18, 2024

    @vstinner
    vstinner
    on Sep 18, 2024
    Member

    Bug fixed by the change 5cd50cb. A backport to 3.13 will follow. I close the issue.

    See #122071 for the second traceback bug.

  23. vstinner
    closed this as completedon Sep 18, 2024
  24. github-project-automation
    moved this from Todo to Done in Release and Deferred blockers 🚫on Sep 18, 2024
  25. hroncok
    added a commit that references this issue on Sep 18, 2024

    00439: pythongh-122145: Handle an empty AST body when reporting trace…

    0ec030e
  26. savannahostrowski
    added a commit that references this issue on Sep 22, 2024

    pythongh-122145: Handle an empty AST body when reporting tracebacks (p…

    96c03b3
  27. Yhg1s
    added a commit that references this issue on Sep 30, 2024

    [3.13] gh-122145: Handle an empty AST body when reporting tracebacks (G…

    99dc089
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions