Skip to content

inspect.BlockFinder handles tokenize.NEWLINE too strictly #139783

Closed
Closed
inspect.BlockFinder handles tokenize.NEWLINE too strictly#139783
Assignees
serhiy-storchaka
Labels
3.13only security fixes3.14bugs and security fixes3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@BabakkGraphcore
BabakkGraphcore
opened on Oct 8, 2025

Bug report

Bug description:

In python 3.13.8 calling inspect.getsourcelines on a decorated function with a comment between the decorator and the name only returns the function declaration, the body is skipped.

import inspect

def dummy(fn):
    return fn

@dummy
def test1(a, b):
    return a

@dummy
#comment
def test2(a, b):
    return a

print(inspect.getsourcelines(test1))
print("------------------")
print(inspect.getsourcelines(test2))

returns

(['@dummy\n', 'def test1(a, b):\n', '    return a\n'], 6)
------------------
(['@dummy\n', '#comment\n', 'def test2(a, b):\n'], 10)

the source of test2 is incomplete and missing the return statement.

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

Activity

  1. BabakkGraphcore
    added
    type-bugAn unexpected behavior, bug, or error
    on Oct 8, 2025
  2. picnixz
    added
    stdlibStandard Library Python modules in the Lib/ directory
    on Oct 8, 2025
  3. picnixz
    added this to Inspect issues 🔍on Oct 8, 2025

  4. picnixz commented on Oct 8, 2025

    @picnixz
    picnixz
    on Oct 8, 2025
    Member

    I'll have a look at this.

  5. picnixz
    self-assigned this
    on Oct 8, 2025

  6. picnixz commented on Oct 8, 2025

    @picnixz
    picnixz
    on Oct 8, 2025
    Member

    The issue is with the BlockFinder which, because of the comment, considers that the body is not part of the function anymore.

  7. picnixz
    changed the title [-]`inspect.getsourcelines` incomplete when called on decorated function with a comment[/-] [+]`inspect.BlockFinder` handles `tokenize.NEWLINE` too strictly[/+] on Oct 8, 2025

  8. picnixz commented on Oct 8, 2025

    @picnixz
    picnixz
    on Oct 8, 2025
    Member

    Similar issues:

    @dummy

@dummy
def f():
    return 1

print(inspect.getsourcelines(f))  # ['@dummy\n', '\n', '@dummy\n']

    So it's not only comments that are poorly handled, it's anything that "breaks" the "prelude" of the function (that is decorators). I'm not entirely sure we can do much thing here without adding a layer of complexity. It shouldn't be possible to say that the source lines of something only contains decorators or new lines/comments as this is not valid Python code. So, we should do something else.

    It would be quite easy to actually preprocess the source and eliminate "un-necessary" lines but then we'll need to handle the offset properly. In a single pass, we'll need to determine whether the block ended correctly or not.

    Similarly:

    @\
dummy
def f():
    return 1

    This is equivalent to use @dummy but again, the source lines will only consist of dummy and nothing else! But this is a separate issue actually, where findsource already struggles with that (it will not put @ and dummy together...)

  9. picnixz commented on Oct 8, 2025

    @picnixz
    picnixz
    on Oct 8, 2025
    Member

    I need to go now but I'll try to do something tomorrow or later this week. Thanks for reporting this!

  10. BabakkGraphcore commented on Oct 8, 2025

    @BabakkGraphcore
    BabakkGraphcore
    on Oct 8, 2025
    Author

    I need to go now but I'll try to do something tomorrow or later this week. Thanks for reporting this!

    Great, thanks for picking it up so quickly1

  11. peterbell10
    added a commit that references this issue on Oct 8, 2025

    [NFC] Python3.13 fix (#8403)

    a612ca5

  12. serhiy-storchaka commented on Oct 9, 2025

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 9, 2025
    Member

    I worked with that code recently, so if you need help, I can help.

  13. picnixz commented on Oct 9, 2025

    @picnixz
    picnixz
    on Oct 9, 2025
    Member

    I am a bit time constrained so I would appreciate your help especially if it is something you recently worked on (I only skimmed through the implementation)

  14. albertvillanova commented on Oct 9, 2025

    @albertvillanova
    albertvillanova
    on Oct 9, 2025
    Contributor

    I confirm this issue and I give some information about severity and impact.

    Additional Impact: Breaking PyTorch JIT + HuggingFace Transformers

    This bug is affecting production code in the PyTorch and HuggingFace ecosystem.

    Real-world Example

    The HuggingFace transformers library contains multiple instances of this pattern. For example, in transformers/models/deberta_v2/modeling_deberta_v2.py (lines 105-108): https://github.com/huggingface/transformers/blob/be3fa93b298eec4182a0bb6c5e64270a5f1e8020/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L105-L108

    @torch.jit.script
# Copied from transformers.models.deberta.modeling_deberta.c2p_dynamic_expand
def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
    return c2p_pos.expand([query_layer.size(0), query_layer.size(1), query_layer.size(2), relative_pos.size(-1)])

    When this module is imported on Python 3.13.8, the @torch.jit.script decorator calls inspect.getsourcelines(), which returns only:

    @torch.jit.script
# Copied from transformers.models.deberta.modeling_deberta.c2p_dynamic_expand
def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):

    PyTorch then tries to parse this truncated source with ast.parse(), which fails with:

    File "<unknown>", line 3
    def c2p_dynamic_expand(c2p_pos, query_layer, relative_pos):
                                                               ^
  IndentationError: expected an indented block after function definition on line 3

    Root Cause Confirmed

    I traced through the tokenization and can confirm the issue is in BlockFinder.tokeneater():

    1. After the decorator's NEWLINE token, self.indecorator is set to False
    2. The COMMENT token then enters the if not self.started and not self.indecorator: branch
    3. Since the token is not "def" or "class", it sets self.singleline = True
    4. This causes BlockFinder to treat the comment as a lambda/generator expression
    5. When the function signature's NEWLINE is encountered, it raises EndOfBlock prematurely

    Regression Confirmation

    ✅ Python 3.13.7: Works correctly
    ❌ Python 3.13.8: Broken

    This appears to be a side effect of the fix for gh-137477 (generator expressions):

  15. serhiy-storchaka
    self-assigned this
    on Oct 9, 2025

  16. 24 remaining items

  17. picnixz commented on Oct 25, 2025

    @picnixz
    picnixz
    on Oct 25, 2025
    Member

    Could you ban the user above for spam @python/organization-owners? TiA.

  18. tmoreau89
    added a commit that references this issue on Dec 1, 2025

    [NFC] Python3.13 fix (triton-lang#8403)

    e69fa2e
  19. meta-codesync
    added a commit that references this issue on Mar 24, 2026

    [triton][beta] [Cherry-pick] '[NFC] Python3.13 fix (#8403)'

    f9cb0e5
  20. malfet
    added a commit that references this issue on Apr 1, 2026

    Work around CPython 3.13.8 inspect.getsourcelines() bug in _overload_…

    65a8288
  21. pytorchmergebot
    added 2 commits that reference this issue on Apr 2, 2026

    Work around CPython 3.13.8 inspect.getsourcelines() bug in _overload_…

    f337d06

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (#179066)

    946adcd
  22. weifengpy
    added a commit that references this issue on Apr 7, 2026

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (#179066)

    8f5346d
  23. nklshy-aws
    added a commit that references this issue on Apr 7, 2026

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (pytorch#17…

    ec1c7e6
  24. bobrenjc93
    added a commit that references this issue on Apr 10, 2026

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (pytorch#17…

    2ffbdf8
  25. Sakura286
    added a commit that references this issue on Jun 11, 2026

    python-torch: fix import failure caused by cpython 3.13.8 inspect bug

    975bb6b
  26. gplutop7
    added a commit that references this issue on Jul 15, 2026

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (pytorch#17…

    07760f8
  27. aws-kingrj
    added a commit that references this issue on Jul 29, 2026

    [JIT] Silence CPython 3.13.8 inspect.getsourcelines() bug (pytorch#17…

    789f8fe
  28. 245516766
    added a commit that references this issue on Aug 3, 2026

    [NFC] Python3.13 fix (#8403)

    b982392
  29. danielhanchen
    added 2 commits that reference this issue on Aug 5, 2026

    Ask uv for a Python that can import torch, and skip the one that cannot

    7d0d505

    Ask uv for a Python that can import torch, and skip the one that cann…

    976a115
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.13only security fixes3.14bugs and security fixes3.15bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions