Skip to content

IDLE is unable to open any .py files #104719

Closed
Closed
IDLE is unable to open any .py files#104719
Assignees
pablogsalterryjreedy
Labels
3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-IDLEtype-bugAn unexpected behavior, bug, or error

Description

@AlexWaygood
AlexWaygood
opened on May 21, 2023

With a fresh CPython build (be0c106), IDLE is unable to open any .py files.

To reproduce:

  1. Create an empty .py file with the name repro.py
  2. Run python -m idlelib repro.py

IDLE still seems able to create new .py files and save them; it just can't open pre-existing .py files right now.

Traceback observed

C:\Users\alexw\coding\cpython>python -m idlelib repro.py
Running Debug|x64 interpreter...
Traceback (most recent call last):
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\__main__.py", line 7, in <module>
    idlelib.pyshell.main()
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 1640, in main
    if flist.open(filename) is None:
       ^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\filelist.py", line 37, in open
    edit = self.EditorWindow(self, filename, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\pyshell.py", line 135, in __init__
    EditorWindow.__init__(self, *args)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 289, in __init__
    self.set_indentation_params(is_py_src)
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1327, in set_indentation_params
    i = self.guess_indent()
        ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1574, in guess_indent
    opener, indented = IndentSearcher(self.text, self.tabwidth).run()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\alexw\coding\cpython\Lib\idlelib\editor.py", line 1646, in run
    save_tabsize = tokenize.tabsize
                   ^^^^^^^^^^^^^^^^
AttributeError: module 'tokenize' has no attribute 'tabsize'

Environment

(Given the cause of the bug, the environment details shouldn't really be relevant; but they're included here anyway, for completeness.)

Python 3.12.0a7+ (heads/main:be0c106789, May 21 2023, 12:00:27) [MSC v.1932 64 bit (AMD64)] on win32

Reproduces on a debug and non-debug build, FWIW.

Linked PRs

Activity

  1. AlexWaygood
    added
    type-bugAn unexpected behavior, bug, or error
    topic-IDLE
    release-blocker
    on May 21, 2023
  2. AlexWaygood
    assigned
    pablogsal
    and
    terryjreedy
    on May 21, 2023
  3. ezio-melotti
    added this to Release and Deferred blockers 🚫on May 21, 2023
  4. github-project-automation
    moved this to Todo in Release and Deferred blockers 🚫on May 21, 2023

  5. AlexWaygood commented on May 21, 2023

    @AlexWaygood
    AlexWaygood
    on May 21, 2023
    MemberAuthor

    Bisects to 6715f91 (unsurprisingly 😄)

    Cc. @mgmacias95, @pablogsal, @lysnikolaou, @isidentical

  6. AlexWaygood
    added
    3.12only security fixes
    on May 21, 2023

  7. pablogsal commented on May 21, 2023

    @pablogsal
    pablogsal
    on May 21, 2023
    Member

    Will check this today

  8. pablogsal commented on May 21, 2023

    @pablogsal
    pablogsal
    on May 21, 2023
    Member

    Looks like IDLE is using a bunch of non documented attributes of the tokenize module that are no longer used internally. The easier course of action is probably just restore these but note that there will be no guarantee that whatever idle is going with them will continue working.

    I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

  9. AlexWaygood commented on May 21, 2023

    @AlexWaygood
    AlexWaygood
    on May 21, 2023
    MemberAuthor

    I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

    Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

  10. AlexWaygood commented on May 21, 2023

    @AlexWaygood
    AlexWaygood
    on May 21, 2023
    MemberAuthor

    I ran the buildbots and the CI before merge so another question is why CI didn't catch this.

    Do any of the buildbots run the test suite with -u gui (or -u all), I wonder? A bunch of IDLE's tests actually open and close GUI windows, so they're not run unless you run the test suite with that option.

    That doesn't seem relevant; python -m test test_idle -u gui passes on main, on both debug and non-debug builds. So it does seem like there's some missing test coverage here.

  11. pablogsal commented on May 21, 2023

    @pablogsal
    pablogsal
    on May 21, 2023
    Member

    I checked a bit more how the constant that was eliminated is being used and seems that IDLE is monkey patching the value to modify the behaviour of the now deleted _tokenize private function to adapt something regarding tabsize.

    Unfortunately I don't think this is going to work and was out of contract anyway so I think the best course of action here is to eliminate that code from idlelib as restoring the constant will not restore whatever was happening before.

  12. AlexWaygood commented on May 21, 2023

    @AlexWaygood
    AlexWaygood
    on May 21, 2023
    MemberAuthor

    FWIW, I can find other places in prominent open-source projects where these undocumented constants (which have now been removed) are used:

    So it might be worth adding them back anyway, to limit the disruption caused by this change.

  13. 33 remaining items

  14. terryjreedy
    added this to IDLE Issueson May 22, 2023
  15. terryjreedy
    added a commit that references this issue on May 22, 2023

    pythongh-104719: IDLE - test editor.IndentSearcher

    446ddc4

  16. terryjreedy commented on May 22, 2023

    @terryjreedy
    terryjreedy
    on May 22, 2023
    Member

    I decided that it would in general be a good idea to have test coverage of references to non-idlelib modules. I have a toktest branch that adds coverage of the class containing the failure and all tokenize references in editor.py. I will make a PR tomorrow with or without coverage of tokenize references in 2 other idlelib modules.

  17. Yhg1s commented on May 22, 2023

    @Yhg1s
    Yhg1s
    on May 22, 2023
    Member

    FWIW, whether to deprecate or document the unused globals isn't really a RM question, but considering the difficulty of emitting warnings for them, I'd rather not do that right now (but for 3.13 it's fine). Documenting them as for internal use only and/or deprecated can be done after b1.

  18. terryjreedy
    moved this to In Progress in IDLE Issueson May 23, 2023
  19. terryjreedy
    added a commit that references this issue on May 24, 2023

    gh-104719: IDLE - test existence of all tokenize references. (#104767)

    e561c09
  20. miss-islington
    added 2 commits that reference this issue on May 24, 2023

    pythongh-104719: IDLE - test existence of all tokenize references. (p…

    1061d30

    pythongh-104719: IDLE - test existence of all tokenize references. (p…

    63336e2
  21. terryjreedy
    added 2 commits that reference this issue on May 24, 2023

    [3.12] gh-104719: IDLE - test existence of all tokenize references. (G…

    5e1799e

    [3.11] gh-104719: IDLE - test existence of all tokenize references. (G…

    22618a8

  22. terryjreedy commented on May 24, 2023

    @terryjreedy
    terryjreedy
    on May 24, 2023
    Member

    IDLE now has a regression test that will fail if any tokenizer names change. I suggest we close this and someone open a new issue about the now obsolete constants.

  23. AlexWaygood commented on May 24, 2023

    @AlexWaygood
    AlexWaygood
    on May 24, 2023
    MemberAuthor

    IDLE now has a regression test that will fail if any tokenizer names change. I suggest we close this and someone open a new issue about the now obsolete constants.

    That sounds good to me. I'll try to open an issue later today.

    Thanks @terryjreedy, @pablogsal and @mgmacias95!

  24. AlexWaygood
    closed this as completedon May 24, 2023
  25. github-project-automation
    moved this from In Progress to Done in Release and Deferred blockers 🚫on May 24, 2023
  26. github-project-automation
    moved this from In Progress to Done in IDLE Issueson May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.12only security fixesstdlibStandard Library Python modules in the Lib/ directorytopic-IDLEtype-bugAn unexpected behavior, bug, or error

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions