python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936)

how to replicate the issue

Python 3.11.3/3.11.2/3.11.1/3.11.0 (python-3.11.3-amd64.exe etc)
Microsoft Windows [Version 10.0.19044.2130] (Active code page: 936)

tmp.py

#coding:utf-8
print("中文")

python -m pdb tmp.py

Traceback (most recent call last):
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1774, in main
    pdb._run(target)
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 1652, in _run
    self.run(target.code)
             ^^^^^^^^^^^
  File "C:\Users\xond\AppData\Local\Programs\Python\Python311\Lib\pdb.py", line 167, in code
    return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"
                           ^^^^^^^^^
UnicodeDecodeError: 'gbk' codec can't decode byte 0xae in position 28: illegal multibyte sequence
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\users\xond\appdata\local\programs\python\python311\lib\pdb.py(167)code()
-> return f"exec(compile({fp.read()!r}, {self!r}, 'exec'))"

Linked PRs

This is an encoding issue - we can simply use io.read_code to avoid it, as we are always expecting the read code with pdb. PR is done with the regression test.

gh-103578: Fix pdb reading code with non-utf8 encoding (#103581)

pythongh-103578: Fix pdb reading code with non-utf8 encoding (pythonG…

[3.11] gh-103578: Fix pdb reading code with non-utf8 encoding (GH-103581

pythongh-103578: Fix pdb reading code with non-utf8 encoding (python#…

Thanks again!

This is a case that always should've gone through io.open_code, and people may be caught out by having code execution bypass that path.

We should backport to 3.8+

I take that back - 3.8-3.10 were already fixed, it was a change in 3.11 that regressed this. So no backports required.

Thanks for checking, sorry I could have more explicitly mentioned this! :-)

Navigation Menu

Repository navigation

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

python -m pdb <utf-8 encoding file.py> return UnicodeDecodeError on Windows 8.1/10 (code page 936) #103578

Description

Linked PRs

Activity

gaogaotiantian commented on Apr 16, 2023

hauntsaninja commented on Apr 26, 2023

zooba commented on Apr 27, 2023

zooba commented on Apr 27, 2023

hauntsaninja commented on Apr 27, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions