(🐞) Column marker in error message off when line contains non ASCII character

b"Ā"

👉 python test.py
  File "/test.py", line 1
    b"Ā"
         ^
SyntaxError: bytes can only contain ASCII literal characters

Here the caret is pointing to blackspace after the code

So each non-ASCII character is adding more incorrect offset:

👉 $c:temp = "b'ĀĀĀĀĀĀĀĀ'"
👉 py temp
  File "C:\temp", line 1
    b'ĀĀĀĀĀĀĀĀ'
                       ^

Expected

👉 python test.py
  File "/test.py", line 1
    b"Ā"
      ^
SyntaxError: bytes can only contain ASCII literal characters

Linked PRs

Can confirm on Python 3.12.0a5+ (heads/main:bcadcde712, Feb 26 2023, 12:54:46) [MSC v.1929 64 bit (AMD64)] on win32.

Note: Ā here is UTF8-encoded as C4 80 (a single U+0100 LATIN CAPITAL LETTER A WITH MACRON). So incorrect row advancing on combined code points is totally ruled out until text normalization happens inside the parser.

@benjaminp

@benjaminp, @pablogsal (as parser experts)

update: @gvanrossum, @lysnikolaou (as new parser experts)

In IDLE, Python 3.12.0a5+ (heads/main:0f89acf6cc, Feb 27 2023, 21:33:28) says "incomplete input" for this and "\N{a}", reported in #102312, and marks the closing quote, but this is effect of how codeop compiles.
In REPL, I see the above and the #102312 message. I closed #102312 as duplicate.

Verified bug in installed 3.10.10, 3.11.2, and 3.12.a5.

What do you think about this?

» ./python.exe ex.py
  File "/Users/sobolev/Desktop/cpython/ex.py", line 1
    x = b"Ā"
        ^^^^^
SyntaxError: bytes can only contain ASCII literal characters

This can be achieved with:

Or do we need to highlight the symbol exactly?
If my solution is fine, I can add a test case and send a PR.

@sobolevn

If my solution is fine, I can add a test case and send a PR.

@sobolevn In absence of other feedback, your solution looks better than nothing. So it's worth to create a PR and get it merged before 3.12b1.

Will do :)

pythongh-102310: Change error range for invalid bytes literals

gh-102310: Change error range for invalid bytes literals (#103663)

pythongh-102310: Change error range for invalid bytes literals (pytho…

[3.11] gh-102310: Change error range for invalid bytes literals (GH-1…

Can this be closed?

Edit: Actually, I'm closing this. Re-open in case there's more to it.

Navigation Menu

Repository navigation

(🐞) Column marker in error message off when line contains non ASCII character #102310

Expected

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

(🐞) Column marker in error message off when line contains non ASCII character #102310

Description

Expected

Linked PRs

Activity

arhadthedev commented on Feb 28, 2023

arhadthedev commented on Feb 28, 2023

terryjreedy commented on Feb 28, 2023

terryjreedy commented on Feb 28, 2023

sobolevn commented on Feb 28, 2023

arhadthedev commented on Apr 21, 2023

sobolevn commented on Apr 21, 2023

lysnikolaou commented on Apr 25, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions