Bug with the colorizer dependent on tokenizer

Bug report

Bug description:

wrong_str = "
123456
123456
"

It is a SyntaxError. However, in many web page, the 123456 are colored as string.
This question appeared also in REPL. The IDLE had no this question originally, but after using tokenizer to color the codes, it appears.

It may mislead the learners.

Now that the github page has no question on it, but discuss has.

CPython versions tested on:

CPython main branch, 3.14

Operating systems tested on:

Windows

Note that this is just an idea for IDLE, not something that has been implemented.

This is about the REPL; IDLE works according to OP's expectations.

In either case, the code has to be pasted in to demonstrate the problem — otherwise, the interpreter will report a syntax error as soon as the Enter key is pressed after the opening quotation mark.

However, in IDLE the invalid pasted code will show up with invalid highlighting: the intended contents of the string literal have the default colour rather than string-literal colour. (The opening quote will, however, only be highlighted after pressing Enter.)

At the REPL, meanwhile, the pasted code is coloured as if it were a valid string literal.

I'm not sure that this is particularly important, since proper feedback is almost immediate and even delaying that feedback a little bit requires copying and pasting invalid code from elsewhere. However, it's a strange inconsistency.

As for syntax highlighting on web pages, that is obviously outside of CPython's control. Every web page author can make the code appear with whatever styling is desired, even if it's misleading, and there's nothing that CPython could do about that. (As far as I'm aware, the CPython documentation doesn't show invalid string literals like this, except specifically to explain that they are invalid.)

@ambv

While there may be a preference for how syntax erroneous code is colorized, there is no correct way. I believe @ambv has said something similar. There is no violation of even an implied specification. Hence, I do not see a bug here and am removing the bug label.

That said, I cannot reproduce the claimed REPL behavior. After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

PS C:\Users\Terry> py -3.14
Python 3.14.0 (tags/v3.14.0:ebf955d, Oct  7 2025, 10:15:03) [MSC v.1944 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> wrong_str = "
  File "<python-input-0>", line 1
    wrong_str = "
                ^
SyntaxError: unterminated string literal (detected at line 1)
>>> 123456
123456
>>> 123456
123456
>>>  # Trailing " becomes a space.

Ditto in Command Prompt with 3.15 (main). Is there a way to change REPL paste behavior to be more like IDLE's?

After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

May be use "Ctrl + shift + V"?

Another way is here:

Input "wrong_str=''' "
Enter
Finally "↑" and delete the '' at the first line.

However, now the way to draw the code makes this code look "valid". Maybe it is not a good idea.

In fact, there is another problem in it: the syntax label is unstable in this absolute invalid syntax.

For example, I wrote the code, but I want to print a string before to debug it. After I input a " or ' the code after were change its color. This effect is bad.

That said, I cannot reproduce the claimed REPL behavior. After copying the code above and running with the same binary, each line is run separately, as with the old REPL.

On Linux, it is not run till one presses Enter, so I can reproduce the reported colorizing.

It's not just for strings that we have this disparity. For instance, on Linux:

wrong = from x import (
  1 + 1
)

In the REPL, I get (before hitting ENTER):

and once I hit ENTER, I will have a SyntaxError. On the other hand, in IDLE (before hitting ENTER), I get:

When someone C/C in the REPL, it gets colorized independently of whether it's legit code or not, while in IDLE, it gets colorized but not everything. I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE before hitting ENTER (I would say yes since they should only depend on the input lines that could be tokenizied in one go and then colorized).

In some situations, it makes sense to keep highlights (for instance, in my example, if I were to change 1 + 1 to a legitimate import, but keep wrong =, with colorization, I would see that wrong = should be removed but without it, it would be less explicit).

I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE

We are working on that:-) See the PR linked in the issue description.

I don't know if it's possible to have a common implementation for colorizing inputs by REPL/IDLE

We are working on that:-) See the PR linked in the issue description.

I guess you're referring to #140337, now linked from #140347 (not here — until now ;) )?

Apologies I was on mobile and I could not provide a link, but this hyperlink the original post leads me to it?

but after using tokenizer to color the codes, it appears.

Is there anything actionable here? This behavior is intentional: we are not parsing the input as it will be partial (i.e. incomplete, therefore "invalid") in most cases. We don't want to flicker highlighting depending on whether it parsed well or not. Therefore, we are only tokenizing it, which will lead to some minor differences with runtime behavior like in the case of string literals or context-dependent soft keywords (we can only colorize them with limited information, so it won't be 100% correct every time).

The only reason IDLE highlights things differently right now is that it's based on a buncha regexes, which is a thing we actually want to rectify. We will categorically NOT be introducing regexes to colorizing in the REPL.

For example, I wrote the code, but I want to print a string before to debug it. After I input a " or ' the code after were change its color. This effect is bad.

I disagree this is bad. IPython behaves the same, VScode behaves the same.

I disagree this is bad. IPython behaves the same, VScode behaves the same.

No. Both pycharm and vscode won't color the wrong string in this way. It showed in the discussion

To prove it:

There is no qualitative difference between an invalid string being colored as a string versus other tokens. It's just as incorrect for Python.

The reason PyREPL colors it like that is because f-strings can be multiline now if split on an expression and, again, we are not parsing, just tokenizing:

The reason PyREPL colors it like that is because f-strings can be multiline now if split on an expression and, again, we are not parsing, just tokenizing:PyREPL

So that if it is not f-string or t-string, it shouldn't colorize like that.
The f-string and t-string have different analysis with normal string.

Navigation Menu

Repository navigation

Bug with the colorizer dependent on tokenizer #140367

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Bug with the colorizer dependent on tokenizer #140367

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

StanFromIreland commented on Oct 20, 2025

zahlman commented on Oct 20, 2025

terryjreedy commented on Oct 20, 2025

Locked-chess-official commented on Oct 21, 2025

Locked-chess-official commented on Oct 21, 2025

StanFromIreland commented on Oct 21, 2025

picnixz commented on Oct 21, 2025

StanFromIreland commented on Oct 21, 2025

zahlman commented on Oct 21, 2025

StanFromIreland commented on Oct 21, 2025

ambv commented on Oct 22, 2025

Locked-chess-official commented on Oct 22, 2025

ambv commented on Oct 22, 2025

Locked-chess-official commented on Oct 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions