Python 3.12 tokenize generates invalid locations for f'\N{unicode}'

Bug report

Bug description:

from tokenize import untokenize, generate_tokens
from io import StringIO

untokenize(generate_tokens(StringIO("f'\\N{EXCLAMATION MARK}'").readline))

ValueError: start (1,22) precedes previous end (1,24)

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

@pablogsal

cc @pablogsal

The tokens parsing f'\\N{EXCLAMATION MARK}' and traceback unparsing the list (toks) are

[TokenInfo(type=59 (FSTRING_START), string="f'", start=(1, 0), end=(1, 2), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=60 (FSTRING_MIDDLE), string='\\N{EXCLAMATION MARK}', start=(1, 2), end=(1, 22),
   line="f'\\N{EXCLAMATION MARK}'"), 
TokenInfo(type=61 (FSTRING_END), string="'", start=(1, 22), end=(1, 23), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=4 (NEWLINE), string='', start=(1, 23), end=(1, 24), line="f'\\N{EXCLAMATION MARK}'"),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
#
Traceback (most recent call last):
  File "F:\dev\tem\tem2.py", line 6, in <module>
    print(untokenize(toks))
  File "C:\Programs\Python313\Lib\tokenize.py", line 294, in untokenize
    out = ut.untokenize(iterable)
  File "C:\Programs\Python313\Lib\tokenize.py", line 223, in untokenize
    self.add_whitespace(start)
  File "C:\Programs\Python313\Lib\tokenize.py", line 176, in add_whitespace
    raise ValueError("start ({},{}) precedes previous end ({},{})"
ValueError: start (1,22) precedes previous end (1,24)

The problem is that when Untokenizer.untokenize, line215 sees the FSTRING_MIDDLE token, it replaces '{' and '}' with '{{' and '}}' and bumps the end position by 2, making the end column 2 more than the next start column. This works when the presence of curly brackets results from the reverse process, but not when the lexer recognizes \N{name} as a unicode named literal without replacing it with the indicated character. Other escapes are resolved, as with '\ueeee' being tokenized as a single character. Unless the tokenizer replaces \N{name} with a character, the untokenizer must recognize it also and not do the replacement.

@isidentical

CC: @isidentical

pythongh-115154: Fix untokenize handling of unicode named literals

fixup! pythongh-115154: Fix untokenize handling of unicode named lite…

gh-115154: Fix untokenize handling of unicode named literals (#115171)

pythongh-115154: Fix untokenize handling of unicode named literals (p…

[3.12] gh-115154: Fix untokenize handling of unicode named literals (G…

pythongh-115154: Fix untokenize handling of unicode named literals (p…

Navigation Menu

Repository navigation

Python 3.12 tokenize generates invalid locations for f'\N{unicode}' #115154

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python 3.12 tokenize generates invalid locations for f'\N{unicode}' #115154

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

Eclips4 commented on Feb 8, 2024

terryjreedy commented on Feb 8, 2024

pablogsal commented on Feb 8, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions