untokenize of specially crafted escaped characters does not round trip properly

Bug report

Bug description:

this small program does not roundtrip through tokenize / untokenize -- it appears to mishandle the escaped quote as a \N{NAMED ESCAPE}

bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

this is what it produces after a round of untokenization:

$ python3 t.py t3.py 
bar = 1
print(f"{bar} \"{ SNOWMAN}} {{foo}}")

annoyingly, tokenize_rt suffers from a different related bug which is why I was investigating this to begin with. an aside, the handling of curly braces in 3.12+ tokenization is a huge pain!

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

@pablogsal

cc @pablogsal ecf16ee

Sorry is a bit late here and I had a long day, but what I am missing here:

>>> import tokenize
>>> import io
>>> code = r'f"{bar} \"{{SNOWMAN}} {{foo}}"'
>>> tokens = list(tokenize.generate_tokens(io.StringIO(code).readline))
>>> tokenize.untokenize(tokens)
'f"{bar} \\"{{SNOWMAN}} {{foo}}"'
>>> code
'f"{bar} \\"{{SNOWMAN}} {{foo}}"'

Same with a file:

$ cat lel.py
bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

$ ./python.exe
Python 3.14.0a1+ (heads/main-dirty:03f9264ecef, Oct 22 2024, 09:15:42) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> data = open('lel.py')
>>> tokens = list(tokenize.generate_tokens(data.readline))
>>> print(tokenize.untokenize(tokens))
bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

Sorry if I am missing anything obvious. Can you provide a small reproducer I can run?

Ah this was fixed in db23b8b in main and was back ported to 3.12 but not yet released:

#125021

Closing as duplicate of #125008

hah yep -- just checked my vod and I spent all the time waiting on a build only to rerun my system python 🤦

Navigation Menu

Repository navigation

untokenize of specially crafted escaped characters does not round trip properly #125821

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

untokenize of specially crafted escaped characters does not round trip properly #125821

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

asottile commented on Oct 21, 2024

pablogsal commented on Oct 22, 2024

pablogsal commented on Oct 22, 2024

pablogsal commented on Oct 22, 2024

asottile commented on Oct 22, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions