Extended ASCII characters in multiline strings cause "SystemError: Negative size passed to PyUnicode_New" when the encoding is not specified

Bug report

In some cases, when dealing with multi-line string with non-utf8 encoded files, python will throw a SystemError: Negative size passed to PyUnicode_New and not execute any code.

Minimal test case:

print("""
ą""")

This is only a problem if the non-utf8 character lies on a new line (at any point in the line)

A similar test case behaves correctly

print("""ą""")

And reports an encoding warning, which is the expected behavior

SyntaxError: Non-UTF-8 code starting with '\xb1' in file C:\Users\xxxxx\test.py on line 2, but no encoding declared; see https://python.org/dev/peps/pep-0263/ for details

Since this is an encoding related errors, both files are attached (as .txt, GitHub does not allow .py attachments).
test.txt - single line (correct behavior)
test_ml.txt - multi line (bug)

My environment

CPython versions tested on: Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Operating system and architecture: Windows 10 Pro 21H2 (19044.1826)

#96270 may fix this. Let me confirm.

#96270 may fix this. Let me confirm.

It does not fix this.

Since copy-and-paste doesn't usually preserve broken encodings, this is a convenient way to reproduce the bug:

open("x.py", "wb").write(b'print("""\n\xb1""")')

$ python x.py

In tok_get() in "Parser/tokenizer.c", the following code blindly handles EOF returned from tok_nextc() as if it's the end of the file.

cpython/Parser/tokenizer.c

Lines 1936 to 1948 in 6744490

    
           /* Get rest of string */ 
        
           while (end_quote_size != quote_size) { 
        
               c = tok_nextc(tok); 
        
               if (c == EOF || (quote_size == 1 && c == '\n')) { 
        
                   assert(tok->multi_line_start != NULL); 
        
                   // shift the tok_state's location into 
        
                   // the start of string, and report the error 
        
                   // from the initial quote character 
        
                   tok->cur = (char *)tok->start; 
        
                   tok->cur++; 
        
                   tok->line_start = tok->multi_line_start; 
        
                   int start = tok->lineno; 
        
                   tok->lineno = tok->first_lineno;

In this case, however, tok->done is E_DECODE instead of E_EOF. This gets set by error_ret(), which also clears tok->start and tok->cur to NULL. The above code increments tok->cur to 1. Subsequently, _syntaxerror_range() tries to decode the text for the syntax error using the negative size 1 - tok->line_start.

pythongh-96611: Fix error message for invalid UTF-8 in mid-multiline …

gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (…

pythongh-96611: Fix error message for invalid UTF-8 in mid-multiline …

gh-96611: Fix error message for invalid UTF-8 in mid-multiline string (…

@polprog

Thanks for the report, @polprog, and the diagnostics, @eryksun.

Navigation Menu

Repository navigation

Extended ASCII characters in multiline strings cause "SystemError: Negative size passed to PyUnicode_New" when the encoding is not specified #96611

Bug report

My environment

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	/* Get rest of string */
	while (end_quote_size != quote_size) {
	c = tok_nextc(tok);
	if (c == EOF \|\| (quote_size == 1 && c == '\n')) {
	assert(tok->multi_line_start != NULL);
	// shift the tok_state's location into
	// the start of string, and report the error
	// from the initial quote character
	tok->cur = (char *)tok->start;
	tok->cur++;
	tok->line_start = tok->multi_line_start;
	int start = tok->lineno;
	tok->lineno = tok->first_lineno;

Extended ASCII characters in multiline strings cause "SystemError: Negative size passed to PyUnicode_New" when the encoding is not specified #96611

Description

Bug report

My environment

Activity

mdboom commented on Sep 6, 2022

mdboom commented on Sep 6, 2022

mdboom commented on Sep 6, 2022

eryksun commented on Sep 6, 2022

mdboom commented on Sep 7, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions