Repository navigation
Interpreter exits on Windows due to ValueError raised in linecache.py #122170
Description
Activity
I'm surprised that
os.statraises a ValueError in this case actually and not an OSError. I'll make a PR for handling this case in linecache.I'll make a PR for handling this case in linecache.
I have one ready.
Oh I was just going to push it now xD Go for yours then, I'll review it.
Yours is much more complete and polished, is it OK if I close mine?
Err... it's up to you.
I'll close it, I'd rather have code of higher quality in the standard library than code I wrote :)
Reacted by Bénédikt Tran and pan324I'm surprised that
os.statraises a ValueError in this case actually and not an OSError. I'll make a PR for handling this case in linecache.For some reason, code was added to the
path_tconverter to raiseValueErrorif the path length exceeds 32767 characters. The length limit is actually 32766 characters, because the terminating null must be included in the buffer size. Properly, the expressionUNICODE_STRING_MAX_CHARS - 1should be used instead of 32766. But really, apath_tusing function should just let the system API fail and raise anOSError, either for the system error codeERROR_FILENAME_EXCED_RANGEorERROR_PATH_NOT_FOUND, depending on the system API function that's called.Lines 1331 to 1336 in 2c1b1e7
Lines 1398 to 1403 in 2c1b1e7
Reacted by Bénédikt Tran and pan324I get a different error when using the reproducer on Linux:
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/zero/Desktop/Projects/Python/cpython/Lib/runpy.py", line 198, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/zero/Desktop/Projects/Python/cpython/Lib/runpy.py", line 88, in _run_code exec(code, run_globals) File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/__main__.py", line 6, in <module> __pyrepl_interactive_console() File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/main.py", line 57, in interactive_console run_multiline_interactive_console(console) File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/simple_interact.py", line 156, in run_multiline_interactive_console more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single") # type: ignore[call-arg] File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 303, in push more = self.runsource(source, filename, symbol=_symbol) File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/console.py", line 200, in runsource self.runcode(code) File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 95, in runcode self.showtraceback() File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/console.py", line 168, in showtraceback super().showtraceback(colorize=self.can_colorize) File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 147, in showtraceback lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next, colorize=colorize) File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 155, in format_exception return list(te.format(chain=chain, colorize=colorize)) File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 1384, in format yield from _ctx.emit(exc.stack.format(colorize=colorize)) File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 747, in format formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize) File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 583, in format_frame_summary show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors) File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 701, in _should_show_carets statement = tree.body[0] IndexError: list index out of rangeCC @picnixz, this might be a separate issue. This still occurs on both PRs.
I get a different error when using the reproducer on Linux:
[...] File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 701, in _should_show_carets statement = tree.body[0] IndexError: list index out of rangeThat's #122145.
Reacted by Peter Bierma@eryksun I see the ValueError was implemented in #118355, which is recent enough. However, I think that
os.statraising an OSError would be more correct because on Linux, this is what happens:OSError: [Errno 36] File name too long: 'but on Windows, IIRC, it should be a "FileNotFound" (but I have no way to check this now). Do you think I should open another issue foros.stat? (and maybe more functions?)@eryksun I see the ValueError was implemented in #118355, which is recent enough. However, I think that
os.statraising an OSError would be more correct because on Linux, this is what happens:OSError: [Errno 36] File name too long: 'but on Windows, IIRC, it should be a "FileNotFound" (but I have no way to check this now). Do you think I should open another issue foros.stat? (and maybe more functions?)The issue is with the
path_tconverter, not specifically withos.stat(). I'd be in favor of letting the system API call fail and raising whicheverOSErrorexception corresponds to the error code. CENAMETOOLONGisn't used on Windows. It's going to be eitherENOENTorEINVAL, and only the former gets specialized asFileNotFoundError.If long paths are enabled at the system level (via the "LongPathsEnabled" registry value in Windows 10+), then the legacy DOS
MAX_PATH(260) limit is removed, and only native NT limits are enforced. In this case, WinAPICreateFileW()fails withERROR_FILENAME_EXCED_RANGE(206) if the pathname length exceeds the system's pathname length limitUNICODE_STRING_MAX_CHARS(32767, including the terminating null), or fails withERROR_INVALID_NAME(123) if the filename length exceeds the filesystem's filename length limit, which is typically the system's filename length limitMAXIMUM_FILENAME_LENGTH(256, including the terminating null). The C runtime library mapsERROR_FILENAME_EXCED_RANGEto CENOENT(2), and it mapsERROR_INVALID_NAMEto CEINVAL(22). For the sake of consistency, Python follows suit in "PC/errmap.h".If long paths are disabled or not supported (i.e. prior to Windows 10), then pathnames that aren't prefixed by "\\?\" are limited to
MAX_PATH(260), and filenames are typically limited toMAXIMUM_FILENAME_LENGTH. In this case, WinAPICreateFileW()fails withERROR_PATH_NOT_FOUND(3) if the pathname length exceedsMAX_PATHor withERROR_FILENAME_EXCED_RANGEif the pathname length exceedsUNICODE_STRING_MAX_CHARS. Usually in this case the filename length limit doesn't affect the error, assuming the filesystem's limit isMAXIMUM_FILENAME_LENGTH. In the rare case where the pathname of the working directory is only 4 characters (e.g. "C:\" plus the terminating null), then a filename length of exactly 256 characters causes the error to beERROR_INVALID_NAMEinstead ofERROR_PATH_NOT_FOUND. The C runtime library mapsERROR_PATH_NOT_FOUNDto CENOENT, and Python follows suit.I'll file an issue to summarize what you told me (and to make the discussion distinct from this issue). If we were to patch the
path_tconverter, then we would not need the patch for linecache and it could be better IMO (I don't really like to intercept this ValueError that is raised byos.statbecause of a conversion function and not because of the underlying C call).If we were to patch the
path_tconverter, then we would not need the patch for linecache and it could be better IMO (I don't really like to intercept this ValueError that is raised byos.statbecause of a conversion function and not because of the underlying C call).I believe that would also fix two minor related issues I've found (posting here so I don't forget about them):
> $Env:PYTHONPATH="a" * 33000 > py -3.13 Exception ignored in running getpath: Traceback (most recent call last): File "<frozen getpath>", line 668, in <module> OSError: failed to make path absolute Fatal Python error: error evaluating path Python runtime state: core initialized Current thread 0x00005a8c (most recent call first): <no Python frame>
>>> import sys >>> sys.path.insert(0, "a" * 33000) >>> import email Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 1512, in _path_importer_cache KeyError: 'aaaaaaaaaaaaaaaaaaaaaaaa[...]aaaaaaaaaa' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<python-input-6>", line 1, in <module> import email File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1322, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 1262, in _find_spec File "<frozen importlib._bootstrap_external>", line 1555, in find_spec File "<frozen importlib._bootstrap_external>", line 1527, in _get_spec File "<frozen importlib._bootstrap_external>", line 1514, in _path_importer_cache File "<frozen importlib._bootstrap_external>", line 1490, in _path_hooks File "<frozen importlib._bootstrap_external>", line 1714, in path_hook_for_FileFinder File "<frozen importlib._bootstrap_external>", line 173, in _path_isdir File "<frozen importlib._bootstrap_external>", line 158, in _path_is_mode_type File "<frozen importlib._bootstrap_external>", line 152, in _path_stat ValueError: stat: path too long for Windows
Actually, I probably found another bug in linecache where requesting a filename with null bytes makes it fail:
$ ./python -c 'import linecache;linecache.getlines("a\x00bc")' Traceback (most recent call last): File "<string>", line 1, in <module> import linecache;linecache.getlines("a\x00bc") ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/lib/python/cpython/Lib/linecache.py", line 41, in getlines return updatecache(filename, module_globals) File "/lib/python/cpython/Lib/linecache.py", line 100, in updatecache stat = os.stat(fullname) ValueError: stat: embedded null character in path
but linecache is not supposed to fail =/.
(So the PR is still correct, would not be fixed even if we change the
path_tconverter since raising ValueError for NUL bytes seems to be something that was done for ages)Now that the PRs have been merged, I think this specific issue can be closed. It remains to fix the other issue at the level of imports and
sys.path.insert, but I'm not sure it's something related to linecache. For now, I'm closing this issue (though, it remains to merge #122161 so if anyone can do it, I'd be happy) and we should open a new one for the imports related issue (could you do it @devdanzin please?)For the record, while I think it's okay for
linecacheto silently ignore this issue, I don't think that applies generally. Any code that's attempting to use the bad value ought to raiseValueError, and most likely it ought to propagate out to whoever provided the bad value.So I'm not going to insist on reverting anything here, but I do intend to hold up further changes until they've been properly considered.
Reacted by Bénédikt Tran
Bug report
Bug description:
It's possible to make the interpreter exit on Windows in 3.13.0b4 and main due to
ValueError: stat: path too long for Windowsbeing raised byos.stat()inupdatecachein linecache.py when trying to print a traceback:CPython versions tested on:
3.13, CPython main branch
Operating systems tested on:
Windows
Linked PRs
os.statfor too long filename on Windows #122175ValueErrorraised byos.statinlinecache#122176