Repository navigation
sys.executable can be not normalized #75213
Description
| BPO | 31030 |
|---|---|
| Nosy | @terryjreedy, @vstinner, @serhiy-storchaka, @eryksun |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2017-07-25.12:41:50.673>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10', 'tests']
title = 'sys.executable can be not normalized'
updated_at = <Date 2021-02-27.20:06:55.248>
user = 'https://github.com/serhiy-storchaka'bugs.python.org fields:
activity = <Date 2021-02-27.20:06:55.248>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Tests']
creation = <Date 2017-07-25.12:41:50.673>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 31030
keywords = []
message_count = 8.0
messages = ['299065', '299067', '299068', '299444', '299456', '387775', '387782', '387787']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'serhiy.storchaka', 'eryksun']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31030'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']
Activity
serhiy-storchaka commentedon Jul 25, 2017
test_executable in Lib/test/test_sys.py tests that sys.executable is absolute. But this test is too strong, it fails if sys.executable is not normalized.
test_sys failed
1 test failed:
test_sys
Total duration: 2 sec
Tests result: FAILURE
I don't know what is wrong: the value of sys.executable or the test.
serhiy-storchaka commentedon Jul 25, 2017
test_venv fails for the same cause.
test_venv failed
1 test failed:
test_venv
Total duration: 166 ms
Tests result: FAILURE
vstinner commentedon Jul 25, 2017
Technically, /home/serhiy/py/cpython/Lib/../python is absolute and is a valid path. I'm not sure that it's possible to normalize sys.executable early in the Python initialization.
If we modify the site module to normalize sys.executable, it would mean that sys.executable wouldn't be normalized when using the python3 -S command line option. I dislike having subtle differences between site and no site.
I suggest to normalize sys.executable in the unit test.
terryjreedy commentedon Jul 29, 2017
On Windows, sys.executable is normalized before or during startup. C:\Programs\Python36>Lib\..\python -m test test_sys prints
test_executable (main.SysModuleTest) ... ok
C:\Programs\Python36>Lib\..\python -c "import sys; print(sys.executable)"
C:\Programs\Python36\python.exe
eryksun commentedon Jul 29, 2017
Terry, the Windows implementation calls GetModuleFileNameW to get the executable's fully-qualified path from the loader. However, depending on how Python is started, this could be a \\?\ path, which leads to a bug in the startup code that determines the prefix path based on the landmark "lib\os.py". Probably the easiest solution would be to normalize the executable path to remove the \\?\ or \\.\ prefix. This prefix isn't useful here anyway since Windows doesn't properly support running programs from long paths, not even in Windows 10. I'll open a new issue for this.
eryksun commentedon Feb 27, 2021
In Python 3.10 in POSIX, it's still the case that executable, prefix, exec_prefix, base_prefix, and base_exec_prefix in the sys module do not get normalized. For example, in Linux:
$ .local/bin/../bin/python3.10 -c "import sys; print(sys.executable)" /home/someone/.local/bin/../bin/python3.10In test/test_sys.py, assertEqual(os.path.abspath(sys.executable), sys.executable) fails. In test/test_venv.py, assertIn('home = %s' % path, data) and assertEqual(out.strip(), expected.encode()) both fail, respectively in test_defaults and test_prefixes.
In Windows, prior to Python 3.6, this can be a problem if Python is run via CreateProcessW using a non-normalized path in lpApplicationName (rare, but possible) instead of letting the system search for the executable in lpCommandLine. For example:
>>> cmd = 'python -c "import sys; print(sys.executable)"' >>> exe = r'C:\Program Files\Python35\..\Python35\python.exe' >>> subprocess.call(cmd, executable=exe) C:\Program Files\Python35\..\Python35\python.exe 0It's no longer an issue in Windows with Python 3.6 and above. When getting the program path while initializing, the GetModuleFileNameW(NULL, ..) result gets canonicalized (e.g. via PathCchCanonicalizeEx). For example:
>>> exe = r'C:\Program Files\Python36\..\Python36\python.exe' >>> subprocess.call(cmd, executable=exe) C:\Program Files\Python36\python.exe 0terryjreedy commentedon Feb 27, 2021
This issue was opened with 2 related facts of 2017:
a) sys.executable is not normalized
b) as a result, two tests failed
Serhiy then said "I don't know what is wrong: the value of sys.executable or the test." If it was the tests, they seem to have been fixed elsewhere.
If it is the non-normalization of sys.executable, my concern for IDLE is that it starts the execution subprocess with subprocess.Popen([sys.executable, ...]). Could this be a reason for mysterious IDLE failures?
Eryk, you stated above that on Windows a \\?\ path is possible but will not work for startup, and that you would open an issue for this. Have you?
Do any of the non-normalized Linux (or *nix) executable paths fail for startup?
eryksun commentedon Feb 27, 2021
The cited tests haven't been changed to work with non-normalized sys.executable, sys.prefix, etc. But it's not common to run the test suite with a non-normalized path such as "Lib/../build/python".
Sorry, I don't remember the details of the bug that I mentioned. But it became a non-issue in 3.6+, which canonicalizes the executable path. For example:
>>> exe = r'\\?\C:\Program Files\Python36\python.exe' >>> cmd = 'python -c "import sys; print(sys.executable)"' >>> subprocess.call(cmd, executable=exe) C:\Program Files\Python36\python.exe 0Compare the latter to the output of GetModuleFileNameW(NULL, ...) in this case, which returns a \\?\ extended (verbatim) path:
>>> exe = r'\\?\C:\Program Files\Python36\python.exe' >>> cmd = 'python -c "import _winapi; print(_winapi.GetModuleFileName(0))"' >>> subprocess.call(cmd, executable=exe) \\?\C:\Program Files\Python36\python.exe 0Not that I'm aware of. AFAIK, it's just about the particular tests failing in this case.
iritkatriel commentedon May 23, 2024
I think that was fixed by now. On main I see