Repository navigation
Non-UTF-8 interpreter paths break DiskCache lookups #198
Description
An interpreter path containing non-UTF-8 bytes makes a DiskCache lookup raise UnicodeEncodeError. The same candidate works without DiskCache.
Affected code
src/python_discovery/_cache.py:135 at the reported commit. The reproducer also returns BUG on current main 004294957ed227fe7e52b2864952ba09fd28b220.
Mechanism
The cache key hashes the path after a strict UTF-8 encode, before the interpreter can be probed or skipped.
Reproduction
Check out the reported commit or current main. Save the files below as named, then build and run:
docker build -t python-discovery-audit -f - . <<'DOCKERFILE'
FROM python:3.15-bookworm
RUN python -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
COPY . /src
WORKDIR /src
ENV SETUPTOOLS_SCM_PRETEND_VERSION=1.9.1
RUN pip install -e . --group test
DOCKERFILE
docker run --rm --network=none --memory=2g -u 0 -w /src \
-v "$PWD/repro:/tmp/repro:ro" -v "$PWD/repro.d:/tmp/repro.d:ro" \
python-discovery-audit /tmp/reprorepro
#!/usr/bin/env python3
import os
import shutil
import subprocess
import sys
import tempfile
# set up: the ~/.local/bin of a home directory named in Latin-1, holding a working python3.11
assert os.path.exists("/usr/bin/python3.11"), "the candidate interpreter /usr/bin/python3.11 is missing"
work = tempfile.mkdtemp()
local_bin = os.fsencode(work) + b"/Andr\xe9/.local/bin" # a lone 0xe9 byte is not valid UTF-8
os.makedirs(local_bin)
os.symlink("/usr/bin/python3.11", local_bin + b"/python3.11")
candidate = os.fsdecode(local_bin) + "/python3.11" # as os.environ and os.listdir spell it: 0xe9 -> '\udce9'
search_path = os.fsdecode(local_bin) + ":/usr/bin"
cache_dir = os.path.join(work, "cache")
print("PATH:", ascii(search_path))
# run: the same search with a DiskCache and without one
find_python311 = [sys.executable, "/tmp/repro.d/find_python311.py", search_path]
with_cache = subprocess.run([*find_python311, cache_dir], capture_output=True, text=True)
without_cache = subprocess.run(find_python311, capture_output=True, text=True)
shutil.rmtree(work)
print(f"--- with DiskCache (exit {with_cache.returncode}):\n{with_cache.stdout}{with_cache.stderr}")
print(f"--- without cache (exit {without_cache.returncode}):\n{without_cache.stdout}{without_cache.stderr}")
# check: without a cache the candidate is probed and selected; with one, discovery must not crash on its path
selected_without_cache = f"selected: {ascii(candidate)}" in without_cache.stdout
crashed_hashing_cache_key = (
"UnicodeEncodeError: 'utf-8' codec can't encode character '\\udce9'" in with_cache.stderr
and ", in py_info\n" in with_cache.stderr
)
print("BUG" if selected_without_cache and crashed_hashing_cache_key else "OK")repro.d/find_python311.py
import sys
from pathlib import Path
from python_discovery import DiskCache, get_interpreter
def print_spawned_argv(event, args):
if event == "subprocess.Popen":
print("spawned:", ascii(args[1]))
search_path = sys.argv[1]
cache = DiskCache(root=Path(sys.argv[2])) if len(sys.argv) > 2 else None
sys.addaudithook(print_spawned_argv)
result = get_interpreter("3.11", env={"PATH": search_path}, cache=cache)
print("selected:", ascii(result.executable), "version", ".".join(map(str, result.version_info[:3])))Output on 004294957ed227fe7e52b2864952ba09fd28b220:
PATH: '/tmp/tmpt7uelg3r/Andr\udce9/.local/bin:/usr/bin'
--- with DiskCache (exit 1):
spawned: ['/usr/local/bin/python3.15', '/src/src/python_discovery/_py_info_collect.py', '9bf28a0b57fd532deac863f8534a99be', '14f7f960d6b4deef0239cc99f9cc7d99']
Traceback (most recent call last):
File "/tmp/repro.d/find_python311.py", line 16, in <module>
result = get_interpreter("3.11", env={"PATH": search_path}, cache=cache)
File "/src/src/python_discovery/_discovery.py", line 52, in get_interpreter
if result := _find_interpreter(spec_str, preferred_paths, cache, env, predicate):
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 135, in _find_interpreter
for interpreter, impl_must_match in propose_interpreters(spec, try_first_with, cache, env):
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 203, in propose_interpreters
yield from _propose_from_path(spec, cache, env, tested_exes, all_implementations=all_implementations)
File "/src/src/python_discovery/_discovery.py", line 261, in _propose_from_path
interpreter = PathPythonInfo.from_exe(exe_raw, cache, raise_on_error=False, env=env)
File "/src/src/python_discovery/_py_info.py", line 385, in from_exe
return cls._from_exe_with_args(
~~~~~~~~~~~~~~~~~~~~~~~^
exe,
^^^^
...<5 lines>...
env=env,
^^^^^^^^
)
^
File "/src/src/python_discovery/_py_info.py", line 411, in _from_exe_with_args
proposed = from_exe(
cls,
...<5 lines>...
executable_args=executable_args,
)
File "/src/src/python_discovery/_cached_py_info.py", line 65, in from_exe
result = _get_from_cache(cls, cache, exe, env, ignore_cache=ignore_cache, executable_args=executable_args)
File "/src/src/python_discovery/_cached_py_info.py", line 87, in _get_from_cache
result = _CACHE[key] = _get_via_file_cache(cls, cache, key[0], exe, env, executable_args=executable_args)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_cached_py_info.py", line 114, in _get_via_file_cache
py_info_store = resolved_cache.py_info(path)
File "/src/src/python_discovery/_cache.py", line 135, in py_info
key = sha256(str(path).encode("utf-8")).hexdigest()
~~~~~~~~~~~~~~~~^^^^^^^^^
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 21: surrogates not allowed
--- without cache (exit 0):
spawned: ['/usr/local/bin/python3.15', '/src/src/python_discovery/_py_info_collect.py', '0c6ce1ec1fb66174f1f0cd53ccf893ac', '4c9244727f9c6f18abb6c37d157f7f73']
spawned: ['/tmp/tmpt7uelg3r/Andr\udce9/.local/bin/python3.11', '/src/src/python_discovery/_py_info_collect.py', '5c792c8b15a9abdf6daff8d484c3cb04', '991e8c0bbb0cde6a9b8e82a54856d8ae']
spawned: ['/usr/bin/python3.11', '/src/src/python_discovery/_py_info_collect.py', 'c86dbdb2c788ccdf05d348f72f4d55e1', 'be106ca36f9f1f81e65706bc5b22f6eb']
selected: '/tmp/tmpt7uelg3r/Andr\udce9/.local/bin/python3.11' version 3.11.2
BUG
Expected: A path's filesystem bytes should be handled without aborting discovery.
Impact
Reproducibility is moderate because the setup requires a specific candidate or configuration. Exposure is unlikely: the filesystem must contain a path with bytes that are invalid UTF-8. The reproduced effect stays within the caller's configured cache, search path, or interpreter.
Credits
Reported by Anthropic's OSS Scanner: ANT-2026-VPI28HMC.
Activity