Repository navigation
DiskCache read errors abort interpreter discovery #187
Description
A damaged or read-only DiskCache raises an uncaught OSError during interpreter discovery. The same interpreter can be found with caching disabled.
Affected code
src/python_discovery/_cached_py_info.py:121 at the reported commit. The reproducer also returns BUG on current main 004294957ed227fe7e52b2864952ba09fd28b220.
Mechanism
The cache lock and JSON read paths let filesystem errors escape the lookup, so discovery never reaches a later candidate or an uncached probe.
Reproduction
Check out the reported commit or current main. Save the files below as named, then build and run:
docker build -t python-discovery-audit -f - . <<'DOCKERFILE'
FROM python:3.15-bookworm
RUN python -m venv /opt/venv
ENV PATH=/opt/venv/bin:$PATH
COPY . /src
WORKDIR /src
ENV SETUPTOOLS_SCM_PRETEND_VERSION=1.9.1
RUN pip install -e . --group test
DOCKERFILE
docker run --rm --network=none --memory=2g -u 0 -w /src \
-v "$PWD/repro:/tmp/repro:ro" -v "$PWD/repro.d:/tmp/repro.d:ro" \
python-discovery-audit /tmp/reprorepro
#!/usr/bin/env python3
import os
import shutil
import subprocess
PYTHON = "/opt/venv/bin/python" # python-discovery installed from /src
OTHER_USER = 65534 # nobody: may write the shared cache, cannot touch /usr/bin/python3.11
CACHE_ROOT = "/tmp/common-cache"
def fresh_shared_cache():
shutil.rmtree(CACHE_ROOT, ignore_errors=True)
os.mkdir(CACHE_ROOT)
os.chown(CACHE_ROOT, OTHER_USER, OTHER_USER)
def other_user_plants_directory(suffix):
# fed on stdin: nobody may not be allowed to read /tmp/repro.d
with open("/tmp/repro.d/plant_directory_in_cache.py") as attack:
subprocess.run(
[PYTHON, "-", CACHE_ROOT, "/usr/bin/python3.11", suffix],
stdin=attack,
user=OTHER_USER,
group=OTHER_USER,
extra_groups=[],
check=True,
)
def root_discovers_python311(label):
# a new process every time: python-discovery also caches in memory
run = subprocess.run(
[PYTHON, "/tmp/repro.d/discover_python.py", CACHE_ROOT], capture_output=True, text=True, timeout=60
)
print(f"===== {label}: exit status {run.returncode}")
print(run.stderr + run.stdout)
return run
# control: a healthy shared cache
fresh_shared_cache()
healthy = root_discovers_python311("nothing planted")
assert "selected: PathPythonInfo(spec=CPython3.11" in healthy.stdout, "setup: python3.11 not found with a healthy cache"
# attack: nobody puts a directory where root's cache entry for /usr/bin/python3.11 goes ...
fresh_shared_cache()
other_user_plants_directory(".json")
entry_planted = root_discovers_python311("directory at <key>.json")
# ... or where its lock file goes
fresh_shared_cache()
other_user_plants_directory(".lock")
lock_planted = root_discovers_python311("directory at <key>.lock")
# check: an unusable cache entry should mean an uncached lookup that still selects python3.11,
# not an exception escaping get_interpreter() from the cache code
is_a_directory = f"IsADirectoryError: [Errno 21] Is a directory: '{CACHE_ROOT}/py_info/4/"
raised = [
run.returncode == 1
and run.stderr.splitlines()[-1].startswith(is_a_directory)
and "in _get_via_file_cache" in run.stderr
for run in (entry_planted, lock_planted)
]
print("get_interpreter() raised IsADirectoryError from _get_via_file_cache: .json", raised[0], "/ .lock", raised[1])
print("BUG" if all(raised) else "OK")repro.d/discover_python.py
import logging
import sys
from pathlib import Path
from python_discovery import DiskCache, get_interpreter
cache_root = Path(sys.argv[1])
logging.basicConfig(format="%(name)s: %(message)s")
logging.getLogger("python_discovery").setLevel(logging.DEBUG) # logs the argv of every interrogation
info = get_interpreter("python3.11", cache=DiskCache(root=cache_root), env={"PATH": "/usr/bin"})
print("selected:", info)repro.d/plant_directory_in_cache.py
import hashlib
import os
import sys
cache_root, interpreter, suffix = sys.argv[1:]
key = hashlib.sha256(interpreter.encode()).hexdigest() # as DiskCache.py_info() names the entry
os.makedirs(os.path.join(cache_root, "py_info", "4", key + suffix))Output on 004294957ed227fe7e52b2864952ba09fd28b220:
===== nothing planted: exit status 0
python_discovery._discovery: find interpreter for spec PythonSpec(major=3, minor=11, free_threaded=False)
python_discovery._cached_py_info: get interpreter info via cmd: /usr/local/bin/python3.15 /src/src/python_discovery/_py_info_collect.py c27f40fac93b98cf7aed8810ed533dc9 86f8a1ec6d48505026078caa90681a1b
python_discovery._cache: wrote python info at /tmp/common-cache/py_info/4/755e5f8e9348854516c9513236b846adb15e9a21490fd9dc391537946af27457.json
python_discovery._compat: filesystem is case-sensitive
python_discovery._discovery: proposed PythonInfo(spec=CPython3.15.2.final.0-64-x86_64, system=/usr/local/bin/python3.15, exe=/opt/venv/bin/python, platform=linux, version='3.15.2 (main, Oct 28 2026, 04:34:17) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
python_discovery._discovery: discover PATH[0]=/usr/bin
python_discovery._cached_py_info: get interpreter info via cmd: /usr/bin/python3.11 /src/src/python_discovery/_py_info_collect.py 9101736350dfa404878cc3bbc9ada429 f5ff0214c71edbf7971ec22f90dd17eb
python_discovery._cache: wrote python info at /tmp/common-cache/py_info/4/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.json
python_discovery._discovery: proposed PathPythonInfo(spec=CPython3.11.2.final.0-64-x86_64, exe=/usr/bin/python3.11, platform=linux, version='3.11.2 (main, Oct 7 2026, 12:35:07) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
python_discovery._discovery: accepted PathPythonInfo(spec=CPython3.11.2.final.0-64-x86_64, exe=/usr/bin/python3.11, platform=linux, version='3.11.2 (main, Oct 7 2026, 12:35:07) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
selected: PathPythonInfo(spec=CPython3.11.2.final.0-64-x86_64, exe=/usr/bin/python3.11, platform=linux, version='3.11.2 (main, Oct 7 2026, 12:35:07) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
===== directory at <key>.json: exit status 1
python_discovery._discovery: find interpreter for spec PythonSpec(major=3, minor=11, free_threaded=False)
python_discovery._cached_py_info: get interpreter info via cmd: /usr/local/bin/python3.15 /src/src/python_discovery/_py_info_collect.py 0d9f13563287a3704ad87bedf4c5a90b baa0d9d6e5b4641579b29953c59a9110
python_discovery._cache: wrote python info at /tmp/common-cache/py_info/4/755e5f8e9348854516c9513236b846adb15e9a21490fd9dc391537946af27457.json
python_discovery._compat: filesystem is case-sensitive
python_discovery._discovery: proposed PythonInfo(spec=CPython3.15.2.final.0-64-x86_64, system=/usr/local/bin/python3.15, exe=/opt/venv/bin/python, platform=linux, version='3.15.2 (main, Oct 28 2026, 04:34:17) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
python_discovery._discovery: discover PATH[0]=/usr/bin
python_discovery._cache: failed to read /tmp/common-cache/py_info/4/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.json
Traceback (most recent call last):
File "/src/src/python_discovery/_cache.py", line 82, in read
data = json.loads(self._file.read_text(encoding="utf-8"))
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.15/pathlib/__init__.py", line 986, in read_text
with self.open(mode='r', encoding=encoding, errors=errors, newline=newline) as f:
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.15/pathlib/__init__.py", line 970, in open
return io.open(self, mode, buffering, encoding, errors, newline)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/tmp/common-cache/py_info/4/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.json'
python_discovery._cached_py_info: get interpreter info via cmd: /usr/bin/python3.11 /src/src/python_discovery/_py_info_collect.py e77b4c002ca25e5e42ad2c5ec91fa8c5 e709fbbe455e8ce9afe30e53bfdafb98
Traceback (most recent call last):
File "/tmp/repro.d/discover_python.py", line 12, in <module>
info = get_interpreter("python3.11", cache=DiskCache(root=cache_root), env={"PATH": "/usr/bin"})
File "/src/src/python_discovery/_discovery.py", line 52, in get_interpreter
if result := _find_interpreter(spec_str, preferred_paths, cache, env, predicate):
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 135, in _find_interpreter
for interpreter, impl_must_match in propose_interpreters(spec, try_first_with, cache, env):
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 203, in propose_interpreters
yield from _propose_from_path(spec, cache, env, tested_exes, all_implementations=all_implementations)
File "/src/src/python_discovery/_discovery.py", line 261, in _propose_from_path
interpreter = PathPythonInfo.from_exe(exe_raw, cache, raise_on_error=False, env=env)
File "/src/src/python_discovery/_py_info.py", line 385, in from_exe
return cls._from_exe_with_args(
~~~~~~~~~~~~~~~~~~~~~~~^
exe,
^^^^
...<5 lines>...
env=env,
^^^^^^^^
)
^
File "/src/src/python_discovery/_py_info.py", line 411, in _from_exe_with_args
proposed = from_exe(
cls,
...<5 lines>...
executable_args=executable_args,
)
File "/src/src/python_discovery/_cached_py_info.py", line 65, in from_exe
result = _get_from_cache(cls, cache, exe, env, ignore_cache=ignore_cache, executable_args=executable_args)
File "/src/src/python_discovery/_cached_py_info.py", line 87, in _get_from_cache
result = _CACHE[key] = _get_via_file_cache(cls, cache, key[0], exe, env, executable_args=executable_args)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_cached_py_info.py", line 138, in _get_via_file_cache
py_info_store.write({**entry_meta, "content": py_info.to_dict()})
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_cache.py", line 97, in write
self._file.write_text(json.dumps(content, sort_keys=True, indent=2), encoding="utf-8")
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.15/pathlib/__init__.py", line 1008, in write_text
with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.15/pathlib/__init__.py", line 970, in open
return io.open(self, mode, buffering, encoding, errors, newline)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/tmp/common-cache/py_info/4/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.json'
===== directory at <key>.lock: exit status 1
python_discovery._discovery: find interpreter for spec PythonSpec(major=3, minor=11, free_threaded=False)
python_discovery._cached_py_info: get interpreter info via cmd: /usr/local/bin/python3.15 /src/src/python_discovery/_py_info_collect.py f56cd71b05cc9ac61e8f8529fb5c2bec 15a98d310f0855617c12f1a5be98c9b8
python_discovery._cache: wrote python info at /tmp/common-cache/py_info/4/755e5f8e9348854516c9513236b846adb15e9a21490fd9dc391537946af27457.json
python_discovery._compat: filesystem is case-sensitive
python_discovery._discovery: proposed PythonInfo(spec=CPython3.15.2.final.0-64-x86_64, system=/usr/local/bin/python3.15, exe=/opt/venv/bin/python, platform=linux, version='3.15.2 (main, Oct 28 2026, 04:34:17) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8)
python_discovery._discovery: discover PATH[0]=/usr/bin
Traceback (most recent call last):
File "/tmp/repro.d/discover_python.py", line 12, in <module>
info = get_interpreter("python3.11", cache=DiskCache(root=cache_root), env={"PATH": "/usr/bin"})
File "/src/src/python_discovery/_discovery.py", line 52, in get_interpreter
if result := _find_interpreter(spec_str, preferred_paths, cache, env, predicate):
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 135, in _find_interpreter
for interpreter, impl_must_match in propose_interpreters(spec, try_first_with, cache, env):
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_discovery.py", line 203, in propose_interpreters
yield from _propose_from_path(spec, cache, env, tested_exes, all_implementations=all_implementations)
File "/src/src/python_discovery/_discovery.py", line 261, in _propose_from_path
interpreter = PathPythonInfo.from_exe(exe_raw, cache, raise_on_error=False, env=env)
File "/src/src/python_discovery/_py_info.py", line 385, in from_exe
return cls._from_exe_with_args(
~~~~~~~~~~~~~~~~~~~~~~~^
exe,
^^^^
...<5 lines>...
env=env,
^^^^^^^^
)
^
File "/src/src/python_discovery/_py_info.py", line 411, in _from_exe_with_args
proposed = from_exe(
cls,
...<5 lines>...
executable_args=executable_args,
)
File "/src/src/python_discovery/_cached_py_info.py", line 65, in from_exe
result = _get_from_cache(cls, cache, exe, env, ignore_cache=ignore_cache, executable_args=executable_args)
File "/src/src/python_discovery/_cached_py_info.py", line 87, in _get_from_cache
result = _CACHE[key] = _get_via_file_cache(cls, cache, key[0], exe, env, executable_args=executable_args)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/src/src/python_discovery/_cached_py_info.py", line 121, in _get_via_file_cache
with py_info_store.locked():
~~~~~~~~~~~~~~~~~~~~^^
File "/usr/local/lib/python3.15/contextlib.py", line 203, in __enter__
return next(self.gen)
File "/src/src/python_discovery/_cache.py", line 111, in locked
with FileLock(str(lock_path)):
~~~~~~~~^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_api.py", line 1087, in __enter__
self.acquire()
~~~~~~~~~~~~^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_api.py", line 1232, in acquire
self._poll_until_acquired(
~~~~~~~~~~~~~~~~~~~~~~~~~^
blocking=blocking,
^^^^^^^^^^^^^^^^^^
...<3 lines>...
start_time=start_time,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/venv/lib/python3.15/site-packages/filelock/_api.py", line 1482, in _poll_until_acquired
self._acquire_with_fork_tracking()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_api.py", line 1544, in _acquire_with_fork_tracking
self._acquire()
~~~~~~~~~~~~~^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_unix.py", line 79, in _acquire
if (missing_flock := self._acquire_native()) is not None:
~~~~~~~~~~~~~~~~~~~~^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_unix.py", line 84, in _acquire_native
if (opened := self._open_lock_file()) is None:
~~~~~~~~~~~~~~~~~~~~^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_unix.py", line 140, in _open_lock_file
return self._open_existing_lock_file(open_flags, open_mode)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.15/site-packages/filelock/_unix.py", line 144, in _open_existing_lock_file
return os.open(self._lock_path, open_flags, open_mode), False
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IsADirectoryError: [Errno 21] Is a directory: '/tmp/common-cache/py_info/4/ca3ed784184f1b3bb7c3539bfb45e71710cd27667424f92c2d5bb4df9c107c23.lock'
get_interpreter() raised IsADirectoryError from _get_via_file_cache: .json True / .lock True
BUG
Expected: Discovery should skip the unusable cache entry and probe the interpreter.
Impact
Reproducibility is moderate because the setup requires a specific candidate or configuration. Exposure is niche: a caller must supply a cache whose files or permissions have changed. The reproduced effect stays within the caller's configured cache, search path, or interpreter.
Credits
Reported by Anthropic's OSS Scanner: ANT-2026-6N6E3MNU.
Activity