Python may startup fails if the home variable in pyvenv.cfg is inaccurate

Bug report

Bug description:

The problem comes from getpath.py. When getpath.py cannot find STDLIB_LANDMARKS from executable_dir, it will set prefix to the PREFIX. If the directory set in PREFIX does not exist, an error occurs.

See details:

cpython/Modules/getpath.py

Lines 590 to 599 in 49f15d8

    
           # Detect prefix by searching from our executable location for the stdlib_dir 
        
           if STDLIB_SUBDIR and STDLIB_LANDMARKS and executable_dir and not prefix: 
        
               prefix = search_up(executable_dir, *STDLIB_LANDMARKS) 
        
               if prefix and not stdlib_dir: 
        
                   stdlib_dir = joinpath(prefix, STDLIB_SUBDIR) 
        
           if PREFIX and not prefix: 
        
               prefix = PREFIX 
        
               if not any(isfile(joinpath(prefix, f)) for f in STDLIB_LANDMARKS): 
        
                   warn('Could not find platform independent libraries <prefix>')

How to reproduce

# 1. build cpython with prefix
./configure --prefix=/home/xx/tmp/install_dir
make
make install

# 2. copy install_dir to another
cd /home/xx/tmp
cp -r ./install_dir/ ./copy_dir

# 3. create a symlink from copy_dir/python
ln -s ./copy_dir/bin/python3 ./pylink

# 4. delete original install_dir
rm -rf ./install_dir/

# 5. use python symlink to create a venv
./pylink -m venv ./bad_venv

# 6. failed to start!
./bad_venv/bin/python3

Error message

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './bad_venv/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/home/xx/tmp/install_dir/lib/python3.12'
  sys._base_executable = '/home/xx/tmp/copy_dir/bin/python3.12'
  sys.base_prefix = '/home/xx/tmp/install_dir'
  sys.base_exec_prefix = '/home/xx/tmp/install_dir'
  sys.platlibdir = 'lib'
  sys.executable = '/home/xx/tmp/bad_venv/bin/python3'
  sys.prefix = '/home/xx/tmp/install_dir'
  sys.exec_prefix = '/home/xx/tmp/install_dir'
  sys.path = [
    '/home/xx/tmp/install_dir/lib/python312.zip',
    '/home/xx/tmp/install_dir/lib/python3.12',
    '/home/xx/tmp/install_dir/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

We can find that sys._base_executable here is correct, but sys.base_prefix is wrong.
We could configure base_prefix through the real interpreter path, which is used to search for libraries that Python depends on.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

gh-127440: Try using base_executable to detect prefix and exec_prefix #127441

I don't think it's a good idea to bypass pyvenv.cfg's home for the prefix calculation by looking at base_executable.

That said, I do think the failure mode and error log are pretty bad. We should instead fail with a message explaining that there is a mismatch in the Python installation used to create the virtual environment, and the one being used to run it. We should also suggest using python -m venv --upgrade to fix the issue.

We should also suggest using python -m venv --upgrade to fix the issue.

Thanks for the suggestion :) But python -m venv --upgrade doesn't work for me because the error occurs before the venv is created.

>>> ./pylink -m venv ./bad_venv --upgrade

Error: Command '['/home/xx/tmp/bad_venv/bin/pylink', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Very few people use ./bad_venv/bin/python3 to start python (I use this for debugging), so they don’t even see the error message above.

I'm not trying to bypass home, but provide a way to calculate prefix when the home is inaccurate, just like if PREFIX and not prefix: ... does. Not sure if this has potential side effects, because the tests all passed, please let me know if there are.

Don't you have access to the base Python interpreter? The venv upgrade command is meant to be run with that, not with venv interpreter.

If not, I would like to see if we can figure out other way to deal with your use-case before opting for this approach.

I guess this error has nothing to do with the base interpreter, it comes from:

# Lib/venv/__init__.py
...
def _setup_pip(self, context):
    """Installs or upgrades pip in a virtual environment"""
    self._call_new_python(context, '-m', 'ensurepip', '--upgrade',
                          '--default-pip', stderr=subprocess.STDOUT)

The python used here is the python in the venv (/venv/bin/python).

>>> python3 -VV
Python 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]

>>> which python3
/usr/bin/python3

>>> python3 -m venv --upgrade ./bad_venv/
Error: Command '['/home/xx/tmp/bad_venv/bin/python3', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

I think we should treat this as a venv bug, it should check home and update it if necessary.

@FFY00

I found that the doc for venv describes home like this:

python -m venv /path/to/new/virtual/environment
This creates the target directory (including parent directories as needed) and places a pyvenv.cfg file in it with a home key pointing to the Python installation from which the command was run.

It seems that the doc assumes that the location to run python command is always in the Python installation directory (Normally, this is correct).
But now the home does not always point to the Python installation, it may point to any directory containing a python symlink used to create a venv. I'm not sure if this is what the designers of venv intended.

Now we seem to expect home to always point to the real Python installation. Can anyone else help us identify the solution? @FFY00

After looking into this a bit more, I think your proposed solution is reasonable. To fix the root of the issue, though, I opened GH-127895. We can apply your solution to 3.12 and 3.13 as a bugfix, and move away from using home entirely in 3.14.

Navigation Menu

Repository navigation

Python may startup fails if the `home` variable in `pyvenv.cfg` is inaccurate #127440

Bug report

Bug description:

How to reproduce

Error message

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	# Detect prefix by searching from our executable location for the stdlib_dir
	if STDLIB_SUBDIR and STDLIB_LANDMARKS and executable_dir and not prefix:
	prefix = search_up(executable_dir, *STDLIB_LANDMARKS)
	if prefix and not stdlib_dir:
	stdlib_dir = joinpath(prefix, STDLIB_SUBDIR)

	if PREFIX and not prefix:
	prefix = PREFIX
	if not any(isfile(joinpath(prefix, f)) for f in STDLIB_LANDMARKS):
	warn('Could not find platform independent libraries <prefix>')

Python may startup fails if the home variable in pyvenv.cfg is inaccurate #127440

Description

Bug report

Bug description:

How to reproduce

Error message

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

FFY00 commented on Dec 4, 2024

CharlieZhao95 commented on Dec 5, 2024

FFY00 commented on Dec 5, 2024

CharlieZhao95 commented on Dec 5, 2024

FFY00 commented on Dec 5, 2024

CharlieZhao95 commented on Dec 9, 2024

FFY00 commented on Dec 12, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Python may startup fails if the `home` variable in `pyvenv.cfg` is inaccurate #127440