Skip to content

site.py is no longer read at startup time (as it is a deep-frozen module) #107344

Open
Open
site.py is no longer read at startup time (as it is a deep-frozen module)#107344
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@pelson
pelson
opened on Jul 27, 2023

Bug report

Before https://github.com/python/cpython/pull/29118/files#diff-1f0a8db227d22005511b0d90f5339b97db345917b863954b3b3ccb9ec308767cR873 it was possible to modify site.py and control the behaviour of Python at a site level. Indeed, the comments in site.py reflect this (e.g. set it to False to disable the feature or True to force the feature) (also in the docs https://docs.python.org/3/library/site.html#site.ENABLE_USER_SITE, esp "None means it was disabled for security reasons ... or by an administrator.").

The change was introduced in order to optimise the startup time. It would be retrograde to stop doing that, but at the same time, having .py files in the shipped distribution (stdlib) which are not used is confusing and wasteful. At a minimum we could no longer ship frozen .py files (since they aren't used), thereby saving disk space and avoiding confusion for people who might edit those files.

If the startup time optimisation is a result of avoiding stat-ing files, then it would be great if there was a convenient way to re-compile the site.py frozen module (so that we can still customize the site). If it isn't such a concern to stat, the alternative would be to look to see if there is a site.py file, and if so, use that instead of the frozen one (assuming that the site.py is removed from the distribution as suggested above).

Above all, the desire is to be able to configure the site of a distribution after compilation, without using tricks such as sitecustomize, activations scripts, or .pth files.

Your environment

I have reproduced this on Python 3.11.4 and main (6e850c3), on Linux.

Relates to #89824.

Activity

  1. pelson
    added
    type-bugAn unexpected behavior, bug, or error
    on Jul 27, 2023

  2. njsmith commented on Jul 28, 2023

    @njsmith
    njsmith
    on Jul 28, 2023
    Contributor

    Can you elaborate on why sitecustomize doesn't work for you? Is the issue that it's missing features, or something else?

  3. pelson commented on Jul 28, 2023

    @pelson
    pelson
    on Jul 28, 2023
    ContributorAuthor

    Can you elaborate on why sitecustomize doesn't work for you? Is the issue that it's missing features, or something else?

    Mostly sitecustomize can work, but a few reasons that it is more challenging/impossible:

    • It is too late to disable user site-packages at sitecustomize (your issue It should be possible for a Python environment to disable the "user site" functionality without patching the stdlib #99312)
    • There is already third-party package code which has been executed by the time sitecustomize is run (via .pth), and I need to influence how all code is executed (e.g. setting default environment variables before anything else)
    • I want to be able to add additional site-packages directories, with higher precedence than user site-packages (an prior processing in e.g. .pth files should follow that precedence)
    • It would be nice if the consumers of my built Python distribution (which they can install themselves) could configure the distribution using the standard sitecustomize mechanism, but this isn't possible if I use sitecustomize for controlling the distribution's behaviour itself. Essentially, I am a Python distributor (i.e. I provide an installer), not a Python distribution provider, and at the same time, I am re-using an existing build, not building from source (so can't patch site).

    To be clear, my biggest concern is that site.py is shipped (in the stdlib), but is entirely ignored. I would find it more appropriate to either not ship site.py at all, or to provide a mechanism that allows overriding a frozen module (though I acknowledge that this implies more failed file stats in most cases, which might invalidate some of the benefit of the change in the first place?)

  4. iritkatriel
    added
    stdlibStandard Library Python modules in the Lib/ directory
    on Dec 1, 2023

  5. morotti commented on Jan 22, 2025

    @morotti
    morotti
    on Jan 22, 2025
    Contributor

    I unfortunately come across this issue. massive outage when some service accounts got a ~/.local/lib/python3.11/site-packages/... populating by something.

    the site.py was edited to set site.ENABLE_USER_SITE = False to prevent exactly this issue from happening.
    but that stopped working as per this bug ticket :(

    I went through the bug tracker and I haven't seen any mention of a fix or any workaround to address the issue.
    The alternative I could come up is to add this snippet to the top of your sitecustomize.py, for lack of a better solution.

    However it is not equivalent to the previous solution of editing site.py, because the sitecustomize.py is loaded after the import paths have been configured to load from the user site.
    the sitecustomize.py cannot prevent from loading packages from the user sites, it's called too late.

    import site
import sys


# ~/.local/lib/python3.8/site-packages
# disable user site otherwise it takes precedence over packages in the venv.
# it was meant to be disabled by editing the site.py file, but that stopped working.
# https://github.com/python/cpython/issues/107344

site.ENABLE_USER_SITE = False
user_site = site.getusersitepackages()
if user_site in sys.path:
    sys.path.remove(user_site)

    You can run the interpreter with python -v and see that imports are still contaminated by the .local directory.

    Processing user site-packages
Adding directory: '/root/.local/lib/python3.11/site-packages'
Processing global site-packages
...
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.cpython-311-x86_64-linux-gnu.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.abi3.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.so
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.py
# trying /root/.local/lib/python3.11/site-packages/_distutils_hack.pyc
...
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.cpython-311-x86_64-linux-gnu.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.abi3.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.so
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.py
# trying /root/.local/lib/python3.11/site-packages/sitecustomize.pyc

  6. pelson commented on Jan 23, 2025

    @pelson
    pelson
    on Jan 23, 2025
    ContributorAuthor

    The original change came from #28398. Perhaps @ericsnowcurrently could help us determine if there is something we can do to conditionally run site.py (not the frozen version) during startup only if it is changed?

  7. FFY00
    pinned this issue on May 11, 2026

  8. StanFromIreland commented on May 11, 2026

    @StanFromIreland
    StanFromIreland
    on May 11, 2026
    Member

    @FFY00 FYI pinning this issue pins it for everyone, as such it shows up here:

    Image

    I presume this wasn't what you intended, so I'll unpin it.

  9. StanFromIreland
    unpinned this issue on May 11, 2026

  10. FFY00 commented on May 19, 2026

    @FFY00
    FFY00
    on May 19, 2026
    Member

    Thanks! I am pretty sure I accidentally pinned it on mobile when trying to subscribe 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions