Prefix search is filesystem-centric

@terryjreedy

BPO	1116520
Nosy	@terryjreedy, @gpshead, @ncoghlan, @carljm, @merwok, @ericsnowcurrently, @zooba, @iselind

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2005-02-04.22:16:00.000>
labels = ['interpreter-core', 'type-feature']
title = 'Prefix search is filesystem-centric'
updated_at = <Date 2019-12-19.00:29:24.734>
user = 'https://bugs.python.org/holdenweb'

bugs.python.org fields:

activity = <Date 2019-12-19.00:29:24.734>
actor = 'brett.cannon'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2005-02-04.22:16:00.000>
creator = 'holdenweb'
dependencies = []
files = []
hgrepos = []
issue_num = 1116520
keywords = []
message_count = 12.0
messages = ['60644', '113379', '113406', '154337', '154370', '222630', '283964', '283965', '284001', '284005', '284006', '284011']
nosy_count = 10.0
nosy_names = ['holdenweb', 'terry.reedy', 'gregory.p.smith', 'ncoghlan', 'carljm', 'eric.araujo', 'eric.snow', 'steve.dower', 'superluser', 'patriki']
pr_nums = []
priority = 'low'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1116520'
versions = ['Python 3.6']

With the introduction of zipimport I experimented to
determine how much of the standard library could be
provided in zip format.

I discovered that I could entirely remove the standard
library, replacing it with /lib/python24.zip, with the
following caveats (this is under Cygwin, where /usr/lib
appears to be a loopback mount of /lib: paths will
differ on other platforms):

The /lib/python2.4/lib-dynload directory had to be
copied to the /lib directory to make zlib available to
zipimport;
The interpreter now produced three error messages:
"Could not find platform independent libraries <prefix>"
"Could not find platform dependent libraries <exec_prefix>"
"Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]"

With the move towards esoteric import mechanisms it
seems that the searches for os.py in the filesystem
might no longer be an appropriate way to start
executing the interpreter.

Should some import hook API be available to determine
whether standard libraries are available without
actually importing anything?

Is this request still relevant for 3.2?

Personally I think it's just as relevant as it always was, particularly with the introduction of importlib, but Brett will have a more informed opinion. I won't be surprised if this issue is closed as wontfix.

Yeah, I'm pretty sure the bootstrap mechanism needs to be able to get hold of os.py directly so it can be injected into the importlib._bootstrap namespace.

However, it may be worth figuring out and documenting the bare minimum that has to exist on the filesystem in order for importlib to get going.

It's even possible that Brett freezes enough into the interpreter binary that the required set has shrunk to zero.

Importlib actually requires no files from disk; everything that is required for importlib are built-in modules or are constants in importlib itself (e.g. os.sep). So technically this should be doable since my bootstrap work freezes importlib itself.

Is this something that we could get into 3.5?

How much smaller would the stdlib for 3.5 become if you compress it with zip?

What about the performance penalty for zipping stdlib? is it significant?

When would you like to zip stdlib? For embedded systems with limited disk
space?

Patrik Iselind

On Sat, Dec 24, 2016 at 6:34 PM, Patrik Iselind <report@bugs.python.org>
wrote:

Patrik Iselind added the comment:

How much smaller would the stdlib for 3.5 become if you compress it with
zip?

----------
nosy: +patriki

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue1116520\>

Originally zip file importing was faster than standard importing from disk because of the fewer stat calls, but importlib caches such things so I don't know if it's still beneficial. As for space savings, I have no idea; you can try zipping the files yourself to find out the space savings.

Is it enough to include everything in the Lib folder, excluding
__pycache__, site-packages and the test folder in Lib? Would that be
representative enough?

Patrik Iselind

Den 2016-12-25 kl. 17:31, skrev Brett Cannon:

Brett Cannon added the comment:

Originally zip file importing was faster than standard importing from disk because of the fewer stat calls, but importlib caches such things so I don't know if it's still beneficial. As for space savings, I have no idea; you can try zipping the files yourself to find out the space savings.

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue1116520\>

Don't forget that the built-in modules may need to be available before the
zipimporter is. A long time ago (when sys.metapath was introduced) I
experimented with imports from non-filesystem sources and that hit me until
I realised what was going on.

S

Steve Holden

On Sun, Dec 25, 2016 at 5:48 PM, Patrik Iselind <report@bugs.python.org>
wrote:

Patrik Iselind added the comment:

Is it enough to include everything in the Lib folder, excluding
__pycache__, site-packages and the test folder in Lib? Would that be
representative enough?

Patrik Iselind

Den 2016-12-25 kl. 17:31, skrev Brett Cannon:
> Brett Cannon added the comment:
>
> Originally zip file importing was faster than standard importing from
disk because of the fewer stat calls, but importlib caches such things so I
don't know if it's still beneficial. As for space savings, I have no idea;
you can try zipping the files yourself to find out the space savings.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue1116520\>
> _______________________________________

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue1116520\>

Note that Steve Dower made some significant changes to sys.path initialisation on Windows in Python 3.6 that could potentially be generalised to other platforms: https://docs.python.org/3/using/windows.html#finding-modules

(There's nothing inherently Windows specific about them, they're mainly useful for the case of embedding CPython inside a larger application)

One of those was to allow pythonXY.zip to be used as a sentinel indicating the location of PYTHONHOME.

The 2005 symptom is gone at tip — re-verified today: a zipped stdlib alongside lib-dynload (the report's setup) now starts with zero warnings.

What changed: pythonXY.zip became a prefix landmark on POSIX in 3.11 (bpo-45582: Port getpath[p].c to Python #29041 — generalizing the Windows sentinel noted above in 2016), so the zip itself anchors the search.
The one remaining warning (zip present but lib-dynload missing) is deliberate — revisited in the March 2026 path-config work (GH-145273: don't skip missing platstdlib warning if stdlib_zip is found #145544) — and can be silenced with -X pathconfig_warnings=0 (new in 3.15, GH-145275: add -X pathconfig_warnings and PYTHON_PATHCONFIG_WARNINGS #145277).
The original questions have answers now: checking whether a stdlib module is available without executing it is importlib.util.find_spec() (plus sys.stdlib_module_names for the canonical list), and the landmark mechanism itself is documented in the sys.path initialization docs these days.

So this looks resolved in practice by later work — suggest closing.

@soreavis

@soreavis thanks for checking!

@soreavis thanks for checking!

Glad I could help

Navigation Menu

Repository navigation

Prefix search is filesystem-centric #41525

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Prefix search is filesystem-centric #41525

Description

Activity

holdenweb commented on Feb 4, 2005

terryjreedy commented on Aug 9, 2010

holdenweb commented on Aug 9, 2010

ncoghlan commented on Feb 26, 2012

brettcannon commented on Feb 26, 2012

BreamoreBoy commented on Jul 9, 2014

iselind commented on Dec 24, 2016

iselind commented on Dec 24, 2016

brettcannon commented on Dec 25, 2016

iselind commented on Dec 25, 2016

holdenweb commented on Dec 25, 2016

ncoghlan commented on Dec 26, 2016

soreavis commented on Jul 19, 2026

brettcannon commented on Jul 20, 2026

soreavis commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions