Repository navigation
[meta] Freeze commonly used stdlib modules. #89824
Description
| BPO | 45661 |
|---|---|
| Nosy | @gvanrossum, @tiran, @methane, @ericsnowcurrently, @FFY00, @kumaraditya303 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2021-10-28.19:39:40.296>
labels = ['type-bug', 'build', '3.11']
title = '[meta] Freeze commonly used stdlib modules.'
updated_at = <Date 2022-01-07.20:05:59.914>
user = 'https://github.com/ericsnowcurrently'bugs.python.org fields:
activity = <Date 2022-01-07.20:05:59.914>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2021-10-28.19:39:40.296>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 45661
keywords = []
message_count = 7.0
messages = ['405239', '405240', '409947', '409948', '409958', '409959', '410024']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'christian.heimes', 'methane', 'eric.snow', 'FFY00', 'kumaraditya']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45661'
versions = ['Python 3.11']
Activity
ericsnowcurrently commentedon Oct 28, 2021
We're already freezing modules needed for bootstrapping the runtime, as well as the modules imported during startup. [1][2] We're also planning on freezing argparse. [3] There may be other modules (or files) worth freezing (assuming we don't just freeze the entire stdlib, which would have some potential downsides).
This issue is meant to cover the broader idea. The work to actually freeze modules should be handled in separate issues.
[1] https://bugs.python.org/issue45020
[2] https://bugs.python.org/issue45654
[3] https://bugs.python.org/issue45660
ericsnowcurrently commentedon Oct 28, 2021
One to consider that isn't necessarily used that commonly is sysconfig. However, that could also involve freezing the "sysconfigdata" file (without needing to expose it as a module).
kumaraditya303 commentedon Jan 7, 2022
functools and contextlib are very common so they should be deep-frozen. warnings and re would be nice to do.
methane commentedon Jan 7, 2022
I don't against deep freezing functools and contextlib.
But I think we should optimize and utilize zipimport or something similar, because we can not deep-freeze all stdlib or 3rd party libraries.
See also: faster-cpython/ideas#158 (reply in thread)
tiran commentedon Jan 7, 2022
Deep freezing is not a miracle cure for startup performance issues. Every additional frozen module increases the size of the main binary / shared library. This can have negative affects for performance and usability, too. For WASM builds we want to make the wasm binary as small as possible.
kumaraditya303 commentedon Jan 7, 2022
Can you elaborate a bit about "This can have negative affects for performance" as I think freezing modules has only one downside of increasing binary size ?
If WASM would become an officially supported platform then there can be a patch to disable freezing on such platforms otherwise I don't see how because of WASM we cannot deepfreeze for other platforms which benefit from it.
gvanrossum commentedon Jan 7, 2022
Deep-freezing is definitely not a miracle cure. We should continue to trim or delay unneeded imports (like Christian just did for setuptools' _distutils_hack).
What *would* be a miracle cure would be if we could deep-freeze the module contents after it has *executed*. But that is much harder in general, since two executions (even on the same platform) could result in different module contents (e.g. conditionally defining something based on environment contents).
Instagram's Cinder has something that works for this, strict modules (https://github.com/facebookincubator/cinder#strict-modules). It is a complex system! But if we could do this it could really speed up a lot of imports tremendously. (Though still at the cost of binary size increase.)