Repository navigation
Please restore _Py_InitializeMain or provide an alternate API for it #142417
Description
As discussed in #129033, opening a new issue at the request of @vstinner. See #129033 (comment) for the reasons this API is needed.
In short, a way to run python code and insert a meta importer into sys.meta_path before the "main" phase of initialization runs is needed. This allows, for instance, to bundle all python code as pre-compiled bytecode into a single mmapped file and directly load it from there for improved performance and hygiene.
I am aware that the state of the interpreter after "core" initialization is not entirely specified, however, in practice it is complete enough to write a meta path importer.
This facility has been used for many years by at least two projects that I know of: bypy (link to source code) https://github.com/kovidgoyal/bypy/blob/master/bypy/freeze/bypy-freeze.h#L835 and PyOxidizer: https://github.com/indygreg/PyOxidizer/blob/1ceca8664c71f39e849ce4873e00d821504b32bd/pyembed/src/technotes.rs#L155
I dont much care about this API in particular, but I would like some way to hook into the initialization process after "core" initialization and before "main" initialization.
Some context about use case. bypy is used in the calibre and kitty programs, both of which embed a "frozen" Python interpreter. _Py_InitializeMain() allows that frozen importer to import all its python code, including stdlib code from a single mmaped file contianing pre-compiled bytecode. This has obvious performance benefits especially on windows.
Activity
vstinner commentedon Dec 8, 2025
If this feature is used, we should add a clean C API to use it, rather than just restore the private API to move back to the status quo.
I dislike
config._init_main=0to stop the initialization at the "core" phase, and then having to call_Py_InitializeMain()to "finish" the initialization to the main phase.IMO a simple callback would be enough for bypy and PyOxidizer use cases: it would be called between core and main initialization phases. In practice, it would be called once the Python interpreter is initialized and before the first import. Something like:
@kovidgoyal: Would it fit your use cases?
cc @ncoghlan @ericsnowcurrently
vstinner commentedon Dec 8, 2025
For the 3.14 branch, we can restore the private _Py_InitializeMain() function, but only in Python 3.14.3. It means that Python 3.14.0, 3.14.1 and 3.14.2 cannot be used.
kovidgoyal commentedon Dec 8, 2025
Yes, I can make that work in bypy. It will require a little bit of
re-writing, but no big deal. Far more important for me is to have an
official stable API for this. Whatever shape that API takes I can adapt
to it, provided it allows me to run my callback immediately before the
first import but after the interpreter is initialized. Your proposed
PyConfig_SetInitCallback() would be fine for me.
Personally, I am anyway patching Python 3.14 currently to restore that
function (since I embed private python in my apps, that's easy to do).
So to me it does not matter what happens in 3.14 lifecycle or later. I
cant speak for other projects, of course.
Workaround for Python 3.14 removing _Py_InitializeMain
pythongh-142417: Add PyInitConfig_SetInitCallback() function
pythongh-142417: Add PyInitConfig_SetInitCallback() function
pythongh-142417: Add PyInitConfig_SetInitCallback() function
pythongh-142417: Add PyInitConfig_SetInitCallback() function
pythongh-142417: Add PyInitConfig_SetInitCallback() function
Python 3.14 compat
pythongh-142417: Add Py_InitializeMain() function
pythongh-142417: Add Py_InitializeMain() function
pythongh-142417: Add Py_InitializeMain() function
vstinner commentedon Jan 15, 2026
@encukou and @zooba dislike the callback approach (PR gh-142420), so I wrote PR gh-143877 to add
Py_InitializeMain()function and renamePyConfig._init_maintoPyConfig.init_main.kovidgoyal commentedon Jan 15, 2026
OK, works for me.
zooba commentedon Jan 15, 2026
For the record, I also dislike adding it as a public API, and would prefer to just restore the private one until we've designed initialization properly.
vstinner commentedon Feb 6, 2026
@zooba:
What do you mean by designing initialization properly? IMO the initialization callback is step towards a better initialization design (PR gh-142420).
IMO adding back the private API is step backwards, I would prefer avoid that.
zooba commentedon Feb 6, 2026
"A step" is not the better design. Do the entire design and let's discuss it as a whole, and if we all agree that it's better overall, then we'll implement the whole design.
Until then, we should preserve what people are currently using.
ncoghlan commentedon Feb 23, 2026
I agree with @zooba on this one (unbreak users first, then add the cleaner design after discussing it without time pressure, then potentially remove the legacy private API only after folks consuming that private API have had the chance to add the relevant conditional usage of the new API).
While I think we can probably do something cleaner with a public unstable API that exposes the relevant substeps in a nicer way than the current private API, exposing this stage of the startup as a regular public API seems like a bad idea (I know I proposed that myself way back in PEP 432, but there are a bunch of good reasons we never actually implemented that design, and these days I consider this one of them).
pythongh-142417: Restore private _Py_InitializeMain() function
vstinner commentedon Mar 3, 2026
The majority prefers restoring the private
_Py_InitializeMain()function, rather than my proposed public C APIs: @encukou (comment), @zooba (comment) and @ncoghlan (comment).So I wrote PR gh-145472 to restore the private
_Py_InitializeMain()function in Python 3.15.gh-142417: Restore private _Py_InitializeMain() function (#145472)
vstinner commentedon Mar 4, 2026
The change 31343cf restores
_Py_InitializeMain()function.I close the issue.
kovidgoyal commentedon Mar 4, 2026
Thanks!
encukou commentedon Mar 4, 2026
Well, I see your proposal as a first draft, which now has time to growr. Hopefully we can make things better and not break users.
pythongh-142417: Restore private _Py_InitializeMain() function (pytho…