Repository navigation
forkserver slow to start because of __main__ preloading issue #137996
Description
Activity
I am new here, is this a good first issue for me to work on?
@emaxx-google can you please share the testcase or the benchmark script.I am new here, is this a good first issue for me to work on?
I don't think so. If you want a good first issue, check for the ones with the
easylabel.Reacted by Krishna Chaitanya, Maksim Ivanov and An LongI have a PR already, but as I'm not deeply familiar with the internals I'm not 100% sure about it.
Reacted by Krishna Chaitanya- added a commit that references this issue
on Aug 28, 2025 Thanks for pointing this out! The implementation change looks almost identical, and you have a regression test already; do you know what's blocking that change from landing?
No worries at all, and yeah, our independent fixes are basically the same, which is nice 🙂
I don't think anything is blocking the change as such, but AIUI the maintainers and core devs in general are overworked and have to prioritise where they spend their time and attention. Before you raised this issue it was a fairly academic fix: it had been broken for many years, seemingly without anyone noticing. It is useful to know it has caused a noticeable performance regression in real-world use!
Reacted by Maksim Ivanov and Gregory P. SmithReacted by Gregory P. Smith
Bug report
Bug description:
When switching a Python project from the default multiprocessing
forkmode toforkserver, I've noticed the processes in the pool got significantly slower to start. Specifyingpreloadmodules didn't help.Debugging internals of
multiprocessingshowed that most of the time (50 ms in each process in my test) is spent in_fixup_main_from_path(data['init_main_from_path']):cpython/Lib/multiprocessing/spawn.py
Line 246 in eae9d7d
This seemed surprising because
__main__was mentioned in thepreloadparameter, however I've noticed thatforkserver.pytries to populate themain_pathparameter fromspawn.get_preparation_data():cpython/Lib/multiprocessing/forkserver.py
Lines 149 to 151 in eae9d7d
However, the latter only writes the path to the
init_main_from_pathparameter:cpython/Lib/multiprocessing/spawn.py
Line 202 in eae9d7d
The end effect is that the
__main__module wasn't preloaded in practice, and every child process had to re-run the main script. Unless I'm missing something, the logic inforkserver.pyneeds to getmain_pathfrom the value ofinit_main_from_path?CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs