Repository navigation
multiprocessing.Pool fails if command has a long list of arguments (regression in 3.14.3, 3.13.12) #144503
Description
Bug report
Bug description:
A multiprocessing pool can fail to start when the command that starts the pool was ran with a lot of arguments.
Minimal working example:
# mwe.py
import multiprocessing
if __name__ == "__main__":
with multiprocessing.Pool():
passRun this using the following script:
#!/bin/bash
PAGE_SIZE=$(getconf PAGE_SIZE)
MAX_ARG_STRLEN=$((32 * PAGE_SIZE))
COUNT=$((MAX_ARG_STRLEN / 100 ))
echo $COUNT
yes "$(printf 'a%.0s' {1..100})" | head -n $COUNT | xargs python3 mwe.py
What this will do is effectively run the following:
python3 mwe.py aaa..a aaa..a aaa..a ... aaa.a
The Python process itself launches correctly, as each argument is only 100 bytes long (way under the maximum argument length of MAX_ARG_STRLEN).
The creation of the multiprocessing pool however fails, because the sys.argv variable is taken, and used to build a single string in multiprocessing.forkserver.ForkServer.ensure_running here:
cpython/Lib/multiprocessing/forkserver.py
Line 175 in d5cb9f6
This is caused by a recent change: #143717, which was also backported to 3.13 and 3.14. (It works on 3.14.2, fails on 3.14.3).
CPython versions tested on:
3.15, 3.14
Operating systems tested on:
Linux
Linked PRs
- gh-144503: Pass
sys.argvvia a route less limited by single command line argument length #144508 - gh-144503: Pass sys.argv to forkserver as real argv elements #148194
- [3.14] gh-144503: Pass sys.argv to forkserver as real argv elements (GH-148194) #148195
- [3.13] gh-144503: Pass sys.argv to forkserver as real argv elements (GH-148194) #148196
Activity
ZeroIntensity commentedon Feb 5, 2026
cc @gpshead
pythongh-144503: Pass `sys.argv` as separate command line arguments.
JaapJoris commentedon Feb 5, 2026
Confirmed; when I run the minimal example under Python 3.14.3 I get this traceback:
pythongh-144503: Accept non-importable resource
pythongh-144503: Add news entry
pythongh-144503: Accept non-importable resource
duaneg commentedon Feb 8, 2026
This is tricky. We need to pass
sys.argvdata, but the current approach obviously won't work given the need to support largeargvs. I think we need to rework how forkserver'smainworks, to read initialisation data from a pipe rather than providing it inline as part of the python command.@gpshead (and/or others) does that sound right?
sjoerdjob commentedon Feb 9, 2026
Writing initialization data to a pipe instead of providing it inline does indeed sound like a better idea, but it is also a bit more complex idea.
The maximum number of bytes to be written to a pipe before the other end reads from it is also limited; based on https://www.man7.org/linux/man-pages/man7/pipe.7.html, one can expect a default capacity of 16 pages, or 65356 bytes. That is less than the limit of 32 pages that we currently run into.
As such, writing to and reading from the initialization pipe would have to have buffering applied, which is certainly feasible, but it does introduce more failure modes (e.g.: if the forkserver process won't launch, the writes will block, switching the symptom from 'exception' to 'crash').
Also: what data format to use? Using pickles would make sense given the rest of the multiprocessing also works with pickles, however pickles bring with them security risks. To do that safely, gh-97514 introduced authentication keys. Using JSON instead would eliminate at least that (potential) safety concern. Likewise, using
repr(some_dict)/ast.literal_eval(some_dict)would also work.Looking at the underlying assumptions of the authkey pipe, being that only the main process can write to it, and only the forkserver process can read from it, this should in itself be a secure channel, and any data format should be usable.
The current system uses
%r(effectivelyrepr(...)), so the assumption definitely is that all items can by round-tripped correctly viareprand parsing. Effectively, however, all values are strings or lists of strings, and usingjsonis I think more natural these days.I'll try and see if I can get it working by sending the initialization data over the pipe instead of via the CLI arguments.
pythongh-144503: Send initialization data for preload over pipe inste…
13 remaining items
gpshead commentedon Apr 7, 2026
I put up an alternate PR #148194 that does this in a simpler manner by just using actual command line arguments. If that works out - it feels safer for a bugfix backport. I wouldn't mind changing to doing it via a pipe in 3.15 though.
pythongh-144503: Harden init pipe I/O in forkserver
gh-144503: Pass sys.argv to forkserver as real argv elements (GH-148194)
[3.14] pythongh-144503: Pass sys.argv to forkserver as real argv elem…
[3.13] pythongh-144503: Pass sys.argv to forkserver as real argv elem…
sjoerdjob commentedon Apr 7, 2026
The alternate fix in #148194 is also what I went with in my PR at first, but did switch to using a pipe to send data after considering the case in the description of the alternate PR:
I think that the edge-case triggered by the two extra arguments is very unlikely to be triggered, but it does still exist. For unblocking the 3.13 / 3.14 releases; the alternate fix is "good enough".
[3.14] gh-144503: Pass sys.argv to forkserver as real argv elements (G…
[3.13] gh-144503: Pass sys.argv to forkserver as real argv elements (G…
gpshead commentedon Apr 7, 2026
Okay the regression is Fixed with those merging, removing release blocker. I'll keep this open though to track the further improvement in how this is done for 3.15 in the other PR.
pythongh-144503: Pass sys.argv to forkserver as real argv elements (p…