Repository navigation
3.15 profiling, can't use ProcessPoolExecutor.map() in a simple script #140729
Description
Bug report
Bug description:
# test/test_unit.py
import concurrent.futures
...
def exercise(units=11, rounds=2):
with concurrent.futures.ProcessPoolExecutor() as executor:
...
results = list(
executor.map(
step,
[f"app/{i}" for i in range(units)],
[config] * units,
[local_app_data] * units,
[peers_data] * units,
)
)
def step(
unit: str,
config: Mapping[str, str | int | float | bool],
local_app_data: Mapping[str, JSON],
all_units_data: Mapping[ops.testing.UnitID, Mapping[str, JSON]],
) -> tuple[dict[str, JSON] | None, dict[str, JSON], str | None, str]:
...
if __name__ == "__main__":
import sys
exercise(int(sys.argv[1]), int(sys.argv[2]))(jgol-peer) 🦐/c/j/peer (main)> sudo -E env PYTHONPATH=src python -m profiling.sampling test/test_unit.py 100 100
Profiler coordinator error: Error executing script '/code/jgol/peer/test/test_unit.py': Can't pickle <function step at 0x102b7aae0>: it's not found as __main__.stepCPython versions tested on:
3.15
Operating systems tested on:
macOS
Linked PRs
- gh-140729: Add __mp_main__ as a duplicate for __main__ for pickle to work #140735
- gh-140729: Fix subprocess handling in test_process_pool_executor_pickle #141688
- gh-140729: profiling or tracing multiprocessing can cause error #141874
- gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process #144700
- gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process #144715
- [3.15] gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process (GH-144715) #153243
- [3.14] gh-140729: Fix the cProfile module when the executed script contains calls to multiprocessing.Process (GH-144715) #153265
Activity
picnixz commentedon Oct 28, 2025
cc @pablogsal
yihong0618 commentedon Oct 29, 2025
This is interesting
its very similar like viztracer issue which is gaogaotiantian/viztracer#423
in ProcessPoolExecutor() the
__main__may because__mp_main__a simple reproducer
cat ttt.py
the patch can be like what viztracer did
after the patch
cc @dimaqq
mehetabalom181-lab commentedon Oct 29, 2025
YvesDup commentedon Oct 30, 2025
The execution of
ttt.pyscript fails when run via thecProfilemodule as below:and with the
profilemodule too.Based on my initial tests, the proposed fix is not valid for these two modules. Should we open a new issue ?
yihong0618 commentedon Oct 30, 2025
using the same method can fix this?
YvesDup commentedon Oct 30, 2025
I don`t know yet
YvesDup commentedon Oct 30, 2025
@picnixz, @StanFromIreland I am wondering if a topic-profiling (if exists) or a topic-multiprocessing label should not be better than topic-subinterpreters.
StanFromIreland commentedon Oct 30, 2025
I added topic-multiprocessing, FYI all labels are listed here. topic-profiling
does not exist.10 remaining items
pablogsal commentedon Nov 17, 2025
Thanks everyone for the help! ❤️
pablogsal commentedon Nov 17, 2025
Certainly a new PR
That would be a bit more aggressive that what I would like since that addition can have some unintended side effects but if you want to give it a go to see how it looks let's try it
pablogsal commentedon Nov 17, 2025
Reopening until we figure out Profile
pythongh-140729: Fix subprocess handling in test_process_pool_executo…
gh-140729: Fix subprocess handling in test_process_pool_executor_pick…
YvesDup commentedon Nov 20, 2025
Ok, I have got a new PR dedicated to the
cProfilemoduleThank you for your agreement. But after working about the PR, I realize I don't need this correction. We can be cool, there won't be any side effects. Sorry for this comment
pythongh-140729: Add __mp_main__ as a duplicate for __main__ for pick…
pythongh-140729: Fix subprocess handling in test_process_pool_executo…
gh-140729: Fix the cProfile module when the executed script contains …
[3.15] gh-140729: Fix the cProfile module when the executed script co…
[3.14] gh-140729: Fix the cProfile module when the executed script co…