Repository navigation
ProcessPoolExecutor exception memory leak #90943
Description
| BPO | 46787 |
|---|---|
| Nosy | @themylogin |
| PRs | ProcessPoolExecutor exception memory leak #31408 |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2022-02-18.15:16:10.733>
labels = ['library', '3.9', 'performance']
title = 'ProcessPoolExecutor exception memory leak'
updated_at = <Date 2022-02-18.15:23:21.328>
user = 'https://github.com/themylogin'bugs.python.org fields:
activity = <Date 2022-02-18.15:23:21.328>
actor = 'python-dev'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-18.15:16:10.733>
creator = 'themylogin'
dependencies = []
files = ['50628']
hgrepos = []
issue_num = 46787
keywords = ['patch']
message_count = 1.0
messages = ['413485']
nosy_count = 2.0
nosy_names = ['python-dev', 'themylogin']
pr_nums = ['31408']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue46787'
versions = ['Python 3.9']
Activity
themylogin commentedon Feb 18, 2022
If an exception occurs in ProcessPoolExecutor work item, all the exception frame local variables are not garbage collected (or are garbage collected too lately) because they are referenced by the exception's traceback.
Attached file is a test case. With unpatched python 3.9 (debian bullseye) it prints:
root@truenas[~/freenas/freenas]# python test.py
At iteration 0 memory usage is 226070528
At iteration 1 memory usage is 318763008
At iteration 2 memory usage is 318509056
At iteration 3 memory usage is 321662976
At iteration 4 memory usage is 321404928
At iteration 5 memory usage is 324292608
At iteration 6 memory usage is 324296704
At iteration 7 memory usage is 326922240
At iteration 8 memory usage is 326922240
At iteration 9 memory usage is 329543680
With the proposed patch there is no memory usage growth:
At iteration 0 memory usage is 226410496
At iteration 1 memory usage is 226451456
At iteration 2 memory usage is 226451456
At iteration 3 memory usage is 226443264
At iteration 4 memory usage is 226443264
At iteration 5 memory usage is 226435072
At iteration 6 memory usage is 226426880
At iteration 7 memory usage is 226426880
At iteration 8 memory usage is 226435072
At iteration 9 memory usage is 226426880
clchiou commentedon Sep 25, 2022
I wonder if this patch also needs to be applied to Lib/multiprocessing/pool.py as it appears to me that Lib/concurrent/futures/process.py and Lib/multiprocessing/pool.py are quite similar in this aspect: sending exceptions with (stringified) traceback across process boundary. What do you think? @themylogin