Skip to content

Expose max_queue_size in ThreadPoolExecutor #73781

Open
Open
Expose max_queue_size in ThreadPoolExecutor#73781
Labels
3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

Description

@prayerslayer
prayerslayer
mannequin
opened on Feb 17, 2017
BPO 29595
Nosy @rhettinger, @pitrou, @applio, @zhangyangyu, @tomMoral, @DimitrisJim, @prayerslayer, @stephenoneal, @iforapsy, @tianc777
PRs
  • bpo-29595: Expose max_queue_size in ThreadPoolExecutor #143
  • bpo-29595: Add queue size for pool. #23864
  • bpo-29595: Add queue size for pool. #23865
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2017-02-17.22:10:23.402>
labels = ['3.8', 'type-feature', 'library']
title = 'Expose max_queue_size in ThreadPoolExecutor'
updated_at = <Date 2021-10-14.21:59:47.298>
user = 'https://github.com/prayerslayer'

    bugs.python.org fields:

    activity = <Date 2021-10-14.21:59:47.298>
actor = 'iforapsy'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-02-17.22:10:23.402>
creator = 'prayerslayer'
dependencies = []
files = []
hgrepos = []
issue_num = 29595
keywords = ['patch']
message_count = 6.0
messages = ['288043', '289082', '290253', '290949', '314699', '314743']
nosy_count = 11.0
nosy_names = ['rhettinger', 'pitrou', 'python-dev', 'davin', 'xiang.zhang', 'tomMoral', 'Jim Fasarakis-Hilliard', 'prayerslayer', 'stephen.oneal.04', 'iforapsy', 'tianc777']
pr_nums = ['143', '23864', '23865']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29595'
versions = ['Python 3.8']

    Activity

    1. prayerslayer commented on Feb 17, 2017

      @prayerslayer
      prayerslayermannequin
      on Feb 17, 2017
      MannequinAuthor

      Hi!

      I think the ThreadPoolExecutor should allow to set the maximum size of the underlying queue.

      The situation I ran into recently was that I used ThreadPoolExecutor to parallelize AWS API calls; I had to move data from one S3 bucket to another (~150M objects). Contrary to what I expected the maximum size of the underlying queue doesn't have a non-zero value by default. Thus my process ended up consuming gigabytes of memory, because it put more items into the queue than the threads were able to work off: The queue just kept growing. (It ran on K8s and the pod was rightfully killed eventually.)

      Of course there ways to work around this. One could use more threads, to some extent. Or you could use your own queue with a defined maximum size. But I think it's more work for users of Python than necessary.

    2. prayerslayer
      added
      type-featureA feature request or enhancement
      on Feb 17, 2017
    3. Mariatta
      added
      stdlibStandard Library Python modules in the Lib/ directory
      3.7 (EOL)end of life
      on Feb 18, 2017

    4. prayerslayer commented on Mar 6, 2017

      @prayerslayer
      prayerslayermannequin
      on Mar 6, 2017
      MannequinAuthor

      Hello again, there's a reviewed PR open for this issue and it hasn't even received authoritative feedback yet (ie whether or not you intend to support this feature at all). I would be very happy if a core dev could look over this change before everyone forgets about it :)

    5. prayerslayer commented on Mar 24, 2017

      @prayerslayer
      prayerslayermannequin
      on Mar 24, 2017
      MannequinAuthor

      Ping. That's really a two-line change, can be easily reviewed in 15 minutes :)

    6. rhettinger commented on Mar 31, 2017

      @rhettinger
      rhettinger
      on Mar 31, 2017
      Contributor

      Prayslayer, please don't shove. Your PR request was responded to by Mariatta so it wasn't ignored.

      Making decisions about API expansions takes a while (making sure it fits the intended use, that it isn't a bug factory itself, that it is broadly useful, that it is the best solution to the problem, that is doesn't complicate the implementation or limit future opportunities, that there are unforeseen problems). Among the core developers, there are only a couple part-time contributors who are qualified to make these assessments for the multi-processing module (those devs don't include me).

    7. rhettinger
      assigned
      applio
      on Mar 31, 2017

    8. stephenoneal commented on Mar 30, 2018

      @stephenoneal
      stephenonealmannequin
      on Mar 30, 2018
      Mannequin

      My project we're going into the underlying _work_queue and blocking adding more elements based on unfinished_tasks to accomplish this, bubbling this up to the API would be a welcome addition.

    9. pitrou commented on Mar 31, 2018

      @pitrou
      pitrou
      on Mar 31, 2018
      Member

      Please note the PR here has some review comments that need addressing.
      Also, it needs its conflicts with git master resolved.

      I'm cc'ing Thomas Moreau, who has done a lot of work recently on the concurrent.futures internals.

    10. pitrou
      added
      3.8 (EOL)end of life
      and removed
      3.7 (EOL)end of life
      on Mar 31, 2018
    11. pitrou
      unassigned
      applio
      on Mar 31, 2018
    12. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    13. serhiy-storchaka
      added this to Threading issues 🧵on May 20, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions