Add concurrent.futures.InterpreterPoolExecutor

@brianquinlan

Feature or enhancement

Proposal:

While we wait on PEP 734, there's nothing blocking us from adding a new executor for multiple interpreters. Doing so would allow people to start trying them out. (I wish I had thought of this for 3.13!)

My planned design:

mostly built on top of ThreadPoolExecutor
callables (and args) passed as initializer or to submit() or map() will be pickled (same as ProcessPoolExecutor)
initializer and submit() arg can be a script instead of a callable (a script for map() doesn't make sense since there are no args)
__init__() will take a new "shared" arg that corresponds to what Interpreter.prepare_main() takes in PEP 734, to allow each worker interpreter to share some data

In the future we could support a more efficient "sharing" scheme than pickle. When (or if) PEP 734 is accepted, we can refactor the executor to use the interpreters module rather than using _interpreters directly.

Open questions:

reconstitute and raise uncaught exceptions from running in the interpreter (like we do for ProcessPoolExecutor)?
(optionally?) exec functions against main instead of the their original module, i.e. use the body of the function as a script?

CC @brianquinlan

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

gh-124694: Add concurrent.futures.InterpreterPoolExecutor (gh-124548)

gh-124694: In test_interpreter_pool, Restore the Asyncio Event Loop P…

pythongh-124694: Add concurrent.futures.InterpreterPoolExecutor (pyth…

pythongh-124694: In test_interpreter_pool, Restore the Asyncio Event …

pythongh-124694: Document missing ctxkwargs argument in ThreadPoolExe…

~~Does the pickling work?~~ I've been trying several things on pickable arguments, but I keep getting concurrent.interpreters.NotShareableError: args not shareable. My first thought was this:

# /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///

import numpy as np

from concurrent.futures import InterpreterPoolExecutor


def identity(*args):
    return args[0]

if __name__ == "__main__":
    x = np.random.rand(10003)
    xs = np.array_split(x, 4)
    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = pool.map(identity, xs)
        final = np.concat(results)

    assert np.all(x == final)

But I get that error. Then I tried explicitly pickling:

# /// script
# dependencies = ["numpy"]
# requires-python = ">=3.14"
# ///

import numpy as np
import pickle

from concurrent.futures import InterpreterPoolExecutor


def identity(*args):
    return pickle.dumps(pickle.loads(args[0]))

if __name__ == "__main__":
    x = np.random.rand(10003)
    xs = np.array_split(x, 4)
    xsp = (pickle.dumps(a) for a in xs)
    with InterpreterPoolExecutor(max_workers=4) as pool:
        results = pool.map(identity, xsp)
        final = np.concat([pickle.loads(r) for r in results])

    assert np.all(x == final)

But I still get this error:

Exception: AttributeError: module '__main__' has no attribute 'identity'

The above exception was the direct cause of the following exception:

concurrent.interpreters.NotShareableError: object could not be unpickled

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/henryschreiner/git/scikit-hep/boost-histogram/example.py", line 21, in <module>
    final = np.concat([pickle.loads(r) for r in results])
                                                ^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 639, in result_iterator
    yield _result_or_cancel(fs.pop())
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 311, in _result_or_cancel
    return fut.result(timeout)
           ~~~~~~~~~~^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 450, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/futures/interpreter.py", line 85, in run
    return self.interp.call(do_call, self.results, *task)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 233, in call
    return self._call(callable, args, kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henryschreiner/.local/share/uv/python/cpython-3.14.0rc1+freethreaded-macos-x86_64-none/lib/python3.14t/concurrent/interpreters/__init__.py", line 215, in _call
    res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
concurrent.interpreters.NotShareableError: args not shareable

Also, I'm not sure why the Exception: AttributeError: module '__main__' has no attribute 'identity' without a traceback is at the top. Other executors are fine with code.

I was actually trying to use this in a test for boost-histogram, trying to pass a numpy array in and a boost-histogram object back out, but it fails even with byte strings from pickle above.

Edit: I do think it's an issue with the import like seen at the top, and the traceback is a red herring. The tests version of this list Exception: ModuleNotFoundError: No module named 'tests' at the top. So it's not able to import the function to run. I thought __main__ support was fixed?

It seems the subinterpreter's don't inherit the current sys.path? Using pytest 7's pythonpath setting doesn't work, they aren't picking up the added test paths.

If I put the function inside my package, I get ImportError: NumPy: dtype is already registered which I think that's occurring when pybind11 is registering my dtype. I've opened an issue pybind/pybind11#5781; if I manually skip over that, it looks like numpy doesn't support subinterpreters yet, so I'd not really be able to do much with it yet.

pythongh-124694: Add versionadded note to InterpreterPoolExecutor

gh-124694: Add versionadded note to InterpreterPoolExecutor (#140468)

pythongh-124694: Add versionadded note to InterpreterPoolExecutor (py…

[3.14] gh-124694: Add versionadded note to InterpreterPoolExecutor (G…

pythongh-124694: Add versionadded note to InterpreterPoolExecutor (py…

Navigation Menu

Repository navigation

Add concurrent.futures.InterpreterPoolExecutor #124694

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add concurrent.futures.InterpreterPoolExecutor #124694

Description

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Linked PRs

Activity

henryiii commented on Jul 31, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions