InterpreterPoolExecutor does not work as replacement of ThreadPoolExecutor.

According to PEP-734, it should work as same as ThreadPoolExecutor, but it looks not.

Code

import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor


def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

with InterpreterPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(fib, n): n for n in range(10)}
    for future in concurrent.futures.as_completed(futures):
        n = futures[future]
        data = future.result()
        print(f"fib({n}): {data}")

Execution (InterpreterPoolExecutor)

Exception: AttributeError: module '__main__' has no attribute 'fib'

The above exception was the direct cause of the following exception:

concurrent.interpreters.NotShareableError: object could not be unpickled

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/xxxx/oss/cpython/op.py", line 14, in <module>
    data = future.result()
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/_base.py", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/Users/denny.na/oss/cpython/Lib/concurrent/futures/interpreter.py", line 85, in run
    return self.interp.call(do_call, self.results, *task)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/interpreters/__init__.py", line 233, in call
    return self._call(callable, args, kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/interpreters/__init__.py", line 215, in _call
    res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
concurrent.interpreters.NotShareableError: args not shareable

Execution (ThreadPoolExecutor)

fib(0): 0
fib(7): 13
fib(2): 1
fib(6): 8
fib(1): 1
fib(4): 3
fib(3): 2
fib(5): 5
fib(9): 34
fib(8): 21

@hugovk

cc @hugovk We may need to add label release blocker for 3.14

I think this is because do_call is treated as a stateful function that cannot be shared between multiple interpreters:

$3 = {total = 14, locals = {total = 5, args = {total = 4, numposonly = 0, numposorkw = 4, numkwonly = 0, varargs = 0, varkwargs = 0}, numpure = 1, cells = {total = 0, numargs = 0, numothers = 0},
    hidden = {total = 0, numpure = 0, numcells = 0}}, numfree = 0, unbound = {total = 9, globals = {total = 5, numglobal = 3, numbuiltin = 3, numunknown = 0}, numattrs = 4, numunknown = 0}}

The _interp_call_pack function will check whether the function is stateless:

cpython/Modules/_interpretersmodule.c

Lines 467 to 488 in 42b251b

    
           static int 
        
           _interp_call_pack(PyThreadState *tstate, struct interp_call *call, 
        
                             PyObject *func, PyObject *args, PyObject *kwargs) 
        
           { 
        
               xidata_fallback_t fallback = _PyXIDATA_FULL_FALLBACK; 
        
               assert(call->func == NULL); 
        
               assert(call->args == NULL); 
        
               assert(call->kwargs == NULL); 
        
               // Handle the func. 
        
               if (!PyCallable_Check(func)) { 
        
                   _PyErr_Format(tstate, PyExc_TypeError, 
        
                                 "expected a callable, got %R", func); 
        
                   return -1; 
        
               } 
        
               if (_PyFunction_GetXIData(tstate, func, &call->_preallocated.func) < 0) { 
        
                   PyObject *exc = _PyErr_GetRaisedException(tstate); 
        
                   if (_PyPickle_GetXIData(tstate, func, &call->_preallocated.func) < 0) { 
        
                       _PyErr_SetRaisedException(tstate, exc); 
        
                       return -1; 
        
                   } 
        
                   Py_DECREF(exc); 
        
               }

_interp_call_pack -> _PyFunction_GetXIData -> _PyFunction_VerifyStateless -> _PyCode_VerifyStateless -> _PyCode_CheckNoExternalState

where do_call fails due to:

cpython/Objects/codeobject.c

Lines 2006 to 2008 in 42b251b

    
           else if (counts->unbound.globals.numglobal > 0) { 
        
               errmsg = "globals not supported"; 
        
           }

And the unbound vars seem to be: ['BaseException', 'interpreters', 'print', 'sys', 'traceback']

@xuantengh

@xuantengh
I am waiting for @ericsnowcurrently 's reply. If this is the intention, most people will treat InterpreterPoolExecutor as the same as ThreadPoolExecutor, but actual behavior does not seem to be so; it should be documented.
But if it is not intended, it should be resolved before officially releasing 3.14.

JFTR: it works if we slightly rewrite this code:

import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor

def calc_fib(n):
    def fib(n):
        if n < 2:
            return n
        return fib(n-1) + fib(n-2)

    return fib(n)

with InterpreterPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(calc_fib, n): n for n in range(10)}
    for future in concurrent.futures.as_completed(futures):
        n = futures[future]
        data = future.result()
        print(f"fib({n}): {data}")

@efimov-mikhail

As fib accesses the global scope to get itself it won't work. Thx to @efimov-mikhail for his example. I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

Maybe I can file a PR for this?

There used to be a comment in concurrent.futures.rst for InterpreterPoolExecutor saying: Functions defined in the main module cannot be pickled and thus cannot be used.

This comment was removed in #133957

I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

Maybe I can file a PR for this?

It would be great, but it totally needs prior discussion. I think we should add a big section about recipes with subinterpreters, because there are more problems, than just defining recursive functions. E.g. top-level imports don't work either for the same reason. We should define preferred way to handle all these cases (ofc the discussion should involve Eric Snow and others involved in subinterpreters). We need to open another issue (or a discuss.python.org topic, potentially) :)

Maybe we should start a d.p.o. thread on "How to teach subinterpreters" or something similar?

Can we just wait Eric's comment before do something?

@ericsnowcurrently

I clarified the issue that I met at #136659 (comment)
I think that we can reduce the scope of the issue because it is okay to run from Python 3.14b4 with __name__ == '__main__' idiom.

So it looks reasonable to close this issue and focus on #136659
cc @ericsnowcurrently

Navigation Menu

Repository navigation

InterpreterPoolExecutor does not work as replacement of ThreadPoolExecutor. #136603

Code

Execution (InterpreterPoolExecutor)

Execution (ThreadPoolExecutor)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	static int
	_interp_call_pack(PyThreadState tstate, struct interp_call call,
	PyObject func, PyObject args, PyObject *kwargs)
	{
	xidata_fallback_t fallback = _PyXIDATA_FULL_FALLBACK;
	assert(call->func == NULL);
	assert(call->args == NULL);
	assert(call->kwargs == NULL);
	// Handle the func.
	if (!PyCallable_Check(func)) {
	_PyErr_Format(tstate, PyExc_TypeError,
	"expected a callable, got %R", func);
	return -1;
	}
	if (_PyFunction_GetXIData(tstate, func, &call->_preallocated.func) < 0) {
	PyObject *exc = _PyErr_GetRaisedException(tstate);
	if (_PyPickle_GetXIData(tstate, func, &call->_preallocated.func) < 0) {
	_PyErr_SetRaisedException(tstate, exc);
	return -1;
	}
	Py_DECREF(exc);
	}

	else if (counts->unbound.globals.numglobal > 0) {
	errmsg = "globals not supported";
	}

InterpreterPoolExecutor does not work as replacement of ThreadPoolExecutor. #136603

Description

Code

Execution (InterpreterPoolExecutor)

Execution (ThreadPoolExecutor)

Activity

corona10 commented on Jul 13, 2025

xuantengh commented on Jul 13, 2025

corona10 commented on Jul 13, 2025

efimov-mikhail commented on Jul 13, 2025

Tapeline commented on Jul 13, 2025

xuantengh commented on Jul 13, 2025

paultiq commented on Jul 13, 2025

Tapeline commented on Jul 13, 2025

efimov-mikhail commented on Jul 13, 2025

corona10 commented on Jul 13, 2025

corona10 commented on Jul 14, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions