multiprocessing.Process start behaving eratically when used inside concurrent.futures.ThreadPoolExecutor

Bug report

Bug description:

When a multiprocessing.Process is used from within a thread managed via concurrent.futures.ThreadPoolExecutor, multiple methods start producing unexpected and even contradictory results. Specifically, my example below exposes these unexpected behaviors:

proc.join(timeout) returns even though the process has not terminated and the timeout has not expired.
proc.close() throws a ValueError, claiming the process is still running, even though proc.kill() and proc.join() previously returned for the process.
proc.join(timeout) runs into a timeout, even though the process should have terminated before the timeout. This might not be a real bug, since scheduling delays, execution times and contention around the available threads in the pool can cause unexpected delays. However, in the test setup below, each thread needs to process 4 tasks on average and each tasks should not take much longer than 1s, i.e. 4s in total. With a timeout of 10s in total, this leaves 6s seconds for any execution or scheduling delays, or 1.5s per task. This seems to me like too big a margin of error to fail this consistently. EDIT: this was probably a known side effect of using the fork start method in a multithreaded program, which is unsupported anyway.

I used this program to reproduce these effects.

import concurrent.futures
import multiprocessing
import time
import traceback

TASK_COUNT = 64
MAX_THREADS = 16

def sub_proc():
    time.sleep(1)

def timeout_proc(expected_end_time: float):
    proc = multiprocessing.Process(target=sub_proc)
    proc.start()
    try:
        timeout = expected_end_time - time.monotonic()
        proc.join(timeout)
        observed_end_time = time.monotonic()
        timeout_remaining = expected_end_time - observed_end_time
        exitcode = proc.exitcode
        if exitcode is None:
            print(f"WARNING: timeout expired: {timeout_remaining:.4f}s")
        assert exitcode is not None or observed_end_time >= expected_end_time
    finally:
        proc.kill()
        proc.join()
        proc.close()

if __name__ == '__main__':
    # EDIT: The problems only manifest with spawn or fork, though the latter doesn't seem relevant, since it is already expected to cause problems in multithreaded programs.
    multiprocessing.set_start_method('spawn')
    end_time = time.monotonic() + 10
    with concurrent.futures.ThreadPoolExecutor(MAX_THREAD) as e:
        futs = [e.submit(timeout_proc, end_time) for _ in range(TASK_COUNT)]
        for fut in futs:
            try:
                fut.result()
            except Exception as e:
                traceback.print_exception(e)

The unexpected behaviors can be recognized from the following symptoms:

proper timeouts of proc.join(timeout) cause messages like this, with a negative number:
```
WARNING: timeout expired: -0.0072s
```

if the time is instead positive, it means that proc.join(timeout) returned before either the timeout expired or the process terminated. In addition, we get an assertion failure:

Traceback (most recent call last):
  File "/home/sven/test.py", line 35, in <module>
    fut.result()
    ~~~~~~~~~~^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/sven/test.py", line 23, in timeout_proc
    assert exitcode is not None or observed_end_time >= expected_end_time
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

Unexpected fails of proc.close() cause an additional exception:

ValueError: Cannot close a process while it is still running. You should first call join() or terminate().
Traceback (most recent call last):
  File "/home/sven/test.py", line 23, in timeout_proc
    assert exitcode is not None or observed_end_time >= expected_end_time
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sven/test.py", line 35, in <module>
    fut.result()
    ~~~~~~~~~~^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/sven/test.py", line 27, in timeout_proc
    proc.close()
    ~~~~~~~~~~^^
  File "/usr/lib/python3.13/multiprocessing/process.py", line 181, in close
    raise ValueError("Cannot close a process while it is still running. "
                     "You should first call join() or terminate().")
ValueError: Cannot close a process while it is still running. You should first call join() or terminate()

curiously, this exception never occurs when the assertion didn't fail.

At first glance this behavior seems similar to #130895, but that issue involves calling join on same process concurrently from multiple threads, whereas in my case, each process is owned by a single thread and only interacted with by the owning thread.

My example program above doesn't produce the observed behaviors deterministically, so you may need to run it more than once, ~~though in my environment it always produces at least some unexpected results and some runs even produce all three kinds of unexpected results.~~ EDIT: Switching to the spawn start method has reduced the frequency of the problems noticably, so you might need to run the example a few times, before seeing all symptoms.

The rate at which these unexpected behaviors occur seems to correlate with the ratio of TASK_COUNT / MAX_THREADS, with more tasks per thread causing more unexpected results. I've never seen any of these effects when MAX_THREADS >= TASK_COUNT, nor when starting threads by hand (through threading.Thread).

CPython versions tested on:

3.13

Operating systems tested on:

Linux

I just noticed that the behavior depends on the start method being used:

with fork, the behavior is as described above
with forkserver I was not able to observe any of the effects described above
with spawn I only observed unexpected early returns of proc.join(timeout) and failures of proc.close(), but no unexpected real timeouts.

I think this means that the real timeouts are irrelevant to this issue, since they are probably caused by using fork from a multithreaded program. The failures that occur with spawn do still seem relevant though.

It's a really bad idea to mix multiprocessing and threads, because threads aren't copied to the child process (as you're seeing with fork). Calling fork with threads active is deprecated.

For more information, please see Using a fork start method is and https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555

@ZeroIntensity

@ZeroIntensity, yes, I agree that using fork in my test case wasn't a good test case, because it is deprecated and expected to fail due to known reasons. However, the effects I'm describing aren't expected to happen with with spawn or forkserver. So that still looks like a bug to me.

@Kritzefitz

This is a fascinating bug report, thank you @Kritzefitz!

I think it is actually a dup of #130895, at least with respect to the fork/spawn method. The reason it triggers with this reproduction is somewhat subtle: as you say, it looks like it should be safe! However...

When instances of multiprocessing.process.BaseProcess are started they add themselves to a global set of child processes (multiprocessing.process._children). Cleanup on that list checks that they are alive by doing a non-blocking poll. This is triggered by various things, including starting processes.

So, a worker thread spawns a process. The process is added to the global set, and other worker threads start doing non-blocking polls on it every now and then via cleanup. The spawning thread joins the process, however that is racing with the non-blocking polls from the other threads!

The bug triggers when one of the other threads "wins" the race, reaps the child, and the thread that originally spawned it is left in a broken state, causing the errors you see.

This is another reproducer which triggers the same bug, in the same way. It reliably asserts, on my system at least:

import multiprocessing
import threading

def trigger_polls(stop):
    while not stop.is_set():
        multiprocessing.active_children()

def main():
    multiprocessing.set_start_method('spawn')
    stop = threading.Event()

    poller = threading.Thread(target=trigger_polls, args=(stop,))
    poller.start()
    try:
        p = multiprocessing.Process()
        p.start()
        p.join()
        assert p.exitcode is not None
    finally:
        stop.set()
        poller.join()

if __name__ == '__main__':
    main()

Both this and the OP's reproduction work correctly with the #130895 fixes applied.

@duaneg

Thank you for the research and detailed explanation, @duaneg! I agree that this probably a duplicate of #130895, so I think this issue can be closed and maybe a reference to the additional reproducers and symptoms here can be added to #130895.

Navigation Menu

Repository navigation

`multiprocessing.Process` start behaving eratically when used inside `concurrent.futures.ThreadPoolExecutor` #140994

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

multiprocessing.Process start behaving eratically when used inside concurrent.futures.ThreadPoolExecutor #140994

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

Kritzefitz commented on Nov 4, 2025

ZeroIntensity commented on Nov 4, 2025

YvesDup commented on Nov 4, 2025

Kritzefitz commented on Nov 4, 2025

duaneg commented on Nov 5, 2025

Kritzefitz commented on Nov 6, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`multiprocessing.Process` start behaving eratically when used inside `concurrent.futures.ThreadPoolExecutor` #140994