3.12 regression: "can't create new thread at interpreter shutdown" from non-daemon threads or atexit handlers

Bug report

Bug description:

I have a single threading.Timer held on a class as self.timer...

    def refresh_timer(self):
        if self.timer is not None:
            self.timer.cancel()
        self.timer = threading.Timer(self.worker_context.timeout_minutes * 60, self.stop_cycle)
        self.timer.start()

As recently as 3.10 this code worked. As of 3.12 it errors with "can't create new thread at interpreter shutdown". The code has not changed, and the interpreter is not shutting down at the time this error is thrown, as in the process is live and code is executing. There are no code changes from where this is working in 3.10 to where it is not in 3.12.1 (upgraded from 3.12 to 3.12.1 and tsted again to see if this had been fixed).

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

Can you please provide a full, self-contained reproducer? The provided snippet does not appear to be complete.

Not easily. The use case is complex, but the error is real, and was not there in 3.10. Maybe you can help me define a case I can get a reproducer for, or help me debug why this is happening now. The process entry point uses importlib.import_module() to load 'external code', starts a thread...

class WorkerThread(threading.Thread):
    def __init__(self, worker):
        threading.Thread.__init__(self)

        self.setDaemon(False)
        self.stopped = False
        self.worker = worker

self.worker_thread = WorkerThread(self)
self.worker_thread.start()

...and starts the timer (which is another thread I guess)...

self.timer = None
self.refresh_timer()

...and then, I don't know... exits, but doesn't because the WorkerThread is not a daemon. The WorkerThread uses the the passed in 'worker' to handle all the business logic. Periodically, as in often, the refresh_timer() method will be called from the WorkerThread instance, via the self.worker reference.

The initial timer setup is fine in the 'main' thread, but after that thread finishes (but doe not exit due to the non-daemon thread), the refresh_timer() is called again and fails with "can't create new thread at interpreter shutdown". Maybe this is related to work on multiple interpreters in 3.12?

Explaining this this way has helped in any case, and maybe I can get these classes setup. I think the module load, which is hard to replicate, may have nothing to do with this and I can try with a main thread,, a worker thread, and a timer thread. However, if the answer is obvious from my use case don't hesitate to save me some work and tell me what's up.

Voila...

import threading
import time

def main():
	
	worker = Worker();

class Worker:
    def __init__(self):
        self.worker_thread = WorkerThread(self)
        self.worker_thread.start()

        self.timer = None
        self.refresh_timer()
        print("worker init complete")

    def refresh_timer(self):
        if self.timer is not None:
            self.timer.cancel()
        self.timer = threading.Timer(30 * 60, self.stop_cycle)
        self.timer.start()

    def stop_cycle(self):
 	    print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")

class WorkerThread(threading.Thread):
    def __init__(self, worker):
        threading.Thread.__init__(self)

        self.setDaemon(False)
        self.stopped = False
        self.worker = worker

    def run(self):
        while True:
        	print("running...")
        	self.worker.refresh_timer()
        	time.sleep(3)

main()

C:\Users\david\Documents\testme.py:30: DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead
  self.setDaemon(False)
running...
worker init complete
running...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Program Files\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()
  File "C:\Users\david\Documents\testme.py", line 37, in run
    self.worker.refresh_timer()
  File "C:\Users\david\Documents\testme.py", line 21, in refresh_timer
    self.timer.start()
  File "C:\Program Files\Python312\Lib\threading.py", line 992, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't create new thread at interpreter shutdown

If I keep the main entry point running, as with timer.sleep(a_long_time), then I don't get the thread creation error. However, that would force me to throw in some waiting loop on a lock to keep the main thread alive until the non-daemon thread terminates, being very careful to trap any error and release the main thread (try/catch/finally). But none of that makes any difference to me. The whole point of designating the second thread as non-daemon is so the main thread does not exit until all the non-daemon threads terminate, so the system is responsible for all that.

My guess is that whatever throws "can't create new thread at interpreter shutdown" is checking some attribute of the main thread, but failing to check for any running non-daemon threads. Or, whatever code makes up the main thread is responding that yes, it has terminated, despite the fact that non-demon child threads are still running. So I think this is a python bug.

On a first glance this behaviour does seem correct:

main() creates a new non-daemon thread that does the actual work, and then returns. That mains the main thread immediately tries to exit the script and is therefore finalising the interpreter, waiting for the WorkerThread to stop before actually doing most of the cleanup and exiting the interpreter.

If you call worker.worker_thread.join() in main() the script will continue to run as long as the worker thread is.

I'm not an expert on the finer details of interpreter shutdown though.

The problem is that I want the function to return. I'll agree that a join is better than an event or a barrier, but anything like those will prevent the init function return, which is ugly, But more than that, it's wrong. The first problem with the argument that the main thread should exit the interpreter, but not 'finish' until the non-daemon thread exits, is that this is a significant change in behavior. This works correctly in 3.10 and throws this error in 3.12. That alone is dangerous. The second problem is that it makes no sense. A thread which is 'finalising' in the interpreter has exited as far as any client usage, and the docs on daemon threads say...

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left.

So the 'program' can not exit due to the non-daemon thread, but the main thread kind of can? Even if that wasn't vague enough, if the program has not exited, why can't the live thread create a timer, or start another thread? The program has not exited. There is code running in the interpreter. But it's now blocked from creating a new thread?

The docs should be amended to say...

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. However, a program running only child non-daemon threads will not be able to create new threads, such as with timer.start().

Does that really make sense?

by the way, def _shutdown() in threading .py, contains this code...

    # Join all non-deamon threads
    while True:
        with _shutdown_locks_lock:
            locks = list(_shutdown_locks)
            _shutdown_locks.clear()

        if not locks:
            break

        for lock in locks:
            # mimic Thread.join()
            lock.acquire()
            lock.release()

        # new threads can be spawned while we were waiting for the other
        # threads to complete

Note the comments. I really think the problem is the check in timer.start() which s not properly deciding to check for active non-daemon threads. I'm trying to find that code. Any pointers to it would be helpful.

There should IMHO at least be documentation about this behaviour. I'm therefore reopening the issue.

I've done some research. Py_FinalizeEx first sets the finalizing flag in the interpreter state, the waits for threads to exit:

cpython/Python/pylifecycle.c

Lines 1823 to 1842 in ac92527

    
           int 
        
           Py_FinalizeEx(void) 
        
           { 
        
               int status = 0; 
        
               _PyRuntimeState *runtime = &_PyRuntime; 
        
               if (!runtime->initialized) { 
        
                   return status; 
        
               } 
        
               /* Get current thread state and interpreter pointer */ 
        
               PyThreadState *tstate = _PyThreadState_GET(); 
        
               // XXX assert(_Py_IsMainInterpreter(tstate->interp)); 
        
               // XXX assert(_Py_IsMainThread()); 
        
               // Block some operations. 
        
               tstate->interp->finalizing = 1; 
        
               // Wrap up existing "threading"-module-created, non-daemon threads. 
        
               wait_for_thread_shutdown(tstate);

Starting a new thread ends up in _threading.start_joinable_thead, which in the end calls do_start_new_thread which raises an exception when the finalizing flag in the interpreter state is set:

cpython/Modules/_threadmodule.c

Lines 1249 to 1265 in ac92527

    
           static int 
        
           do_start_new_thread(thread_module_state* state, 
        
                               PyObject *func, PyObject* args, PyObject* kwargs, 
        
                               int joinable, 
        
                               PyThread_ident_t* ident, PyThread_handle_t* handle) 
        
           { 
        
               PyInterpreterState *interp = _PyInterpreterState_GET(); 
        
               if (!_PyInterpreterState_HasFeature(interp, Py_RTFLAGS_THREADS)) { 
        
                   PyErr_SetString(PyExc_RuntimeError, 
        
                                   "thread is not supported for isolated subinterpreters"); 
        
                   return -1; 
        
               } 
        
               if (interp->finalizing) { 
        
                   PyErr_SetString(PyExc_RuntimeError, 
        
                                   "can't create new thread at interpreter shutdown"); 
        
                   return -1; 
        
               }

This is unfortunately deep as Timer sub-classes Thread, and the error gets thrown in do_start_new_thread() of _threadmodule.c...

    if (interp->finalizing) {
        PyErr_SetString(PyExc_RuntimeError,
                        "can't create new thread at interpreter shutdown");
        return -1;
    }

So the interpreter is responding that it is finalizing, so I think my contention is either that...

The interpreter should be blocked from entering the finalizing state until all non-daemon threads have exited. I think that makes sense if there's a single interpreter because there's code still running. My guess is this bug is tied up with the work on allowing multiple interpreters, which makes sense version wise, and the main thread's interpreter is finalizing and there are now other interpreters for the non-daemon threads, but the thread start only checks the main interpreter?

or

That both the main thread and all non-daemon thread interpreters need to be checked before throwing that error?

There is a specific test for this error being thrown in test_threading.py, but no test that this error is not thrown if there's a non-daemon thread still running.

And yes, what you saw above. Thank you.

Merge branch 'pythongh-113964-thread-shutdown' into nogil-integration

def exit_func():
    thread = threading.Thread(target=foo)
    thread.start()


atexit.register(exit_func)

The current PR is focused on spawning a thread during finalization from a main ended garbage collection-ish context. The atexit context may need its own PR as is opens new questions that need specific answers and a bit of different implementation.

We need additional changes to make that work safely. It raises the question of "when do we call atexit handlers?"... if they allow code to launch a thread during an atexit handler... it'll run concurrently and must not re-trigger the same atexit handlers to be called when it is the last thread standing and itself exits. do atexit handlers executing prevent the registration of new atexit handlers? people could setup creative loops registering new ones from such a thread... newly registered ones weren't in the first list to be called, they could remain and get called. ... pondering ...

recording a use case from the linked duped issue #115219 directly for reference: TL;DR - Subprocess on windows uses a thread thus cannot be used in atexit context in 3.12 there as matplotlib does:

In Matplotlib we are holding onto a latex process and triggering this via a weakref finalize, but a minimal reproducer is:

from subprocess import Popen, PIPE
import atexit

# anything that holds stdin and stdout should work
proc = Popen(['powershell'], stdin=PIPE, stdout=PIPE)

def cleanup():
    print('hi bob')
    proc.kill()
    proc.communicate()

atexit.register(cleanup)

gh-113964: Don't prevent new threads until all non-daemon threads exit (

[3.12] pythongh-113964: Don't prevent new threads until all non-daemo…

[3.12] gh-113964: Don't prevent new threads until all non-daemon thre…

pythongh-113964: Don't prevent new threads until all non-daemon threa…

Just to have another reproducer, we have code that sometimes starts a new thread and sometimes gets called from GC during shutdown (actually from C objects that are destructed). This lead to processes hanging during shutdown.

Code:

from threading import Thread

class CleanupOnDelete:
    def __del__(self):
        print("Starting thread due to deletion")
        thread = Thread(target=lambda: None)
        thread.start()
        print("Thread started")
        thread.join()
        print("Thread joined")

foo = CleanupOnDelete()

This used to work with Python 3.7, lead to interpreter hangs in intermediate versions and not raises an exception:

$ docker run -it --rm -v "$PWD":/src -w /src python:3.7 python exit_gc_hang.py
Starting thread due to deletion
Thread started
Thread joined

$ docker run -it --rm -v "$PWD":/src -w /src python:3.8 python exit_gc_hang.py
Starting thread due to deletion
<hangs...>

$ docker run -it --rm -v "$PWD":/src -w /src python:3.10 python exit_gc_hang.py
Starting thread due to deletion
<hangs...>

$ docker run -it --rm -v "$PWD":/src -w /src python:3.11 python exit_gc_hang.py
Starting thread due to deletion
<hangs...>

$ docker run -it --rm -v "$PWD":/src -w /src python:3.12 python exit_gc_hang.py
Starting thread due to deletion
Exception ignored in: <function CleanupOnDelete.__del__ at 0x7f83834f6200>
Traceback (most recent call last):
  File "/src/exit_gc_hang.py", line 7, in __del__
  File "/usr/local/lib/python3.12/threading.py", line 992, in start
RuntimeError: can't create new thread at interpreter shutdown

$ docker run -it --rm -v "$PWD":/src -w /src python:3.13-rc-bookworm python exit_gc_hang.py
Starting thread due to deletion
Exception ignored in: <function CleanupOnDelete.__del__ at 0x7fd02488d760>
Traceback (most recent call last):
  File "/src/exit_gc_hang.py", line 7, in __del__
  File "/usr/local/lib/python3.13/threading.py", line 971, in start
PythonFinalizationError: can't create new thread at interpreter shutdown

We can actually live with getting an exception, but it is difficult to deal with processes stuck in shutdown.
Currently, we are guarding the thread creation with a check if main_thread().is_alive().

You can try threading.Thread.join() function to block the main thread

Navigation Menu

Repository navigation

3.12 regression: "can't create new thread at interpreter shutdown" from non-daemon threads or atexit handlers #113964

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

21 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	int
	Py_FinalizeEx(void)
	{
	int status = 0;

	_PyRuntimeState *runtime = &_PyRuntime;
	if (!runtime->initialized) {
	return status;
	}

	/* Get current thread state and interpreter pointer */
	PyThreadState *tstate = _PyThreadState_GET();
	// XXX assert(_Py_IsMainInterpreter(tstate->interp));
	// XXX assert(_Py_IsMainThread());

	// Block some operations.
	tstate->interp->finalizing = 1;

	// Wrap up existing "threading"-module-created, non-daemon threads.
	wait_for_thread_shutdown(tstate);

	static int
	do_start_new_thread(thread_module_state* state,
	PyObject func, PyObject args, PyObject* kwargs,
	int joinable,
	PyThread_ident_t* ident, PyThread_handle_t* handle)
	{
	PyInterpreterState *interp = _PyInterpreterState_GET();
	if (!_PyInterpreterState_HasFeature(interp, Py_RTFLAGS_THREADS)) {
	PyErr_SetString(PyExc_RuntimeError,
	"thread is not supported for isolated subinterpreters");
	return -1;
	}
	if (interp->finalizing) {
	PyErr_SetString(PyExc_RuntimeError,
	"can't create new thread at interpreter shutdown");
	return -1;
	}

3.12 regression: "can't create new thread at interpreter shutdown" from non-daemon threads or atexit handlers #113964

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

zware commented on Jan 11, 2024

zenbones commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

ronaldoussoren commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

ronaldoussoren commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

zenbones commented on Jan 12, 2024

21 remaining items

gpshead commented on Mar 19, 2024

gpshead commented on Mar 19, 2024

tlandschoff-scale commented on Jun 19, 2024

3042457129 commented on Sep 2, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions