Repository navigation
SIGSEGV on macOS when calling ssl.SSLContext.load_verify_location during a shutdown #114653
Description
Crash report
What happened?
On MacOS, calling ssl.SSLContext.load_verify_locations from multiple threads causes a SIGSEGV during process termination when an unhandled exception is raised.
I was able to create a repoduction case and I've attached the five MacOS crash reports here.
Reproduction
from multiprocessing.pool import ThreadPool
import ssl
import certifi
ctx = ssl.create_default_context()
location = certifi.where()
def test_func(_):
ctx.load_verify_locations(location)
with ThreadPool(processes=100) as pool:
for idx, item in enumerate(pool.imap_unordered(test_func, range(1000), chunksize=1)):
if idx == 100:
raise RuntimeError()Traceback (most recent call last):
File "/Users/tomforbes/tmp/scanner/repo.py", line 16, in <module>
raise RuntimeError()
RuntimeError
fish: Job 1, 'python repo.py' terminated by signal SIGSEGV (Address boundary error)
MacOS crash reports:
five.txt
four.txt
one.txt
three.txt
two.txt
CPython versions tested on:
3.11
Operating systems tested on:
macOS
Output from running 'python -VV' on the command line:
Python 3.11.7 (v3.11.7:fa7a6f2303, Dec 4 2023, 15:22:56) [Clang 13.0.0 (clang-1300.0.29.30)]
Activity
orf commentedon Jan 27, 2024
I got it to reliably crash again, and it seems that it's related to an exception being thrown in one thread whilst many other threads are inside
_ssl.Upping the number of open files prevents the initial exception from happening, but whenever an exception is thrown python will quit with a
SIGSEGV.Here is a crash due to a
ValueErrorbeing thrown in the main thread, triggered when trying to serialize an invalid object usingjson.dumps:six.txt
serhiy-storchaka commentedon Jan 28, 2024
Is it possible to reproduce without using third-party libraries?
orf commentedon Jan 28, 2024
I can try but I’m not sure if I will be able to and I don’t have a gigantic amount of time I can spend digging into this.
Looking at the thread traces, there don’t appear to be any stack traces that are in 3rd party native modules - it’s all in CPython or ssl internals. But the issue may be triggered by how boto3 implements connection pooling, which is something that I’m not familiar with. I’ll give it a go.
If I’m able to reproduce this again, is there any more information (outside of a reproducible test case) that would help with this?
I tried enabling faulthandler, but it didn’t output anything which makes me think it’s something that is erroring after Python has shut down.
serhiy-storchaka commentedon Jan 28, 2024
Can this be reproduced without tqdm? Can this be reproduced with smaller number of threads or tasks? How many tasks were finished at the moment of the crash, how many tasks were started, and how much data the unfinished tasks read?
orf commentedon Jan 28, 2024
I've managed to create a reproduction case:
On MacOS, this reliably causes it to trigger:
if you up the threads to something silly like 10k, it errors immediately:
[-]SIGSEGV in _ssl on macOS when launching many threads[/-][+]SIGSEGV in _ssl on macOS when calling ssl.SSLContext.load_verify_location during a shutdown [/+][-]SIGSEGV in _ssl on macOS when calling ssl.SSLContext.load_verify_location during a shutdown [/-][+]SIGSEGV on macOS when calling ssl.SSLContext.load_verify_location during a shutdown [/+]orf commentedon Jan 28, 2024
Using
exit()causes the error, butos._exit(0)doesn't. Judging by this issue, it could be because OpenSSL is being shutdown/cleaned up whilst threads are waiting on a lock? Or something?ronaldoussoren commentedon Jan 30, 2024
I get the following stack trace when testing with a recentish checkout of main and openssl 3.0.12. OpenSSL was build without debug information, I'll have to rebuild it with debug information to get more information.
ronaldoussoren commentedon Jan 30, 2024
Stack trace of the crashing thread with openssl debug info, with open 3.0.13:
This tries to unlock a NULL lock, and according to the source for line two this is the global engine lock:
From quickly scanning the openssl code base this lock is cleared from
OPENSSL_cleanupwhich itself is called from an atexit(3) handler.This seems to be caused by a running thread that is still active when the process is already shutting down, which happens here because the thread pool from #114653 (comment) uses daemon threads and AFAIK Py_Finalize() won't wait for daemon threads to exit. And indeed, if I replace the 'raise RuntimeError' statement by 'pass' the script works without problems.
I'm not sure if we can fix this if my analysis is correct, there is a race condition between the main thread invoking the atexit(3) handlers and secondary threads still running openssl functions. From what I've found POSIX doesn't specify any ordering here.
UPATE: For completeness sake here's the stacktrace for the main thread which is running atexit handlers in particular openssl's cleanup:
ronaldoussoren commentedon Feb 4, 2024
Apparently
OPENSSL_init_cryptohas a flagOPENSSL_INIT_NO_ATEXITto disable theatexitbehaviour. AFAIK we don't call this function though.A recent commit to openssl added a no-atexit configure option (openssl/openssl@99fb31c), we could turn that option on in our macOS installer when we switch to a release that includes this option (that's not ideal though, this is not a macOS specific issue and also won't affect other installation (e.g. homebrew, anaconda).