Crash on _ssl__SSLContext_load_cert_chain_impl (requests running w/ cert in multi-threading)

Crash report

What happened?

Hi.
We've been investigating random crashes of our FastAPI application for over 6 months, and we think we've found the culprit.
When calling requests with a custom cert (like in the code below) in a multi-threaded paradigm, it can crashes. On some versions, like 3.12/3.13, it can in some case even block Python in a zombie state, where the process isn't killed but keep being hung.

I tested on all the versions mentionned, and the crash happened on all versions. In the latest ones (>=3.12), it feels like I get more often double free.

import threading
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta
from pathlib import Path

import requests
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives.serialization import pkcs12
from cryptography.x509.oid import NameOID

CERT_PEM = "client_cert.pem"
KEY_PEM = "client_key.pem"
PFX_FILE = "client_cert.pfx"
CERT_PASSWORD = b"password"  # For PFX export


def generate_and_save_cert() -> None:
    """Generate RSA key and self-signed cert, save PEM and PFX. Can be commented out. """
    key = rsa.generate_private_key(
        public_exponent=65537,
        key_size=2048,
    )
    subject = issuer = x509.Name(
        [
            x509.NameAttribute(NameOID.COUNTRY_NAME, "US"),
            x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, "California"),
            x509.NameAttribute(NameOID.LOCALITY_NAME, "San Francisco"),
            x509.NameAttribute(NameOID.ORGANIZATION_NAME, "Test Org"),
            x509.NameAttribute(NameOID.COMMON_NAME, "localhost"),
        ]
    )
    cert = (
        x509.CertificateBuilder()
        .subject_name(subject)
        .issuer_name(issuer)
        .public_key(key.public_key())
        .serial_number(x509.random_serial_number())
        .not_valid_before(
            datetime.now(
                datetime.utcnow()  # for backward compatibility
            )
        )
        .not_valid_after(
            datetime.now(
                datetime.utcnow()  # for backward compatibility
            )
            + timedelta(days=365)
        )
        .sign(key, hashes.SHA256())
    )

    Path(CERT_PEM).write_bytes(cert.public_bytes(serialization.Encoding.PEM))
    Path(KEY_PEM).write_bytes(
        key.private_bytes(
            encoding=serialization.Encoding.PEM,
            format=serialization.PrivateFormat.TraditionalOpenSSL,
            encryption_algorithm=serialization.NoEncryption(),
        )
    )

    pfx = pkcs12.serialize_key_and_certificates(
        name=b"client",
        key=key,
        cert=cert,
        cas=None,
        encryption_algorithm=serialization.BestAvailableEncryption(CERT_PASSWORD),
    )
    Path(PFX_FILE).write_bytes(pfx)


def post_with_cert(url: str, idx: int) -> None:
    """Perform a POST request using PEM cert."""
    try:
        response = requests.get(
            url,
            data={"test": f"thread-{idx}"},
            cert=(CERT_PEM, KEY_PEM),  # PEM files
            timeout=10,
        )
        print(f"Thread {idx}: Status {response.status_code}")
    except Exception as exc:
        print(f"Thread {idx}: Exception {exc}")


def main() -> None:
    generate_and_save_cert()
    url = "https://example.com"  # <-- Change this!
    with ThreadPoolExecutor(max_workers=150) as executor:
        for i in range(150):
            executor.submit(post_with_cert, url, i)


if __name__ == "__main__":
    main()

Here's the crash dump:

Core was generated by `/usr/local/bin/python3.15 foo.py'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000771454ecf624 in ?? () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7713720006c0 (LWP 87696))]

#0  0x0000771454ecf624 in ?? () from /usr/lib/libc.so.6
#1  0x0000771454e75ba0 in raise () from /usr/lib/libc.so.6
#2  0x0000771454e5d582 in abort () from /usr/lib/libc.so.6
#3  0x0000771454e5e3bf in ?? () from /usr/lib/libc.so.6
#4  0x0000771454ed9765 in ?? () from /usr/lib/libc.so.6
#5  0x0000771454edbc8a in ?? () from /usr/lib/libc.so.6
#6  0x0000771454ede9ab in free () from /usr/lib/libc.so.6
#7  0x0000771453dc2eb5 in RSA_free () from /usr/lib/libcrypto.so.3
#8  0x0000771453d5ebd2 in ?? () from /usr/lib/libcrypto.so.3
#9  0x0000771453d5f498 in EVP_PKEY_free () from /usr/lib/libcrypto.so.3
#10 0x0000771454198e8a in ?? () from /usr/lib/libssl.so.3
#11 0x000077145419e4d6 in SSL_CTX_use_PrivateKey_file () from /usr/lib/libssl.so.3
#12 0x000077145429f899 in _ssl__SSLContext_load_cert_chain_impl (self=0x7714536f9b50, certfile=<optimized out>, keyfile=<optimized out>, password=<optimized out>) at ./Modules/_ssl.c:4148
#13 _ssl__SSLContext_load_cert_chain (self=0x7714536f9b50, args=<optimized out>, nargs=<optimized out>, kwnames=<optimized out>) at ./Modules/clinic/_ssl.c.h:1429
#14 0x0000586280553ca0 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x771454583ba0, args=0x7714540e89e0, nargsf=<optimized out>, kwnames=0x0)
    at ./Include/internal/pycore_call.h:169
#15 PyObject_Vectorcall (callable=0x771454583ba0, args=args@entry=0x771371ffe468, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#16 0x00005862806cda53 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:1619
#17 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#18 _PyEval_Vector (tstate=0x58629578de30, func=0x7714538ee090, locals=0x0, args=0x771452165530, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#19 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714538ee090, args=0x771452165530, nargsf=4, kwnames=0x7714521800b0)
    at ./Include/internal/pycore_call.h:169
#20 method_vectorcall (method=<optimized out>, args=0x771452165538, nargsf=<optimized out>, kwnames=0x7714521800b0) at Objects/classobject.c:64
#21 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x77145217da00, tuple=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:285
#22 _PyObject_Call (tstate=0x58629578de30, callable=0x77145217da00, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348
#23 PyObject_Call (callable=0x77145217da00, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#24 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654
#25 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#26 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361b1c0, locals=0x0, args=0x771452157730, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#27 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361b1c0, args=0x771452157730, nargsf=2, kwnames=0x771452164160)
    at ./Include/internal/pycore_call.h:169
#28 method_vectorcall (method=<optimized out>, args=0x771452157738, nargsf=<optimized out>, kwnames=0x771452164160) at Objects/classobject.c:64
#29 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452157680, tuple=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:285
#30 _PyObject_Call (tstate=0x58629578de30, callable=0x771452157680, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348
#31 PyObject_Call (callable=0x771452157680, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#32 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654
#33 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#34 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361be20, locals=0x0, args=0x771452157b70, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#35 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361be20, args=0x771452157b70, nargsf=2, kwnames=0x771452164700)
    at ./Include/internal/pycore_call.h:169
#36 method_vectorcall (method=<optimized out>, args=0x771452157b78, nargsf=<optimized out>, kwnames=0x771452164700) at Objects/classobject.c:64
#37 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452156b80, tuple=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:285
#38 _PyObject_Call (tstate=0x58629578de30, callable=0x771452156b80, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348
#39 PyObject_Call (callable=0x771452156b80, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#40 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654
#41 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#42 _PyEval_Vector (tstate=0x58629578de30, func=0x77145361b8a0, locals=0x0, args=0x7714521553f0, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#43 0x0000586280557fc2 in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x77145361b8a0, args=0x7714521553f0, nargsf=1, kwnames=0x771452127d60)
    at ./Include/internal/pycore_call.h:169
#44 method_vectorcall (method=<optimized out>, args=0x7714521553f8, nargsf=<optimized out>, kwnames=0x771452127d60) at Objects/classobject.c:64
#45 0x0000586280555b98 in _PyVectorcall_Call (tstate=0x58629578de30, func=0x586280557e30 <method_vectorcall>, callable=0x771452154c40, tuple=<optimized out>, kwargs=<optimized out>)
    at Objects/call.c:285
#46 _PyObject_Call (tstate=0x58629578de30, callable=0x771452154c40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:348
#47 PyObject_Call (callable=0x771452154c40, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#48 0x00005862806cde12 in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:2654
#49 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#50 _PyEval_Vector (tstate=0x58629578de30, func=0x7714549d1170, locals=0x0, args=0x771371fff938, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#51 0x000058628055802b in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714549d1170, args=0x771371fff938, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:169
#52 method_vectorcall (method=<optimized out>, args=0x771371fffbd8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72
#53 0x00005862806f784c in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x771452154800, args=0x771371fffbd8, nargsf=<optimized out>, kwnames=0x0)
    at ./Include/internal/pycore_call.h:169
#54 context_run (self=0x7714521549c0, args=0x771371fffbd0, nargs=<optimized out>, kwnames=0x0) at Python/context.c:728
#55 0x00005862806ceefe in _PyEval_EvalFrameDefault (tstate=0x58629578de30, frame=<optimized out>, throwflag=<optimized out>) at Python/generated_cases.c.h:3764
#56 0x00005862806d9efc in _PyEval_EvalFrame (tstate=0x58629578de30, frame=<optimized out>, throwflag=0) at ./Include/internal/pycore_ceval.h:119
#57 _PyEval_Vector (tstate=0x58629578de30, func=0x7714549d1220, locals=0x0, args=0x771371fffdd8, argcount=<optimized out>, kwnames=<optimized out>) at Python/ceval.c:1961
#58 0x000058628055802b in _PyObject_VectorcallTstate (tstate=0x58629578de30, callable=0x7714549d1220, args=0x771371fffdd8, nargsf=1, kwnames=0x0) at ./Include/internal/pycore_call.h:169
#59 method_vectorcall (method=<optimized out>, args=0x586280a26958 <_PyRuntime+90104>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:72
#60 0x00005862807ed50c in thread_run (boot_raw=0x58629578ddf0) at ./Modules/_threadmodule.c:368
#61 0x000058628076d957 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#62 0x0000771454ecd70a in ?? () from /usr/lib/libc.so.6
#63 0x0000771454f51aac in ?? () from /usr/lib/libc.so.6

Since I did run the test on multiple versions:

I did run the test on official Docker Python images (except 3.9/3.10/3.14/3.15)
My Docker openssl version: OpenSSL 3.0.16 11 Feb 2025 (Library: OpenSSL 3.0.16 11 Feb 2025)
My host openssl version: OpenSSL 3.4.1 11 Feb 2025 (Library: OpenSSL 3.4.1 11 Feb 2025)
For CPython main branch, here's my python -VV: Python 3.15.0a0 (heads/main-dirty:1729468016, May 23 2025, 14:52:55) [GCC 14.2.1 20250207]
My distrib: Manjaro Linux 25.0.0
/proc/version: Linux version 6.11.5-lqx1-1-lqx (linux-lqx@archlinux) (gcc (GCC) 14.2.1 20240910, GNU ld (GNU Binutils) 2.43.0) #1 ZEN SMP PREEMPT Tue, 22 Oct 2024 15:40:56 +0000

CPython versions tested on:

3.10, 3.11, 3.12, 3.13, 3.14, CPython main branch, 3.9

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

@ZeroIntensity

Maybe some race condition. @ZeroIntensity want to have a look at this one?

It may also be an issue in OpenSSL though as we're actually releasing the GIL here:

    PySSL_BEGIN_ALLOW_THREADS_S(pw_info.thread_state);
    r = SSL_CTX_use_PrivateKey_file(self->ctx,
        PyBytes_AS_STRING(keyfile ? keyfile_bytes : certfile_bytes),
        SSL_FILETYPE_PEM);
    PySSL_END_ALLOW_THREADS_S(pw_info.thread_state);

when accessing the cert/key/etc. Note that entire function is in a critical section as well so I'm not sure we can do PySSL_BEGIN_ALLOW_THREADS_S.

At a quick glance, releasing the GIL/detaching the tstate looks unsafe, because threads are free to run there without synchronization, which OpenSSL doesn't like. That includes free-threaded builds, because it will release the critical section and also break thread-safety. I'm not sure what the best way to fix this is, because we should definitely be detaching around long operations. Maybe a dedicated mutex?

Here's a shorter repro:

import ssl
import threading

ctx = ssl.create_default_context()

def race():
    ctx.load_cert_chain("./Lib/test/certdata/keycert.pem")

threads = [threading.Thread(target=race) for _ in range(8)]
for thread in threads:
    thread.start()

Issues linked:

Requests v2.32.0-v2.32.3 caused a segmentation fault with multiple simultaneous requests with a certificate. psf/requests#6872
Using custom ssl_context with PoolManager is not thread-safe under some conditions urllib3/urllib3#3036

@ZeroIntensity

@ZeroIntensity Will the patch by any chance backport to older versions of Python, or will we have to wait for Python 3.15?

I think we can backport the fix to 3.13 and 3.14, unless this is causing security problems (e.g., a denial-of-service in production web servers), in which case we can get it all the way back to 3.9. I suspect it's the former, though.

This seems like a known issue in requests as in psf/requests#6726 and psf/requests#6872

I think is covered under the note at https://docs.python.org/3/library/ssl.html#ssl.SSLContext as unsafe use

Note SSLContext is designed to be shared and used by multiple connections. Thus, it is thread-safe as long as it is not reconfigured after being used by a connection.

Yeah, I think requests has its own problem. I don't think that documentation note fully applies here, especially in my repro, where there's no connection being used at all.

Yeah, I think requests has its own problem. I don't think that documentation note fully applies here, especially in #134698 (comment), where there's no connection being used at all.

I think the term "connection" here really means concurrent writing i.e. it is safe to use context in parallel after configuration are done sequentially but after that modifications are not thread safe.

What does "configuration" mean in this context?

I can still reproduce this using solely ssl on the main branch. I'm not convinced that the documentation note is supposed to apply to general concurrent use of SSLContext objects.

@ZeroIntensity

I can also reproduce @ZeroIntensity's simple repro leading to segfaults intermittently. The backtrace in GDB points to a segfault inside openssl comparing internal items, which leads me to believe this is a race on the SSLContext leading to unsafety in openssl, and not #114653 after all.

The note in the docs says:

SSLContext is designed to be shared and used by multiple connections. Thus, it is thread-safe as long as it is not reconfigured after being used by a connection.

I think the second sentence reads to me as "if you attempt to modify an SSLContext concurrently it will cause issues". The safety promised only seems to be on reads of the data in the SSLContext. This lines up with the description David Benjamin gave on the openssl issue tracker:
"A thread may not reconfigure an SSL_CTX while another thread is accessing it."

Therefore we absolutely should add locks around SSL_CTXs. I think this is why I was suggesting a RWLock, the context should be readable from multiple threads simultaneously as long as no one is mutating it. But I think using a plain mutex might be fine, most concurrent operations (i.e. making a new socket) on an SSL_CTX would end up mutating it I think?

One other thought is that we should probably benchmark the performance impact of adding locks around SSLContexts. The change should ideally have minimal impact on time to create a new socket (latency) and rate of socket creation (throughput).

gh-134698: Hold a lock when the thread state is detached in `ssl` (GH…

[3.14] pythongh-134698: Hold a lock when the thread state is detached…

fixed in main, up to the RM to decide if this is 3.14.1 or worthy of being in an 3.14rc. (edit: & the 3.13 backport is ready)

[3.14] pythongh-134698: Hold a lock when the thread state is detached…

[3.13] gh-134698: Hold a lock when the thread state is detached in ssl (

pythongh-134698: Hold a lock when the thread state is detached in `ss…

Let's wait for 3.14.1 for this one.

pythongh-134698: Hold a lock when the thread state is detached in `ss…

Would this fix be back-ported to older Python 3 versions?

We weren't planning on it, no. If you believe this is a security problem, please contact the PSRT.

[3.14] gh-134698: Hold a lock when the thread state is detached in `s…

Navigation Menu

Repository navigation

Crash on _ssl__SSLContext_load_cert_chain_impl (requests running w/ cert in multi-threading) #134698

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

11 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Crash on _ssl__SSLContext_load_cert_chain_impl (requests running w/ cert in multi-threading) #134698

Description

Crash report

What happened?

CPython versions tested on:

Operating systems tested on:

Output from running 'python -VV' on the command line:

Linked PRs

Activity

picnixz commented on May 26, 2025

ZeroIntensity commented on May 26, 2025

ZeroIntensity commented on May 26, 2025

Conobi commented on May 26, 2025

Conobi commented on May 26, 2025

ZeroIntensity commented on May 26, 2025

kumaraditya303 commented on May 26, 2025

kumaraditya303 commented on May 26, 2025

ZeroIntensity commented on May 26, 2025

kumaraditya303 commented on May 26, 2025

ZeroIntensity commented on May 26, 2025

11 remaining items

ZeroIntensity commented on Jul 6, 2025

emmatyping commented on Jul 6, 2025

emmatyping commented on Jul 6, 2025

gpshead commented on Jul 25, 2025

hugovk commented on Aug 13, 2025

1oh1 commented on Aug 27, 2025

ZeroIntensity commented on Aug 27, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions