Skip to content

Data race in test_ssl.test_sni_callback_race #150191

Description

@colesbury

Bug report

There is a sporadic thread sanitizer reported data race in a newly added test

For example:

Here is a summary from Claude with my edits:


On the free-threaded TSAN build, test.test_ssl.ContextTests.test_sni_callback_race sporadically reports a data race inside OpenSSL: one thread reads an ASN.1 string via ASN1_STRING_cmp (no lock held) while another writes the same heap block via ASN1_STRING_set (holding an internal CRYPTO_THREAD_lock rwlock). The test itself does not crash.

Reproducer

CC=clang-20 ./configure -C --disable-gil --with-pydebug --with-thread-sanitizer
make -j

for i in $(seq 1 30); do
  TSAN_OPTIONS="halt_on_error=1" \
    ./python -m test test_ssl -v -m test_sni_callback_race > /tmp/run_$i.log 2>&1
  [ $? -ne 0 ] && echo "FAILED on run $i" && break
done

Reproduces within ~15 runs on a 22-core machine.

TSAN report (abridged)

WARNING: ThreadSanitizer: data race

  Read of size 8 by thread T11:
    #0 memcmp
    #1 ASN1_STRING_cmp           (libcrypto.so.3)
    ...
    #22 thread_run               Modules/_threadmodule.c:388

  Previous write of size 8 by thread T12 (mutexes: write M0):
    #0 memcpy
    #1 ASN1_STRING_set           (libcrypto.so.3)
    ...
    #22 thread_run               Modules/_threadmodule.c:388

  Location is heap block of size 21 allocated by ASN1_STRING_set
  Mutex M0 created by CRYPTO_THREAD_lock_new (libcrypto.so.3)

SUMMARY: ThreadSanitizer: data race in memcmp

The writer holds an OpenSSL-owned rwlock; the reader does not take the same lock. Both call sites enter from two Python worker threads doing concurrent SSL handshakes on the same SSLContext.

Environment

  • CPython main @ c35b0f2 (3.16.0a0, free-threading debug TSAN)
  • Clang 20.1.8
  • OpenSSL 3.0.13 (30 Jan 2024)
  • Linux 6.8.0-101 x86_64

cc @kiri11 @encukou

Linked PRs

Activity

  1. added 2 commits that reference this issue on May 21, 2026
  2. vstinner commented on Jun 11, 2026

    @vstinner
    Member

    Oh, I created a duplicate issue (gh-151277), I forgot to check if the issue was already reported.

  3. added 4 commits that reference this issue on Jul 7, 2026
  4. kumaraditya303 commented on Jul 8, 2026

    @kumaraditya303
    Contributor

    I tried running the test_sni_callback_race test under openssl compiled with TSAN and it exposed another internal data race in openssl:

    Default build — setter write vs final_server_name read:

    WARNING: ThreadSanitizer: data race (pid=36258)
      Write of size 8 at 0x726c00056c30 by thread T15:          # toggler thread
        #0 ssl3_ctx_callback_ctrl        ssl/s3_lib.c:4200      # ctx->ext.servername_cb = fp
        #1 SSL_CTX_callback_ctrl         ssl/ssl_lib.c:3251
        #2 _ssl__SSLContext_sni_callback_set_impl  Modules/_ssl.c:5319
        #4 getset_set → PyObject_SetAttr                        # server_ctx.sni_callback = ...
    
      Previous read of size 8 at 0x726c00056c30 by thread T12:  # handshake worker
        #0 final_server_name             ssl/statem/extensions.c:1004   # reads servername_cb
        #1 tls_parse_all_extensions      ssl/statem/extensions.c:816
        #2 tls_early_post_process_client_hello  ssl/statem/statem_srvr.c:1965
        #7 ossl_statem_accept → #8 SSL_do_handshake
        #9 _ssl__SSLSocket_do_handshake_impl  Modules/_ssl.c:1099       # server.do_handshake()
    
      Location: heap block of size 1768 (SSL_CTX, allocated in SSL_CTX_new
                from _ssl__SSLContext_impl, Modules/_ssl.c:3509)
    

    Free-threading build — same write, different read site:

    WARNING: ThreadSanitizer: data race (pid=36449)
      Read of size 8 at 0x726c00002c30 by thread T12:           # handshake worker
        #0 is_tls13_capable              ssl/statem/statem_lib.c:1932
           # if (sctx->ext.servername_cb != NULL || s->session_ctx->ext.servername_cb != NULL)
        #1 ssl_version_supported         ssl/statem/statem_lib.c:2011
        #2 ssl_choose_server_version     ssl/statem/statem_lib.c:2240
        #9 SSL_do_handshake → _ssl__SSLSocket_do_handshake_impl  Modules/_ssl.c:1099
    
      Previous write of size 8 by thread T15:                   # toggler thread
        #0 ssl3_ctx_callback_ctrl        ssl/s3_lib.c:4200
        #2 _ssl__SSLContext_sni_callback_set_impl  Modules/_ssl.c:5319
    
    SUMMARY: data race statem_lib.c:1932 in is_tls13_capable
    

    There is also a race in test_thread_recv_while_main_thread_sends test:

    WARNING: ThreadSanitizer: data race
      Write of size 4 by main thread:                           # sock.sendall(data)
        #0 ssl3_write_bytes              ssl/record/rec_layer_s3.c:285   # s->rwstate = SSL_NOTHING
        #3 SSL_write_ex2 → #4 SSL_write_ex
        #5 _ssl__SSLSocket_write_impl    Modules/_ssl.c:2832
    
      Previous write of size 4 by thread T272:                  # background sock.recv()
        #0 ssl3_read_bytes               ssl/record/rec_layer_s3.c:681   # s->rwstate = SSL_NOTHING
        #4 SSL_read_ex
        #5 _ssl__SSLSocket_read_impl     Modules/_ssl.c:2989
    
      Location: heap block of size 5560 (the SSL object, allocated in SSL_new
                from newPySSLSocket, Modules/_ssl.c:959, via context.wrap_socket)
    
    SUMMARY: data race rec_layer_s3.c:285 in ssl3_write_bytes
    

    For now, I will create a PR to skip these two tests under TSAN so that we can re-enable the TSAN CI using #153316.

  5. added 3 commits that reference this issue on Jul 8, 2026
  6. kumaraditya303 commented on Jul 18, 2026

    @kumaraditya303
    Contributor

    cc @ZeroIntensity for the races in test_thread_recv_while_main_thread_sends test

  7. ZeroIntensity commented on Jul 18, 2026

    @ZeroIntensity
    Member

    Yeah, that's #143756. It's difficult to fix because adding locking to ssl gives us issues like #137583. Sam had an idea to fix it using nonblocking sockets, but I don't know if he started working on it.

  8. vstinner commented on Aug 6, 2026

    @vstinner
    Member

    Currently, test_sni_callback_race() and test_thread_recv_while_main_thread_sends() of test_ssl are skipped when run with a thread sanitizer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions