test_asyncio: test_create_connection_ssl_failed_certificate() failed on ARM64 macOS 3.x buildbot

@pablogsal

ARM64 macOS 3.x:
https://buildbot.python.org/all/#/builders/725/builds/5088

The test started to fail when OpenSSL was upgrade from 3.0.0 to 3.1.1 at July 13.

Last successful build, build 4995 (July 13 at 5:01PM): https://buildbot.python.org/all/#/builders/725/builds/4995
- test.pythoninfo: ssl.OPENSSL_VERSION: OpenSSL 3.0.0 7 sep 2021
Fist failure, build 4999 (July 13 at 5:23PM): https://buildbot.python.org/all/#/builders/725/builds/4999
- test.pythoninfo: ssl.OPENSSL_VERSION: OpenSSL 3.1.1 30 May 2023
Between build 4995 and 4999, build failed with an exception

Error:

ERROR: test_create_connection_ssl_failed_certificate (test.test_asyncio.test_ssl.TestSSL.test_create_connection_ssl_failed_certificate)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_asyncio/test_ssl.py", line 454, in test_create_connection_ssl_failed_certificate
    self.loop.run_until_complete(client(srv.addr))
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 664, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_asyncio/test_ssl.py", line 441, in client
    reader, writer = await asyncio.open_connection(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/streams.py", line 47, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 1126, in create_connection
    transport, protocol = await self._create_connection_transport(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/base_events.py", line 1159, in _create_connection_transport
    await waiter
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/sslproto.py", line 575, in _on_handshake_complete
    raise handshake_exc
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/asyncio/sslproto.py", line 557, in _do_handshake
    self._sslobj.do_handshake()
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/ssl.py", line 917, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1024)

test.pythoninfo:

os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-mini', release='22.5.0', version='Darwin Kernel Version 22.5.0: Thu Jun  8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103', machine='arm64')
platform.architecture: 64bit
platform.platform: macOS-13.4.1-arm64-arm-64bit
platform.python_implementation: CPython

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 3.1.1 30 May 2023
ssl.OPENSSL_VERSION_INFO: (3, 1, 0, 1, 0)
ssl.OP_ALL: 0x80000050
ssl.OP_NO_TLSv1_1: 0x10000000
ssl.SSLContext.maximum_version: -1
ssl.SSLContext.minimum_version: 771
ssl.SSLContext.options: 2186412112
ssl.SSLContext.protocol: 16
ssl.SSLContext.verify_mode: 2

ssl.default_https_context.maximum_version: -1
ssl.default_https_context.minimum_version: 771
ssl.default_https_context.options: 2186412112
ssl.default_https_context.protocol: 16
ssl.default_https_context.verify_mode: 2

ssl.stdlib_context.maximum_version: -1
ssl.stdlib_context.minimum_version: 771
ssl.stdlib_context.options: 2186412112
ssl.stdlib_context.protocol: 16
ssl.stdlib_context.verify_mode: 0

Linked PRs

The test expects ssl.SSLCertVerificationError exception:

with self.assertRaises(ssl.SSLCertVerificationError):

But a ssl.SSLError error is raised instead:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1024)

According to the error message:

ERR_GET_LIB(e) is ERR_LIB_SSL: SSL: is the error message
ERR_GET_REASON(e) is SSL_R_CERTIFICATE_VERIFY_FAILED: CERTIFICATE_VERIFY_FAILED is the error message

I'm not sure why PySSL_SetError() of Modules/_ssl.c doesn't go to the type = state->PySSLCertVerificationErrorObject; code path. I suppose that the difference between OpenSSL 3.0 and OpenSSL 3.1 is the err.ssl (sslsock->err.ssl) value. open OpenSSL 3.0, it's SSL_ERROR_SSL. Maybe on OpenSSL 3.1, it's different?

@pablogsal

@pablogsal: Would it be possible that I get a SSH access to this buildbot machine?

@gpshead

I believe @gpshead was the most recent person to update the error code handling for the various OpenSSL versions.

this error location can be narrowed between OpenSSL 3.1.1 and 3.09

It smells like some error reporting changes within OpenSSL that we just need to be aware of?

Building against openssl 3.1.1 on Linux I cannot reproduce this. Odd. This shouldn't be platform specific AFAICT.

You can trigger a PR test run on a subset of our stable bots via adding the !buildbot regex-of-bot-name text as comment to a PR.

this error was triggered by OpenSSL commit 00ce831152403589f7f0e558d5291de12bfedc5f

diff --git a/crypto/store/store_lib.c b/crypto/store/store_lib.c
index 5ff9278629..5ac7ff593e 100644
--- a/crypto/store/store_lib.c
+++ b/crypto/store/store_lib.c
@@ -114,13 +114,17 @@ OSSL_STORE_open_ex(const char *uri, OSSL_LIB_CTX *libctx, const char *propq,
         scheme = schemes[i];
         OSSL_TRACE1(STORE, "Looking up scheme %s\n", scheme);
 #ifndef OPENSSL_NO_DEPRECATED_3_0
+        ERR_set_mark();
         if ((loader = ossl_store_get0_loader_int(scheme)) != NULL) {
+            ERR_clear_last_mark();
             no_loader_found = 0;
             if (loader->open_ex != NULL)
                 loader_ctx = loader->open_ex(loader, uri, libctx, propq,
                                              ui_method, ui_data);
             else
                 loader_ctx = loader->open(loader, uri, ui_method, ui_data);
+        } else {
+            ERR_pop_to_mark();
         }
 #endif
         if (loader == NULL

@pablogsal

@pablogsal: Would it be possible that I get a SSH access to this buildbot machine?

I can try to set https://tmate.io/ because I don't have a stable IP infortunately.

pythongh-107077: Raise SSLCertVerificationError even if the error is …

gh-107077: Raise SSLCertVerificationError even if the error is set vi…

pythongh-107077: Raise SSLCertVerificationError even if the error is …

Revert "[3.10] pythongh-107077: Raise SSLCertVerificationError even i…

[3.10] Revert "[3.10] gh-107077: Raise SSLCertVerificationError even …

@pablogsal

Thanks for investigating and fixing this ssl issue @pablogsal!

Backports to 3.8 and 3.9 are still not merged.

[3.8] gh-107077: Raise SSLCertVerificationError even if the error is …

[3.9] gh-107077: Raise SSLCertVerificationError even if the error is …

They are merged now.

Raise SSLCertVerificationError even if the error is set via SSL_ERROR…

Navigation Menu

Repository navigation

test_asyncio: test_create_connection_ssl_failed_certificate() failed on ARM64 macOS 3.x buildbot #107077

Linked PRs

4 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

test_asyncio: test_create_connection_ssl_failed_certificate() failed on ARM64 macOS 3.x buildbot #107077

Description

Linked PRs

Activity

vstinner commented on Jul 25, 2023

vstinner commented on Jul 25, 2023

ned-deily commented on Jul 25, 2023

josephshen commented on Jul 25, 2023

gpshead commented on Jul 26, 2023

josephshen commented on Jul 26, 2023

pablogsal commented on Jul 27, 2023

4 remaining items

vstinner commented on Aug 16, 2023

serhiy-storchaka commented on Jan 31, 2024

ambv commented on Feb 21, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions