Repository navigation
OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower #95031
Description
Bug report
Example code in ubuntu20.04(openssl1.1) is much faster than ubuntu22.04(openssl3.x)
Not just speed, CPU occupancy ubuntu22.04(openssl3.x) is many times of ubuntu20.04(openssl1.1)
I'm not sure whether it's OpenSSL or Python adaptation problem
import socket
import ssl
import time
import certifi
hostname = 'www.python.org' # any support https hostname
times = 100
pem_where = certifi.where()
context = ssl.create_default_context()
verify_total_time = 0
for i in range(times):
with socket.create_connection((hostname, 443)) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
verify_start_time = time.time()
context.load_verify_locations(pem_where)
verify_total_time += time.time() - verify_start_time
ssock.version()
print(f"total {verify_total_time:.4f}, avg {verify_total_time/times:.4f}")in my environment with docker:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3321 root 20 0 304140 81148 12792 S 42.0 0.8 0:29.69 ipython (ubuntu22.04)
3850 root 20 0 203348 52632 11576 S 16.7 0.5 0:06.34 ipython (ubuntu20.04)
total 5.8634, avg 0.0586 (ubuntu22.04)
total 0.6753, avg 0.0068 (ubuntu20.04)
Your environment
- CPython versions tested on: 3.10.5
- Operating system and architecture: ubuntu20.04(openssl1.1) and ubuntu22.04(openssl3.0.2), build from source
- certifi==2022.6.15
Reacted by Hunter Fernandes, Oliver Mannion, Thomas Christlieb, ilotoki0804, Lars Holmberg, Jakub Boukal, TBoyBack, Greg Smethells, Ed Morley, 双草酸酯 and 1 more
Activity
tiran commentedon Jul 20, 2022
It is a problem in OpenSSL 3.0. Python upstream does not support OpenSSL 3.0 for good reasons. It has performance and backwards compatibility issue. On my system
load_verify_locationsis about 5 times slower when using system certificates.By the way you should not combine
ssl.create_default_context()with certifi. A default context already loads the system cert store.tiran commentedon Jul 20, 2022
fcfangcc commentedon Jul 20, 2022
Thansk.Example is separate from requests,httpx......
It seems that using ubuntu22 with Python is not a good idea, it`s default openssl3.
tiran commentedon Jul 20, 2022
I recommend that you raise a bug with OpenSSL. Their
SSL_CTX_set_default_verify_pathsandSSL_CTX_load_verify_locationsfunctions are much slower in 3.0 than in 1.1.1.tiran commentedon Jul 20, 2022
According to "perf', OpenSSL 3.0 is spending a lot of time in
pthread_rwlocklock/unlock followed by sa_doall, getrn, and several libcrypto string functions (ossl_lh_strcasehash, ossl_tolower, OPENSSL_strcasecmp, OPENSSL_sk_value).arhadthedev commentedon Jul 20, 2022
Fortunately, the issue is known (so no need to report it once more): openssl/openssl#16791 initial report and openssl/openssl#18814 pointing to a root issue.
[-]ssl module(load_verify_locations) with openssl3 there are performance problems[/-][+]OpenSSL 3.0 performance issue: SSLContext.set_default_verify_paths / load_verify_locations about 5x slower[/+]iritkatriel commentedon Sep 12, 2022
Should we close this as a third party issue?
risicle commentedon Sep 28, 2023
As an outsider, it appears to me that there's always going to be a risk of this becoming a severe performance bottleneck as long as python's
sslAPI doesn't expose the ability to re-use anX509_STORE- forcing the system's CA bundle to be re-parsed for every newSSLContext.ThomasChr commentedon Oct 5, 2023
This one bit me today. And it did bite quite hard!
Beginning with cPython 3.11.5 (on Windows) we're shipping OpenSSL 3 instead of 1.1
So when I installed 3.12.0 on a customer system it took a few minutes until I got a call. The system is blocked, one Python Process takes all of the CPU and no one can work anymore.
Turns out that my Python Process which uses 32 Threads and does nothing more then sending some simple numbers into the Internet took all of the CPU - instead of 20% as it was with Python 3.11. Thats a very bad performance regression.
One of my other processes has some more accurate numbers - before:
and after:
Pretty sure you could have cooked coffee on the cpu after that.
(With cProfile you could see most of the CPU time in
{method 'load_verify_locations' of '_ssl._SSLContext' objects})On the upper level I'm using the requests library in my code and there where two solutions to the problem:
verify=Falseto all of the requestsI used Method 2, but now I need to stay on Python 3.11.4 for all eternity. Hopefully this will be fixed some day soon.
Hope this post helps other people with the same problem. I will see if I'll add some info to the linked OpenSSL Issues also.
ThomasChr commentedon Oct 5, 2023
As a quick solution:
It seems that the time is spent when OpenSSL verifys a certificate - can't we try to cache this verification? In my code I'm making thousands of requests to the exact same host - I don't need to verify it every time.
It would be a solution for the user to add
verify=Falsebeginning with the second request to the same host himself - but that means he needs to find out the root course first, which is not an easy task.I'm a little bit sad that people will say that Python is dog-slow when really it isn't our fault. But saying "not our fault" won't help much here. Having a plan B would be great.
(I'm not entirely convinced that OpenSSL will fix this problem soon...)
risicle commentedon Oct 5, 2023
(Another workaround - if you're only connecting to one or a few hosts you may find that you're able to put together a custom, extremely minimal ca bundle with only the root cert(s) you need. This will be much faster to parse, though will have the same caveats as pinning certificates)
ThomasChr commentedon Oct 5, 2023
Also one can only use
verify=Truewhen sending important data like passwords or user data. Otherwiseverify=Falsewill do.This is not perfect but really testing the certificate in every connection won‘t be needed of the time.
tiran commentedon Oct 5, 2023
Python has a cache for certificate verification: SSLContext. The simplest solution for your performance problem is a single SSLContext object for all your TLS client connection. Most client application only need a single SSLContext object during their lifetime. You configure the SSLContext according to your security profile, load the trust anchors, and then pass the object to your connection function. Then you have to pay the price for CA loading just once. SSLContext is thread and async-safe.
If you are using requests or httpx, then you want to make use of
requests.Sessionorhttpx.Client, too. They enable HTTP connection pooling, which speeds up multiple requests to the same host a lot.ThomasChr commentedon Oct 6, 2023
@tiran Using a Requests Session was great advice and it speeds up things considerably. But still we're cooking the cpu quite good.
This is Python 3.11 without a Requests Session:
And Python 3.11 with a Requests Session:
This is Python 3.12 without a Requests Session:
And Python 3.12 with a Requests Session:
I could live with that, so Python 3.12 is not a no-go anymore - thanks a lot for your advice!
mm-matthias commentedon May 5, 2024
@tiran You've said here that
Is this official? If yes, can it be added to the documentation?
I am asking, because the slowness with
load_verify_locationspops up in downstream projects such as requests and botocore.Using a session to perform requests only helps so much, some issues prevail in all of these libraries and the only way to solve them seems to be to load the SSLContext just once and then share it across threads, urllib3's
HTTPConnectionPools and other places.So it is important to know if the SSLContext can be freely shared between threads, e.g. see this question in a requests PR to tackle the problem.
fireattack commentedon May 21, 2024
Others have mentioned how bad it is on Windows, but just to demonstrate it clearer here.
Using a slightly modified code based on this, loading
site-packages/certifi/cacert.pem(284KB) which is what urllib3 loads every time it starts a new SSL context, it is 44x slower here on my WIndows 10 computer.nhorman commentedon Jul 29, 2025
FWIW, This was just discussed on openssl/openssl#18814
The problem is twofold:
Debian/Ubuntu, somewhere in a recent LTS (22.04 I think), made a platform change in which a trusted ca was added as a symlink to /etc/ssl/certs. This had the effect of causing set_default_verify_paths to automatically load an parse a very large certificate chain, meaning performance comparisons between 20.02 (where this certificate was not linked) and 22.04 are not an apples to apples comparison. 22.04 and later distributions are triggering openssl to do alot more work in this environment.
Ubuntu 22.04 ships with openssl 3.0.13 which did have several performance issue that have been fixed in interim releases, but have not been pulled in by the ubuntu provided package, so those performance gains have not been recognized.
If you build openssl 3.5 or later, you should see performance on par with 1.1.1
GalaxySnail commentedon Oct 21, 2025
Would it be possible to upgrade the bundled OpenSSL to 3.5.x on Windows? OpenSSL 3.5 is the next LTS series after 3.0 (ref: https://openssl-library.org/source/). As previously discussed, this is a noticeable performance issue for HTTPS clients.
For example, on my machine, a single
httpx.Client()initialization takes about 5 seconds, and assl.create_default_context(cafile=certifi.where())takes about 2.5 seconds on the official python build from python.org. By contrast, on the same machine,ssl.create_default_context(cafile=certifi.where())only takes about 15 ms on both python3.12 with OpenSSL 3.5.2 from MSYS2 UCRT64 and python3.13 with OpenSSL 3.5.3 on Arch Linux.sergey-miryanov commentedon Oct 21, 2025
@GalaxySnail It is already done - #132339
zware commentedon Oct 21, 2025
It's not done, but is vaguely planned :)
sergey-miryanov commentedon Oct 21, 2025
Oops, sorry :)