Repository navigation
test_ssl: ConnectionHandler shuts down ThreadedEchoServer #126500
Description
Recently, test_ssl has been failing intermittently but frequently on macOS buildbots, see for example here (but look at the first failure; the re-run is #126499).
If read() in a ThreadedEchoServer's ConnectionHandler thread raises OSError (except ConnectionError), the ConnectionHandler shuts down the entire server, preventing further connections.
As far as I can see, this is done to avoid the server thread getting stuck, forgotten, in its accept loop. However, since 2011 (5b95eb9) the server is used as a context manager, and its __exit__ does stop() and join().
(I'm not sure if we always used with since that commit, but currently we do.)
The most common failure I saw is the following. Note that it's the third connect call in this test: presumably, if the server thread started shutting down after the first connect it had enough time by now. But sometimes the same happens with the second connect, in this test or another one.
Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_ssl.py", line 1897, in test_connect_with_context
s.connect(self.server_addr)
~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/ssl.py", line 1405, in connect
self._real_connect(addr, False)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/ssl.py", line 1392, in _real_connect
super().connect(addr)
~~~~~~~~~~~~~~~^^^^^^
ConnectionRefusedError: [Errno 61] Connection refused
I'll send a PR soon.
Linked PRs
- gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ #126503
- [3.13] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ (GH-126503) #126571
- [3.12] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ (GH-126503) #126572
- [3.11] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ (GH-126503) (GH-126572) #133148
- [3.10] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in ConnectionHandler; rely on __exit__ (GH-126503) (GH-126572) #134030
Activity
pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError i…
gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError in Conn…
encukou commentedon Nov 7, 2024
I'll watch the buildbot for a day or two before backporting the fix.
pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError i…
pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError i…
[3.13] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError …
[3.12] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError …
pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError i…
pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError i…
stratakis commentedon Apr 14, 2025
We started noticing these failures consistently on Python 3.11 and 3.10 when Fedora Rawhide updated to OpenSSL 3.5. It's fairly easy to reproduce, just compile python3.11 or 3.10 on a rawhide machine and run test_ssl.
Pythons <= 3.9 do not seem affected.
You can also see the failures on the rawhide buildbots e.g. here: https://buildbot.python.org/all/#/builders/965/builds/1163
Not exactly certain of the root cause, however the relevant PR resolves the issue.
Given that 3.10 and 3.11 are on a security fix mode only, I think this would be a good candidate to backport as it resolves the issue with the latest OpenSSL. Similarly a different issue with OpenSSL was backported as well to these releases:
#127361
@encukou what do you think?
[3.12] pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OS…
encukou commentedon Apr 29, 2025
Not my call; let me prepare the backport and ask the release managers.
Thanks for the ping; this slipped from my radar!
[3.11] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError …
[3.12] pythongh-126500: test_ssl: Don't stop ThreadedEchoServer on OS…
[3.10] gh-126500: test_ssl: Don't stop ThreadedEchoServer on OSError …