You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HTTP/2 higher number of timeouts compared with 1.1 #841
Thanks, the results in the gist are very enlightening as to the magnitude of this issue... As you noted in #832, there might be an outstanding race condition or bug we fixed for HTTP/1.1 in the past but not HTTP/2. I guess we’ll have to dig into the HTTP/2 implementation to find out... If you’d like you can turn on TRACE logs and see if there’s anything odd there:
I notice the KeyError’s are exactly 5s apart from one another. This is both the default connect timeout and the keepalive period for connections. So are they caused by our keepalive checking mechanism? Or is it a retry mechanism? Or is it what should have been a ConnectTimeout whose handling resulted in another error?
I think the KeyError's are due to #817 and should have been timeouts? Don't know for sure. But looks like all the connections actually raised an error, but due to #817 KeyError was raised
Wait. 5 is also the number of requests fired off in a batch. I notice the snippet fires them off perfectly in parallel via asyncio.gather(). I remembered this has caused issues in the past for HTTP/2.
Can you:
Add a random jitter delay to each request and see if that has any effect?
Change the batch size and connect timeout to distinct values that are not 5? Eg 3 and 4. That way we can better see what error is caused by what mechanism (parallel Requests, host connection, keepalive expiry, etc.)
Update: now with errors :/ both connect timeout and <class 'httpcore._exceptions.ProtocolError'>: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSE
And now: <class 'httpcore._exceptions.WriteError'>: [Errno 104] Connection reset by peer to a central microservice that everyone connects, and only the one with the httpcore interface receive this error.
Also: ProtocolError: Invalid input ConnectionInputs.RECV_HEADERS in state ConnectionState.CLOSED
@victoraugustolls Okay, so let's take these one at a time, in a bit more detail. They're generally looking roughly like valid behaviors that just need some neater error handling.
I'd suggest starting by opening an issue on httpcore with one of these, complete with a full traceback, plus a bit of information about what server you're using and other potentiatially useful infor for replication, so that we can work through the case comprehensively.
Once we've done that let's have open issues for each of these seperately on httpcore, and work through them fully one at a time.
Activity
florimondmanca commentedon Mar 2, 2020
Thanks, the results in the gist are very enlightening as to the magnitude of this issue... As you noted in #832, there might be an outstanding race condition or bug we fixed for HTTP/1.1 in the past but not HTTP/2. I guess we’ll have to dig into the HTTP/2 implementation to find out... If you’d like you can turn on TRACE logs and see if there’s anything odd there:
victoraugustolls commentedon Mar 2, 2020
@florimondmanca updated gist with results from running with
httpxprofof yours! Will enable trace and post it there.victoraugustolls commentedon Mar 2, 2020
It seems to get stuck here:
TRACE: release_connection connection=HTTPConnection(origin=Origin(scheme='https' host='www.google.com' port=443))victoraugustolls commentedon Mar 2, 2020
Traces are too long and I can't upload files to gist, so I will upload them here:
With HTTP/2 enabled -> trace.txt
With HTTP/2 disabled -> trace_2.txt
victoraugustolls commentedon Mar 2, 2020
But I think the interesting bits are before and after the exceptions:
victoraugustolls commentedon Mar 2, 2020
I tried to find something hanging, but no luck so far...
florimondmanca commentedon Mar 2, 2020
I notice the KeyError’s are exactly 5s apart from one another. This is both the default connect timeout and the keepalive period for connections. So are they caused by our keepalive checking mechanism? Or is it a retry mechanism? Or is it what should have been a ConnectTimeout whose handling resulted in another error?
victoraugustolls commentedon Mar 2, 2020
I think the KeyError's are due to #817 and should have been timeouts? Don't know for sure. But looks like all the connections actually raised an error, but due to #817 KeyError was raised
florimondmanca commentedon Mar 2, 2020
Wait. 5 is also the number of requests fired off in a batch. I notice the snippet fires them off perfectly in parallel via
asyncio.gather(). I remembered this has caused issues in the past for HTTP/2.Can you:
florimondmanca commentedon Mar 2, 2020
As for #817, does #817 (comment) suppress the KeyError for you as well? If so, what do TRACE logs look like with this patch?
victoraugustolls commentedon Mar 2, 2020
Tried with
asyncio.gather()with 10 requests in parallel and this was the output:victoraugustolls commentedon Mar 2, 2020
With 3 requests in parallel:
victoraugustolls commentedon Mar 2, 2020
added
await asyncio.sleep(random.random())before theclient.get()and the result (for 10 parallel requests) was:victoraugustolls commentedon Mar 2, 2020
Will try #817 fix next
victoraugustolls commentedon Mar 2, 2020
If I did this right:
The output was (without jitter):
victoraugustolls commentedon Mar 2, 2020
With a timeout value of 10.0:
victoraugustolls commentedon Mar 2, 2020
It appears that the time between errors is indeed the timeout time (read timeout)
[-]HTTP/2 higher timeouts compared with 1.1[/-][+]HTTP/2 higher number of timeouts compared with 1.1[/+]victoraugustolls commentedon Mar 6, 2020
@florimondmanca @tomchristie excellent news here! I had 0 timeouts after retrying the tests here multiple times with the
httpcore-interfacebranch!lovelydinosaur commentedon Mar 6, 2020
Fantastic! 👍
victoraugustolls commentedon Mar 9, 2020
Just deployed a version of the service using the branch with
httpcoreinterface. So far so good, 0 errors. Will post and update in the end of the day.victoraugustolls commentedon Mar 9, 2020
Update: now with errors :/ both connect timeout and
<class 'httpcore._exceptions.ProtocolError'>: can't handle event type ConnectionClosed when role=SERVER and state=SEND_RESPONSEvictoraugustolls commentedon Mar 9, 2020
And now:
<class 'httpcore._exceptions.WriteError'>: [Errno 104] Connection reset by peerto a central microservice that everyone connects, and only the one with thehttpcoreinterface receive this error.Also:
ProtocolError: Invalid input ConnectionInputs.RECV_HEADERS in state ConnectionState.CLOSEDAlso:
ProtocolError: <ConnectionTerminated error_code:ErrorCodes.NO_ERROR, last_stream_id:1999, additional_data:None>lovelydinosaur commentedon Mar 10, 2020
@victoraugustolls Okay, so let's take these one at a time, in a bit more detail. They're generally looking roughly like valid behaviors that just need some neater error handling.
I'd suggest starting by opening an issue on
httpcorewith one of these, complete with a full traceback, plus a bit of information about what server you're using and other potentiatially useful infor for replication, so that we can work through the case comprehensively.Once we've done that let's have open issues for each of these seperately on
httpcore, and work through them fully one at a time.yeraydiazdiaz commentedon May 13, 2020
Closing as issues are reflected in httpcore encode/httpcore#45 and encode/httpcore#46.
For the record, I just tested the original gist against httpx==0.13.0.dev2 and got the same result for http1.1 and http2.
victoraugustolls commentedon May 13, 2020
Nice! Will update on my end (and my application) to see the result!