Skip to content

Repository navigation

HTTP/2 connection implementation is currently flaky. #80

Closed
#82
Closed
HTTP/2 connection implementation is currently flaky.#80
#82

Description

@manatlan
manatlan
opened on May 25, 2019

Is it me ?
Or the simplest test fails:

 httpcore.Client().get("https://www.google.com")

outputs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/httpcore/concurrency.py", line 65, in read
    self.stream_reader.read(n), timeout.read_timeout
  File "/usr/lib/python3.7/asyncio/tasks.py", line 423, in wait_for
    raise futures.TimeoutError()
concurrent.futures._base.TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 114, in send
    request, stream=stream, ssl=ssl, timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection.py", line 53, in send
    request, stream=stream, timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 77, in send
    await response.read()
  File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 684, in read
    self._content = b"".join([part async for part in self.stream()])
  File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 684, in <listcomp>
    self._content = b"".join([part async for part in self.stream()])
  File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 695, in stream
    async for chunk in self.raw():
  File "/usr/local/lib/python3.7/dist-packages/httpcore/models.py", line 712, in raw
    async for part in self._raw_stream:
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 121, in body_iter
    event = await self.receive_event(stream_id, timeout)
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/http2.py", line 131, in receive_event
    data = await self.reader.read(self.READ_NUM_BYTES, timeout)
  File "/usr/local/lib/python3.7/dist-packages/httpcore/concurrency.py", line 68, in read
    raise ReadTimeout()
httpcore.exceptions.ReadTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/manatlan/Bureau/reqman/reqman.py", line 1414, in <module>
    httpcore.Client().get("https://www.google.com")
  File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 496, in get
    timeout=timeout,
  File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 472, in request
    timeout=timeout,
  File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 665, in send
    timeout=timeout,
  File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 287, in send
    allow_redirects=allow_redirects,
  File "/usr/local/lib/python3.7/dist-packages/httpcore/client.py", line 313, in send_handling_redirects
    request, stream=stream, ssl=ssl, timeout=timeout
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 117, in send
    self.active_connections.remove(connection)
  File "/usr/local/lib/python3.7/dist-packages/httpcore/dispatch/connection_pool.py", line 64, in remove
    del self.all[connection]
KeyError: <httpcore.dispatch.connection.HTTPConnection object at 0x7fadbf88e0f0>

btw, I can't catch the Exception ?!

try:
    httpcore.Client().get("https://www.google.com")
except concurrent.futures._base.TimeoutError:
    print(1)
except httpcore.exceptions.ReadTimeout:
    print(2)

Activity

  1. yeraydiazdiaz commented on May 25, 2019

    @yeraydiazdiaz
    yeraydiazdiaz
    on May 25, 2019
    Contributor

    Thanks for the report. TL;DR Google is using HTTP2 and returning a 400 error which creates an unexpected content which causes the ReadTimeout. I've tried with other URLs and it seems to work correctly.

    Looking deeper there are few things happening:

    • Firstly Google is returning a 400 response and resetting the stream, here are the raw events:
      <ResponseReceived stream_id:1, headers:[(b':status', b'400'), (b'content-type', b'text/html; charset=UTF-8'), (b'referrer-policy', b'no-referrer'), (b'content-length', b'1555'), (b'date', b'Sat, 25 May 2019 10:25:58 GMT')]>
  <StreamReset stream_id:1, error_code:ErrorCodes.PROTOCOL_ERROR, remote_reset:True>
    • The HTTP2 dispatcher attempts to get a body iterator for the content on content = self.body_iter(stream_id), which is then passed to the response.
    • Further on await response.read() the body_iter attempts to read events from the stream and yield data, however the StreamReset is the last event of the stream which is unhandled in the loop, so it attempts to read a new event from the stream and times out.
    • The ReadTimeout exception is caught in the try..except and closes the response in the finally block, the ReadTimeout bubbles up
    • The ConnectionPool.send method also has a try..except block which catches the timeout exception and attempts to remove the connection there, but the connection has already been removed on response.close so a KeyError is raised.

    I'll see if I can put together a couple of PRs for these issues.

  2. lovelydinosaur commented on May 25, 2019

    @lovelydinosaur
    lovelydinosaur
    on May 25, 2019
    Member

    Thanks folks. I should be able to review this on Tuesday.

  3. lovelydinosaur commented on May 25, 2019

    @lovelydinosaur
    lovelydinosaur
    on May 25, 2019
    Member

    Something that’d be really useful for us with debugging would be configuration on the client to determine which HTTP protocols should be supported. Then if we get something like this we can easily narrow it down to eg. “Is this an issue in the HTTP/2 implementation”

  4. lovelydinosaur commented on May 25, 2019

    @lovelydinosaur
    lovelydinosaur
    on May 25, 2019
    Member

    It's specifically an issue with HTTP/2 (doesn't reproduce if I switch the connection to force HTTP/1). Doesn't reproduce on all sites (eg. I'd be testing against the HTTP/2 implementation against https://www.example.com.)

  5. lovelydinosaur
    changed the title [-]httpcore.Client().get("https://www.google.com") doesn't work[/-] [+]HTTP/2 connection implementation is currently flaky.[/+] on May 25, 2019

  6. manatlan commented on May 26, 2019

    @manatlan
    manatlan
    on May 26, 2019
    Author

    thanks in advance ...
    I will use httpcore in https://github.com/manatlan/reqman ...

    Hope you release a httpcore on pypi with cookies soon

  7. lovelydinosaur
    closed this as completedin #82on May 28, 2019

  8. lovelydinosaur commented on May 28, 2019

    @lovelydinosaur
    lovelydinosaur
    on May 28, 2019
    Member

    Now resolved in master.

    >>> import httpcore
>>> httpcore.get('https://www.google.com/')
<Response(200, 'OK')>

  9. manatlan commented on May 28, 2019

    @manatlan
    manatlan
    on May 28, 2019
    Author

    great, thanks !
    Are you planning to release a pypi version with this fix ?

  10. akjanik commented on Jun 6, 2019

    @akjanik
    akjanik
    on Jun 6, 2019

    @tomchristie I am trying to use the lib with Google Cloud services which use HTTP/2 and got hit by that, while trying to authenticate to https://www.googleapis.com/oauth2/v4/token Installing from git resolves the problem, would be it possible to do (minor?) release anytime soon?

  11. lovelydinosaur commented on Jun 6, 2019

    @lovelydinosaur
    lovelydinosaur
    on Jun 6, 2019
    Member

    Gotcha. Released as 0.3.1. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions