Skip to content

Repository navigation

closedResourceError intermittently when using httpx asyncclient to send the request #2337

Closed
Closed
closedResourceError intermittently when using httpx asyncclient to send the request#2337

Description

@jainaj81
jainaj81
opened on Aug 12, 2022

we are observing closedResourceError when sending multiple requests using httpx async client. httpx version used is 0.19.0. is this issue known and fixed in any later version?

Logs:
resp = await asyncio.gather(*[self.async_send_request(method,self.dest_host,self.dest_port,uri,i,payload=payload,headers=headers) for i in range(int(times) )])
File "/env/lib64/python3.9/site-packages/ocnftest_lib/scp_Pcf_client.py", line 175, in async_send_request
return await self.conn.request(method,"http://{}:{}{}".format(self.dest_host,self.dest_port,uri),data=payload,headers=headers,allow_redirects=False)
File "/env/lib64/python3.9/site-packages/httpx/_client.py", line 1494, in request
response = await self.send(
File "/env/lib64/python3.9/site-packages/httpx/_client.py", line 1586, in send
response = await self._send_handling_auth(
File "/env/lib64/python3.9/site-packages/httpx/_client.py", line 1616, in _send_handling_auth
response = await self._send_handling_redirects(
File "/env/lib64/python3.9/site-packages/httpx/_client.py", line 1655, in _send_handling_redirects
response = await self._send_single_request(request, timeout)
File "/env/lib64/python3.9/site-packages/httpx/_client.py", line 1699, in _send_single_request
) = await transport.handle_async_request(
File "/env/lib64/python3.9/site-packages/httpx/_transports/default.py", line 281, in handle_async_request
) = await self._pool.handle_async_request(
File "/env/lib64/python3.9/site-packages/httpcore/_async/connection_pool.py", line 234, in handle_async_request
response = await connection.handle_async_request(
File "/env/lib64/python3.9/site-packages/httpcore/_async/connection.py", line 148, in handle_async_request
return await self.connection.handle_async_request(
File "/env/lib64/python3.9/site-packages/httpcore/_async/http2.py", line 165, in handle_async_request
return await h2_stream.handle_async_request(
File "/env/lib64/python3.9/site-packages/httpcore/_async/http2.py", line 339, in handle_async_request
await self.send_headers(method, url, headers, has_body, timeout)
File "/env/lib64/python3.9/site-packages/httpcore/_async/http2.py", line 398, in send_headers
await self.connection.send_headers(self.stream_id, headers, end_stream, timeout)
File "/env/lib64/python3.9/site-packages/httpcore/_async/http2.py", line 274, in send_headers
await self.socket.write(data_to_send, timeout)
File "/env/lib64/python3.9/site-packages/httpcore/_backends/anyio.py", line 77, in write
return await self.stream.send(data)
File "/env/lib64/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1116, in send
raise ClosedResourceError
anyio.ClosedResourceError

Sample code:
async def send_request(self,payload,times,method,uri,headers):
resp = await asyncio.gather(*[self.async_send_request(method,self.dest_host,self.dest_port,uri,i,payload=payload,headers=headers) for i in range(int(times) )])
await self.conn.aclose()
log.logger.info("[+] @{} {} {}".format(method,resp[int(times)-1].status_code,uri))
return resp[int(times)-1]

async def async_send_request(self,method,dest_host,dest_port,uri,times,payload=None,headers=None):
  if self.security == 'secure':
      return await self.conn.request(method,"https://{}:{}{}".format(self.dest_host,self.dest_port,uri),data=payload,headers=headers,allow_redirects=False)
  else:
      return await self.conn.request(method,"http://{}:{}{}".format(self.dest_host,self.dest_port,uri),data=payload,headers=headers,allow_redirects=False)

def send_message:
self.conn = httpx.AsyncClient(http2=True,http1=False,proxies=self.proxies,timeout=10)
response = asyncio.run(self.send_request(payload,times,method,uri,headers))

Activity

  1. blazing-gig commented on Oct 20, 2022

    @blazing-gig
    blazing-gig
    on Oct 20, 2022

    @jainaj81 I faced the same issue as well. I think this has to do something with using asyncio.gather() maybe. Would help greatly if the core team noticed this. Happy to contribute if needed. Thanks.

  2. lovelydinosaur commented on Nov 8, 2022

    @lovelydinosaur
    lovelydinosaur
    on Nov 8, 2022
    Member

    The first step here is that we would like anyio.ClosedResourceError to be properly mapped to httpx.WriteError.

    I can see in httpcore that we're mapping anyio.BrokenResourceError properly, but not anyio.ClosedResourceError

    See the AnyIO docs https://anyio.readthedocs.io/en/stable/api.html#exceptions

    It's clear to me how to resolve that issue in the httpcore codebase, but I'd like to be able to test it properly too. Here's what I was trying to start with...

    server.py:

    async def app(scope, receive, send):
    assert scope['type'] == 'http'

    more_body = True
    while more_body:
        msg = await receive()
        more_body = msg.get('more_body')

    await send({
        'type': 'http.response.start',
        'status': 200,
        'headers': [
            [b'content-type', b'text/plain'],
        ],
    })
    await send({
        'type': 'http.response.body',
        'body': b'Hello, world!',
    })

    client.py:

    import httpx
import asyncio


async def upload():
    for chunk in range(1_000_000):
        yield b'xxx' * 1_000


async def main():
    async with httpx.AsyncClient() as client:
        await client.post("http://127.0.0.1:8000", content=upload())


asyncio.run(main())

    Then...

    $ uvicorn server:app
    $ python client.py

    And Ctrl-C twice on the server while the client is still uploading.

    Which results in...

    $ python client.py 
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/selector_events.py", line 870, in write
    n = self._sock.send(data)
BrokenPipeError: [Errno 32] Broken pipe

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
    yield
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/backends/asyncio.py", line 49, in write
    await self._stream.send(item=buffer)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/anyio/_backends/_asyncio.py", line 1297, in send
    raise self._protocol.exception
anyio.BrokenResourceError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
    yield
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
    raise exc
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 75, in handle_async_request
    await self._send_request_body(**kwargs)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 128, in _send_request_body
    await self._send_event(event, timeout=timeout)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 137, in _send_event
    await self._network_stream.write(bytes_to_send, timeout=timeout)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/backends/asyncio.py", line 49, in write
    await self._stream.send(item=buffer)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc)
httpcore.WriteError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "client.py", line 15, in <module>
    asyncio.run(main())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 583, in run_until_complete
    return future.result()
  File "client.py", line 12, in main
    await client.post("http://127.0.0.1:8000", content=upload())
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1855, in post
    extensions=extensions,
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1618, in send
    history=[],
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    history=history,
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tomchristie/Temp/venv/lib/python3.7/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.WriteError

    That's not what we need to test this error case.

    I think in order to test this I need a server example working just at the socket layer (rather than with an ASGI web server) so I can properly push the various cases.

    Let's use trio to do the server socket programming instead...

    server.py:

    import trio


async def handler(stream):
    # Read some of the request, then just close the socket.
    for x in range(10):
        data = await stream.receive_some()
    await stream.aclose()


async def main():
    await trio.serve_tcp(handler, 8000)


trio.run(main)

    But I'm still seeing this map to a BrokenResourceError, so I'm not clear how to test the ClosedResourceError at this point.

    Looking at Trio's docs helps here...

    https://trio.readthedocs.io/en/stable/reference-core.html#trio.BrokenResourceError

    For example, you might get this if you try to send data on a stream where the remote side has already closed the connection.

    You don’t get this error if you closed the resource – in that case you get ClosedResourceError.

    Ahhh... to reproduce this we need to client to close the connection while it's still performing an upload.

    I'm not quite sure how to do that.

    Saving my progress so far.

  3. lovelydinosaur commented on Nov 8, 2022

    @lovelydinosaur
    lovelydinosaur
    on Nov 8, 2022
    Member

    @jainaj81 If you format and clean up your example code into a runnable example that would help me progress the issue.

  4. lovelydinosaur commented on Nov 30, 2022

    @lovelydinosaur
    lovelydinosaur
    on Nov 30, 2022
    Member

    My working assumption here is that you've got a bug in the client side, where you're closing the client before you've finished with all the connections.

    That would result in an anyio.closedResourceError, or once encode/httpcore#631 is merged it'd result in a neater httpx.ReadError or httpx.WriteError.

    I'm going to close this off on that assumption. I'm happy to reopen the ticket if a clearer reproducible example is provided.

  5. lovelydinosaur
    closed this as completedon Nov 30, 2022

  6. KnownBlackHat commented on Oct 21, 2023

    @KnownBlackHat
    KnownBlackHat
    on Oct 21, 2023

    i m facing same error repeatedly
    this is sample code: https://paste.disnake.dev/?id=1697879915118304
    this is error: https://paste.disnake.dev/?id=169788019594034

    And this is how i m invoking it
    image

    Summary:
    whenever a request is sent to a link which results 404 or any other status code then client also gets closed and i don't think that kind of behavior should be strictly implemented there. there should be an param in client to set whether to close the client on error or not

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions