Potential ValueError when used with high number of concurrent coros

@tomchristie

Escalating discussion #2043 into an issue. Doesn't currently have a reproducible example, but we've got a confirmation on it, so.

Discussed in #2043

^{Originally posted by pssolanki111 January 22, 2022}
I came across this exception when I make the request to an API and after a bit of digging on my end, this might be a potential issue in the lib.

to start, here is the exception traceback as recorded

Traceback (most recent call last):
  File "/home/pssolanki/projects/tests/lib_test/test.py", line 86, in get_market_cap
    response = await self.ref_client.get_ticker_details_v3(stock, date)
  File "/home/pssolanki/projects/openSource/polygon/polygon/reference_apis/reference_api.py", line 875, in get_ticker_details_v3
    _res = await self._get_response(_path, params=_data)
  File "/home/pssolanki/projects/openSource/polygon/polygon/base_client.py", line 240, in _get_response
    _res = await self.session.request('GET', self.BASE + path, params=params)
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_client.py", line 1513, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_client.py", line 1614, in send
    raise exc
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_client.py", line 1608, in send
    await response.aread()
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_models.py", line 1662, in aread
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_models.py", line 1662, in <listcomp>
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_models.py", line 1678, in aiter_bytes
    async for raw_bytes in self.aiter_raw():
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_models.py", line 1740, in aiter_raw
    await self.aclose()
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_models.py", line 1753, in aclose
    await self.stream.aclose()
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_client.py", line 151, in aclose
    await self._stream.aclose()
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 230, in aclose
    await self._httpcore_stream.aclose()  # type: ignore
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 336, in aclose
    await self._pool.response_closed(self._status)
  File "/home/pssolanki/projects/tests/lib_test/venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 270, in response_closed
    self._requests.remove(status)
ValueError: list.remove(x): x not in list

Now following the traceback, it looks like for some reason httpcore decides to remove an item an item from one of their internal structures BUT the high level API as seen at the top of traceback was just a simple http request.

The use case where it occured

a highly concurrent full async setup. Spawned many concurrent coros which hit the polygon.io API (using polygon package).

What do we think of these observations. Happy to provide more information. Creating a MCVE can be tricky but i can attempt if need. @tomchristie

Thanks :)

For reference, a version of what I saw that triggered this is here: #2043 (reply in thread)

Here's an example that'll raise the same traceback...

import httpx
import trio


async def download(client, url):
    print("started download")
    response = await client.get(url)
    print(response)


async def close_client(client):
    await trio.sleep(1)
    print("close client")
    await client.aclose()


async def main():
    async with httpx.AsyncClient() as client:
        async with trio.open_nursery() as nursery:
            nursery.start_soon(download, client, "https://speed.hetzner.de/100MB.bin")
            nursery.start_soon(close_client, client)


trio.run(main)

Which results in...

python example.py 
started download
close_client
Traceback (most recent call last):
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1601, in send
    await response.aread()
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1662, in aread
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1662, in <listcomp>
    self._content = b"".join([part async for part in self.aiter_bytes()])
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1678, in aiter_bytes
    async for raw_bytes in self.aiter_raw():
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1732, in aiter_raw
    async for raw_stream_bytes in self.stream:
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 145, in __aiter__
    async for chunk in self._stream:
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 239, in __aiter__
    async for part in self._httpcore_stream:
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 328, in __aiter__
    async for part in self._stream:
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 286, in __aiter__
    async for chunk in self._connection._receive_response_body(**kwargs):
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 160, in _receive_response_body
    event = await self._receive_event(timeout=timeout)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/http11.py", line 173, in _receive_event
    self.READ_NUM_BYTES, timeout=timeout
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/backends/trio.py", line 27, in read
    return await self._stream.receive_some(max_bytes=max_bytes)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_ssl.py", line 679, in receive_some
    return await self._retry(self._ssl_object.read, max_bytes)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_ssl.py", line 585, in _retry
    data = await self.transport_stream.receive_some()
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_highlevel_socket.py", line 136, in receive_some
    return await self.socket.recv(max_bytes)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_highlevel_socket.py", line 31, in _translate_socket_errors_to_stream_errors
    raise trio.ClosedResourceError("this socket was already closed") from None
trio.ClosedResourceError: this socket was already closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 24, in <module>
    trio.run(main)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 1932, in run
    raise runner.main_task_outcome.error
  File "example.py", line 21, in main
    nursery.start_soon(close_client, client)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/trio/_core/_run.py", line 815, in __aexit__
    raise combined_error_from_nursery
  File "example.py", line 7, in download
    response = await client.get(url)
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1738, in get
    extensions=extensions,
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1506, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 1606, in send
    await response.aclose()
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_models.py", line 1753, in aclose
    await self.stream.aclose()
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_client.py", line 151, in aclose
    await self._stream.aclose()
  File "/Users/tomchristie/GitHub/encode/httpx/httpx/_transports/default.py", line 244, in aclose
    await self._httpcore_stream.aclose()  # type: ignore
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 336, in aclose
    await self._pool.response_closed(self._status)
  File "/Users/tomchristie/GitHub/encode/httpx/venv/lib/python3.7/site-packages/httpcore/_async/connection_pool.py", line 270, in response_closed
    self._requests.remove(status)
ValueError: list.remove(x): x not in list

@ahopkins

@ahopkins - I've got encode/httpcore#491 to resolve this now.

I don't have permissions to merge it without a review. Would you be up for me inviting you as a contributor so that you've got review/merge permissions on that PR? (Then we can push ahead with a bug fix release once it's all confirmed)

Will do happily.

This issue also occurs when cancelling a request in flight, for example with:

import asyncio
import sys

import httpx


async def main():
    try:
        async with httpx.AsyncClient() as client:
            response = await asyncio.wait_for(slow_request(client), 1)
    except Exception as err:
        print(err.__class__.__name__, err)
    await asyncio.sleep(0)


async def slow_request(client):
    return await client.get('https://httpbin.org/delay/10')


if __name__ == '__main__':
    if sys.version_info < (3, 7):
        print('Using loop.run_until_complete')
        loop = asyncio.get_event_loop()
        loop.run_until_complete(main())
    else:
        print('Using asyncio.run')
        asyncio.run(main())

With encode/httpcore#491, the above code now raises a RuntimeError in Python versions 3.6, 3.8+, but a TimeoutError in Python 3.7.

The issue there is that CancelledError was still a subclass of Exception in 3.7, so the in-flight request is cleaned up as part of the except block at connection_pool.py Line 242.

Also note that while a RuntimeError is raised in 3.6, the same issue above is still present. In 3.6, however, aclose() manages to win the pool lock first, so the RuntimeError masks that behavior.

Bringing these in line with each other is going to depend on which behavior you want.

Explicitly excepting that asyncio.CancelledError and re-raising is a quick fix to make sure response_closed is not called in Python 3.6 and 3.7.

@aworkin

That's a great observation, thanks @aworkin. I think the sensible tack there is to inherit BaseException instead of Exception. (Pretty reasonable for resource-closing actions, anyways)

I like that idea, it ensures the request is closed cleanly for Python 3.7+ and actually avoids the RuntimeError there.

Python 3.6 still raises RuntimeError though. From some cursory tests it looks like the handling order is different for 3.6. The client's __aexit__ is called before the CancelledError is handled.

Looks like the 3.6 issue is an occurrence of https://bugs.python.org/issue32751. Not sure how you'd work around that issue, or if it's even this library's concern at that point.

@aworkin

@aworkin - Doesn't look like there's anything we can (reasonably) do about the 3.6 case.

I've bumped the connection pool response_closed handling up to BaseException in encode/httpcore#491 - thanks again for pointing that out.

@tomchristie

@tomchristie following up on this issue. I was getting this same issue with a high number of coros, a few days after i originally started this discussion.

I saw httpcore/#491. Is that merge confirmed to resolve this issue?

@tomchristie

actually nvm, i see the change @tomchristie

Navigation Menu

Repository navigation

Potential ValueError when used with high number of concurrent coros #2047

Discussed in #2043

The use case where it occured

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Potential ValueError when used with high number of concurrent coros #2047

Description

Discussed in #2043

The use case where it occured

Activity

ahopkins commented on Jan 26, 2022

lovelydinosaur commented on Jan 27, 2022

lovelydinosaur commented on Jan 27, 2022

ahopkins commented on Jan 27, 2022

aworkin commented on Jan 27, 2022

aworkin commented on Jan 27, 2022

lovelydinosaur commented on Jan 27, 2022

aworkin commented on Jan 27, 2022

aworkin commented on Jan 27, 2022

lovelydinosaur commented on Jan 28, 2022

pssolanki111 commented on Feb 1, 2022

pssolanki111 commented on Feb 1, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions