httpx does not wrap asyncio.exceptions.TimeoutError as TimeoutException

Checklist

The bug is reproducible against the latest release and/or master.
There are no similar issues or pull requests to fix it yet.

Describe the bug

httpx does not wrap asyncio.exceptions.TimeoutError as TimeoutException.

the code below raised exception when connecting to proxies failed.

Code:

import asyncio
import time

import httpx

def fastest_url(urls, **kwargs):
    async def test():
        async with httpx.AsyncClient(**kwargs) as client:

            async def req(url):
                start = time.time()
                try:
                    resp = await client.get(url)
                    resp.raise_for_status()
                except (
                    httpx.TimeoutException,
                    httpx.NetworkError,
                    httpx.HTTPStatusError,
                ) as error:
                    return {
                        "url": url,
                        "cost": float("inf"),
                        "error": error,
                    }
                return {"url": url, "cost": time.time() - start, "error": None}

            fs = []
            for u in urls:
                fs.append(req(u))

            for _, coro in enumerate(asyncio.as_completed(fs)):
                result = await coro
                if result["error"]:
                    continue
                return result

    return asyncio.run(test())

Traceback:

  ...
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 609, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/app/setting/search_engines/url_test.py", line 14, in req
    resp = await client.get(url)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1548, in get
    return await self.request(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1371, in request
    response = await self.send(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1406, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1444, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1476, in _send_handling_redirects
    response = await self._send_single_request(request, timeout)
  File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1502, in _send_single_request
    (status_code, headers, stream, ext,) = await transport.arequest(
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 124, in arequest
    return await self._tunnel_request(
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http_proxy.py", line 232, in _tunnel_request
    await proxy_connection.start_tls(host, timeout)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 161, in start_tls
    self.socket = await self.connection.start_tls(hostname, timeout)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 87, in start_tls
    self.socket = await self.socket.start_tls(hostname, self.ssl_context, timeout)
  File "/usr/local/lib/python3.9/site-packages/httpcore/_backends/asyncio.py", line 108, in start_tls
    transport = await asyncio.wait_for(
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 489, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

To reproduce

Expected behavior

asyncio.exceptions.TimeoutError should be wrapped by httpx when connecting to proxies, users should not know the internal dependecies.

Actual behavior

unexpected error raised

Debugging material

Environment

OS: Debian GNU/Linux 10 (buster)
Python version: Python 3.9.0
HTTPX version: 0.16.1
Async environment: asyncio
HTTP proxy: yes
Custom certificates: no

Additional context

Hey!

Sure, if reproducible against a simpler script (eg hitting https://httpbin.org/delay), then the key for fixing this is to submit a pull request to https://github.com/encode/httpcore that updates the asyncio backend. I assume right now it only catches the builtin TimeoutException and misses asyncio's. If that's actually already covered then we'd need to dig deeper, maybe there's a missed wrapping step in the proxies code.

Thanks for the clear report. 📝 👍

From the traceback, it's clear that our start_tls method in the asyncio backend, is missing a wrapper to catch and map timeout exceptions...

https://github.com/encode/httpcore/blob/db81ed09592bac8ce740b3be74cf99c86a68c104/httpcore/_backends/asyncio.py#L110

In contrast, read operations (for example) are properly wrapped up within a block that maps timeout exceptions...

https://github.com/encode/httpcore/blob/db81ed09592bac8ce740b3be74cf99c86a68c104/httpcore/_backends/asyncio.py#L135

I noticed that there is a httpx._exceptions.ProxyError, it is used for handling proxy exceptions.

I think we should use it for this case or create a new exception class because when the real connection fails, I may ignore it and when the proxy connection fails, I will give a warning.

By the way, writing the test I found, that ~~the same~~ another error affects anyio backend (link):

        exc_map = {
            OSError: ConnectError,
            TimeoutError: ConnectTimeout,
            BrokenResourceError: ConnectError,
        }

        with map_exceptions(exc_map):
            async with anyio.fail_after(connect_timeout):
                stream: anyio.abc.ByteStream
                stream = await anyio.connect_tcp(
                    unicode_host, port, local_host=local_address
                )
                if ssl_context:
                    stream = await TLSStream.wrap(
                        stream,
                        hostname=unicode_host,
                        ssl_context=ssl_context,
                        standard_compatible=False,
                    )

As assert isinstance(TimeoutError(), OSError), the above map_exceptions throws ConnectError in case of any timeout (at least in await TLSStream.wrap)

Navigation Menu

Repository navigation

httpx does not wrap asyncio.exceptions.TimeoutError as TimeoutException #1387

Checklist

Describe the bug

To reproduce

Expected behavior

Actual behavior

Debugging material

Environment

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

httpx does not wrap asyncio.exceptions.TimeoutError as TimeoutException #1387

Description

Checklist

Describe the bug

To reproduce

Expected behavior

Actual behavior

Debugging material

Environment

Additional context

Activity

florimondmanca commented on Nov 10, 2020

lovelydinosaur commented on Nov 11, 2020

elonzh commented on Nov 11, 2020

cdeler commented on Nov 13, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions