httpcore update from 0.12.2 to 0.12.3 results in httpcore.CloseError

Using httpx 0.16.1 + Python 3.8.6

I'm not quite sure if this is a bug, but updating httpcore results in various httpcore.CloseError exceptions being raised on my server, it's timing dependend though. I can't reproduce outside of my application, but it comes down to the following code:

(edit: see #1459 (comment) below for a real reproducer)

import asyncio
import httpx

async def main():
    url = "https://github.com/msys2/MINGW-packages/releases/download/srcinfo-cache/srcinfo.json"
    async with httpx.AsyncClient() as client:
        # XXX: github doesn't support redirects with HEAD and returns bogus headers
        async with client.stream('GET', url) as r:
            r.raise_for_status()
            print(r.headers)

asyncio.run(main())

The error:

Traceback (most recent call last):
  File "./app/fetch.py", line 332, in update_loop
    if await check_needs_update():
  File "./app/fetch.py", line 238, in check_needs_update
    for headers in (await asyncio.gather(*awaitables)):
  File "./app/fetch.py", line 230, in get_headers
    return r.headers
  File "/app/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1846, in __aexit__
    await self.response.aclose()
  File "/app/.venv/lib/python3.8/site-packages/httpx/_models.py", line 1299, in aclose
    await self._on_close(self)
  File "/app/.venv/lib/python3.8/site-packages/httpx/_client.py", line 1513, in on_close
    await stream.aclose()
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 69, in aclose
    await self.stream.aclose()
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_bytestreams.py", line 78, in aclose
    await self._aclose_func()
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_async/http11.py", line 192, in _response_closed
    await self.aclose()
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_async/http11.py", line 202, in aclose
    await self.socket.aclose()
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_backends/asyncio.py", line 198, in aclose
    await self.stream_writer.wait_closed()  # type: ignore
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/app/.venv/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
    raise to_exc(exc) from None
httpcore.CloseError: [SSL: APPLICATION_DATA_AFTER_CLOSE_NOTIFY] application data after close notify (_ssl.c:2745)

@lazka

@lazka Hello,

I tried running this script a few times and wasn't able to reproduce, though AFAIU the script wasn't expected to reproduce in the first place. :-)

Can you confirm if…

This is still occurring now (not a transient / temporary issue)?
Does this only reproduce after upgrading to 0.12.3 (i.e. downgrading to 0.12.2 fixes it)?

FWIW, I see this is occurring on a change introduced by encode/httpcore#167. But the fact that we haven't seen other reports of this as a bug makes me wonder if this is occurring just for certain sites, certain circumstances, or if this was just a transient issue.

I tried running this script a few times and wasn't able to reproduce, though AFAIU the script wasn't expected to reproduce in the first place. :-)

yes, I can't reproduce with the script either, but it shows the httpx API/features I use for the code raising the error. Not sure if that helps..

This is still occurring now (not a transient / temporary issue)?

yes, same issue right now

Does this only reproduce after upgrading to 0.12.3 (i.e. downgrading to 0.12.2 fixes it)?

yes, exactly

FWIW, I see this is occurring on a change introduced by encode/httpcore#167. But the fact that we haven't seen other reports of this as a bug makes me wonder if this is occurring just for certain sites, certain circumstances, or if this was just a transient issue.

I'm wondering, isn't httpx leaking a httpcore exception a bug in itself? I'd assumed that it only throws its own exceptions.

ah, got a reproducer:

import asyncio
import uvicorn
import httpx

async def main():
    c = 0
    while 1:
        url = "https://github.com/msys2/MINGW-packages/releases/download/srcinfo-cache/srcinfo.json"
        async with httpx.AsyncClient() as client:
            # XXX: github doesn't support redirects with HEAD and returns bogus headers
            async with client.stream('GET', url) as r:
                r.raise_for_status()
                r.headers
                print(c)
                c += 1

async def app(scope, receive, send):
    asyncio.create_task(main())

uvicorn.run(app)

Update deps; pin httpcore

@lazka

@lazka Thanks for the repro example. I didn't realize right away but we also have #1463 which is essentially the same as this issue. We've done some progress there so I'll close this one off. Thanks for putting some time into this! :-)

I see, thanks. Note the I don't expect any error to be raised with my example above. Unless I'm missing something.

Ahh. Right, I got confused. Actually this issue displays #1463, but it is a different issue — one about "got an SSL close error but didn't expect one". I'll reopen. :-)

@lazka

@lazka I tried to run your repro script. I intermittently get either httpx.ConnectTimeout, either the CloseError with the SSL error message. It's very infrequent — perhaps 1 in 10 cases or less for both.

I noticed that the specific URL you're requesting returns a redirect response. I tried disabling redirects with stream(..., allow_redirects=False). It seemed to make the CloseError go away (?), and ConnectTimeout might still happen once in a while.

Does it reproduce against some other URLs, or is this GitHub release URL causing issues specifically?

Related: https://bugs.python.org/issue39951

Basically this might be just considered as a server-side bug from GitHub. We sent CLOSE_NOTIFY but their server keeps sending data which is a protocol violation. At least that's the case openssl guards against: https://github.com/openssl/openssl/blob/76624df15fef0725f28a8b9d0f31256946669b1a/ssl/record/rec_layer_s3.c#L1635-L1648

It's possible we might want to filter this out in httpcore, by wrapping our call to .wait_closed() and just giving up in case an SSLError exception occurs. (?)

@lazka

@lazka I tried to run your repro script. I intermittently get either httpx.ConnectTimeout, either the CloseError with the SSL error message. It's very infrequent — perhaps 1 in 10 cases or less for both.

I noticed that the specific URL you're requesting returns a redirect response. I tried disabling redirects with stream(..., allow_redirects=False). It seemed to make the CloseError go away (?), and ConnectTimeout might still happen once in a while.

yeah, I want to get the etag/mtime of a github asset without downloading it. When I wrote this github returned bogus header data for HEAD requests, so I went with GET + streaming. But I just tried now and it seems github has fixed this and now returns the correct headers for HEAD requests as well \o/

Does it reproduce against some other URLs, or is this GitHub release URL causing issues specifically?

out of 10 URLs only the github ones are affected.

But I just tried now and it seems github has fixed this and now returns the correct headers for HEAD requests as well \o/

Okay. I'm not able to reproduce this, but presuamably there are server implementations out there that'd still trigger this.

The first thing we ought to try to isolate when we run into gnarly low-level socket/SSL bugs like this is confirming which backends they apply to.

Eg. does it replicate on the regular sync client? Does it replicate with trio? curio?

As it is, I'd assume this is asyncio-specific, but inot sure how to confirm that ATM.

I have seen this error on v0.17.1 and am supplying a reproduction.

The problem: the server closes its half of the TCP connection even after indicating that it is keep-alive. On the client, closing raises an OSError. The error isn't being wrapped: it's an httpcore.CloseError instead of an httpx.CloseError.

This is similar to #1463. #1463 focused on an exception during streaming; this one is about an exception after streaming.

To reproduce: add tests/client/test_newbug.py to a fresh 0.17.1 checkout:

import _socket
import asyncio
import io
import struct

import httpx
import pytest


@pytest.mark.asyncio
async def test_download_stream_connection_reset():
    async def respond_then_die(reader, writer):
        # Read request -- as we should
        await reader.readuntil(b"\r\n\r\n")
        # Send HTTP headers and data, indicating "keep-alive"
        writer.write(
            b"HTTP/1.1 200 OK\r\nContent-Length:3\r\nContent-Type:text/plain\r\n\r\nend"
        )
        await writer.drain()
        # Force a TCP RST packet
        # http://deepix.github.io/2016/10/21/tcprst.html
        fd = writer._transport._sock.fileno()
        linger = struct.pack("=II", 1, 0)
        _socket.socket(fileno=fd).setsockopt(
            _socket.SOL_SOCKET, _socket.SO_LINGER, bytearray(linger)
        )
        writer.close()

    server = await asyncio.start_server(respond_then_die, "127.0.0.1")
    async with server:
        host, port = server.sockets[0].getsockname()
        async with httpx.AsyncClient() as client:
            await client.get(f"http://{host}:{port}")
            # Now, disconnecting will close() ... but the server side is already closed!

The output of pytest tests/client/test_newbug.py:

httpcore.CloseError: [Errno 104] Connection reset by peer

This should either raise an httpx.CloseError (because the client-close op failed) or nothing (because "Connection reset by peer" suggests that both ends have successfully closed their TCP connections).

I suggest a new title for this issue: "httpx.AsyncClient.__aexit__() raises httpcore.CloseError instead of httpx.CloseError"

Should now be resolved by encode/httpcore#310

Navigation Menu

Repository navigation

httpcore update from 0.12.2 to 0.12.3 results in httpcore.CloseError #1459

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

httpcore update from 0.12.2 to 0.12.3 results in httpcore.CloseError #1459

Description

Activity

florimondmanca commented on Feb 4, 2021

lazka commented on Feb 4, 2021

lazka commented on Feb 4, 2021

florimondmanca commented on Feb 12, 2021

lazka commented on Feb 12, 2021

florimondmanca commented on Feb 12, 2021

florimondmanca commented on Feb 12, 2021

florimondmanca commented on Feb 12, 2021

lazka commented on Feb 15, 2021

lovelydinosaur commented on Mar 23, 2021

adamhooper commented on Apr 26, 2021

adamhooper commented on Apr 26, 2021

lovelydinosaur commented on Apr 29, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions