Regression with 0.13.dev0: Missing mandatory Host: header

Checklist

The bug is reproducible against the latest release and/or master.
There are no similar issues or pull requests to fix it yet.

Describe the bug

With an HTTP proxy enabled, the HTTP/1.1 implementation raises a LocalProtocolError exception:

  File "/usr/local/lib/python3.8/dist-packages/obs_proxy-0-py3.8.egg/obs_proxy/repserver.py", line 100, in proxy_request
    resp = await client.send(req, stream=True, auth=build_auth(), timeout=HTTP_TIMEOUT)
  File "/usr/local/lib/python3.8/dist-packages/httpx-0.13.dev0-py3.8.egg/httpx/_client.py", line 1137, in send
    response = await self.send_handling_redirects(
  File "/usr/local/lib/python3.8/dist-packages/httpx-0.13.dev0-py3.8.egg/httpx/_client.py", line 1164, in send_handling_redirects
    response = await self.send_handling_auth(
  File "/usr/local/lib/python3.8/dist-packages/httpx-0.13.dev0-py3.8.egg/httpx/_client.py", line 1201, in send_handling_auth
    response = await self.send_single_request(request, timeout)
  File "/usr/local/lib/python3.8/dist-packages/httpx-0.13.dev0-py3.8.egg/httpx/_client.py", line 1233, in send_single_request
    ) = await dispatcher.request(
  File "/usr/local/lib/python3.8/dist-packages/httpcore-0.8.2-py3.8.egg/httpcore/_async/http_proxy.py", line 76, in request
    return await self._tunnel_request(
  File "/usr/local/lib/python3.8/dist-packages/httpcore-0.8.2-py3.8.egg/httpcore/_async/http_proxy.py", line 145, in _tunnel_request
    proxy_response = await proxy_connection.request(
  File "/usr/local/lib/python3.8/dist-packages/httpcore-0.8.2-py3.8.egg/httpcore/_async/connection.py", line 69, in request
    return await self.connection.request(method, url, headers, stream, timeout)
  File "/usr/local/lib/python3.8/dist-packages/httpcore-0.8.2-py3.8.egg/httpcore/_async/http11.py", line 52, in request
    await self._send_request(method, url, headers, timeout)
  File "/usr/local/lib/python3.8/dist-packages/httpcore-0.8.2-py3.8.egg/httpcore/_async/http11.py", line 77, in _send_request
    event = h11.Request(method=method, target=target, headers=headers)
  File "/usr/local/lib/python3.8/dist-packages/h11-0.9.0-py3.8.egg/h11/_events.py", line 71, in __init__
    self._validate()
  File "/usr/local/lib/python3.8/dist-packages/h11-0.9.0-py3.8.egg/h11/_events.py", line 143, in _validate
    raise LocalProtocolError("Missing mandatory Host: header")
h11._util.LocalProtocolError: Missing mandatory Host: header

The relevant part of the app in question, adjusted to the config used:

client = httpx.AsyncClient(
    http2=False,
    verify=False,
    proxies={
        'all': 'http://...'
    }
)

# <...>

resp = await client.send(req, stream=True, auth=build_auth(), timeout=HTTP_TIMEOUT)

Environment

OS: Debian testing.
Python version: 3.8.3~rc1 as shipped by Debian
HTTPX version: 0.13.dev0
Async environment: asyncio
HTTP proxy: yes
Custom certificates: yes

Thanks for the report, this and #921 are caused by encode/httpcore#75 and should be solved in 0.13.

We could defo consider a 0.13.dev1

@yeraydiazdiaz

@yeraydiazdiaz Forgetting about #927 for a moment... Can we try to isolate this down a bit more, since it's not clear to me why we've got a behaviour regression here.

Is this an issue with FORWARD or TUNNEL requests, and can we be specific about how we had the correct headers set in 0.12 (with a single proxy instance in default mode), but not in 0.13 (with a single proxy instance in default mode)?

@andrewshadura Could you give me a bit more info on how to replicate this, in particular what are you using for the proxy? Are you able to confirm that the behaviour was correct on 0.12.x? Does the behaviour replicate against both http and https URLs?

I'd dig having a pure python setup we can use to replicate any issues with proxying. I took a first look at using the pproxy package to do this, but only got 400 Bad Response when making requests via the proxy, and didn't have a chance to dig into at what point that was being triggered. (Eg. was that a response from the proxy, or a response from the end server?)

The proxy I’m using is mitmproxy, the URLs are https only with snakeoil certificates.
The behaviour was correct on 0.12, in fact when I pinned httpx to 0.12 the code started working with no other changes.

I haven’t yet had time to produce a minimal setup to replicate it, but if you want to play with it in our containers, the specific Docker setup can be found here: https://gitlab.collabora.com/obs/docker-obs/-/tree/collabora-obs-proxy (you need to drop the pip install httpx==0.12 from the Dockerfile, of course). The code in question runs in the proxy-client container, to trigger in curl -v http://proxy-client:5000/getworkercode needs to be executer in the worker.
The code itself is up here: https://gitlab.apertis.org/infrastructure/obs-proxy/

Is this an issue with FORWARD or TUNNEL requests, and can we be specific about how we had the correct headers set in 0.12 (with a single proxy instance in default mode), but not in 0.13 (with a single proxy instance in default mode)?

This is an issue on TUNNEL requests, the difference between 0.12 and 0.13 is the usage of httpcore.

In httpx 0.12 we had a proxy_http dispatch which was moved to httpcore with some changes. The one affecting this particular issue is that the dispatch used httpx.Request models to perform the CONNECT request and Request.prepare sets a series of request headers that include host. This was lost in the move to httpcore in favor of using raw data structures.

This does not affect FORWARD requests because we only perform the one forward request so we merge the proxy headers and the request headers (which may cause errors due to header duplication but that's a different issue).

Gotcha, yup, that makes sense.

One option here might be to start by making sure that httpcore is more closely replicating the proxy behaviour that we had in 0.12, so that we can drop out any bugs and issue a 0.13 release.

So:

Add Host and Accept: */* headers by default on the tunnel CONNECT requests, if they're not already set.
De-duplicate proxy headers vs request headers on forwarded requests, rather than naively concatenating them.

We can then follow up with #927 in a later release, alongside tightening up httpcore so that it's doing less smarts about eg. adding default headers on proxy requests, and tightening up httpx so that it's auto-adding them into the proxy configuration instead.

I'd dig having a pure python setup we can use to replicate any issues with proxying.

I agree, my two cents - Might make sense to enable the Wiki here and perhaps writing up a "Setting a local proxy for debugging" guide (probably with mitmproxy since it's Python-based?)? I got mitmproxy to work while investigating an HTTPCore issue a while ago, but I keep forgetting how to use it - and the online docs somehow aren't super easy to grasp.

So #927 is orthogonal to this issue as it's about client configuration whereas this is a bug in the request preparing and sending.

The key IMO here is that proxying is done at the dispatch level which lives in httpcore.

In order for httpx to simply call dispatch.send and behave properly httpcore needs to implement those two bullet points above. Otherwise we start leaking knowledge of proxying mechanics into httpx, i.e. it'd need to check the proxy mode, figure out it needs to do a CONNECT request, set it up and finally use httpcore. Personally I'd rather keep that knowledge in httpcore, even if it means it has "smarts". 🙂

I got mitmproxy to work while investigating an HTTPCore issue
a while ago, but I keep forgetting how to use
it - and the online docs somehow aren't super easy to grasp.

I guess the docs are so because it’s super easy to use once you know how to, which is why they were not so keen on documenting it 😃

With web GUI, you just run mitmweb and it opens a browser (preferring Chromium) where you can browse things. With command-line interface (not TUI), you run mitmdump and it just dumps things.

You can also use a TUI if you run mitmproxy but I haven’t used it 🙂

Update: apparently it only opens Chromium/Chrome by default, it’s hardcoded.

Update 2: I was wrong, it’s just the matter of hard-coded priorities:

    browsers = (
        "windows-default", "macosx",
        "google-chrome", "chrome", "chromium", "chromium-browser",
        "firefox", "opera", "safari",
    )

Upgrading httpcore should now resolve this. pip install httpcore -U.

Resolved in 0.13.0.dev1

Navigation Menu

Repository navigation

Regression with 0.13.dev0: Missing mandatory Host: header #929

Checklist

Describe the bug

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Regression with 0.13.dev0: Missing mandatory Host: header #929

Description

Checklist

Describe the bug

Environment

Activity

yeraydiazdiaz commented on May 4, 2020

lovelydinosaur commented on May 4, 2020

lovelydinosaur commented on May 5, 2020

lovelydinosaur commented on May 5, 2020

andrewshadura commented on May 5, 2020

yeraydiazdiaz commented on May 5, 2020

lovelydinosaur commented on May 5, 2020

florimondmanca commented on May 5, 2020

yeraydiazdiaz commented on May 5, 2020

andrewshadura commented on May 5, 2020

lovelydinosaur commented on May 6, 2020

lovelydinosaur commented on May 7, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions