access https proxy CONNECT response headers

httpx 0.7.6

I want to access the headers returned by the CONNECT HTTP call.

I'm achieving this now by the following grotesque code. I wish there were a supported way to do this, but unfortunately the current implementation throws these headers away.

class WrapperHTTPProxy(wrapt.ObjectProxy):

    async def send(
        self,
        request: AsyncRequest,
        verify: VerifyTypes = None,
        cert: CertTypes = None,
        timeout: TimeoutTypes = None,
    ) -> AsyncResponse:
        # copy of send from httpx.dispatch.HTTPConnection

    async def acquire_connection(self, origin: Origin) -> HTTPConnection:
        # copy of acquire_connection from httpx.dispatch.HTTPProxy

    async def tunnel_connection(self, origin: Origin) -> HTTPConnection:
        # copy of tunnel_connection from httpx.dispatch.HTTPProxy

    async def request_tunnel_proxy_connection(self, origin: Origin) -> HTTPConnection:
        # copy of request_tunnel_proxy_connection from httpx.dispatch.HTTPProxy
        # with new code at the end
        if not (200 <= proxy_response.status_code <= 299):
            await proxy_response.read()
            raise httpx.ProxyError(
                f"Non-2XX response received from HTTP proxy "
                f"({proxy_response.status_code})",
                request=proxy_request,
                response=proxy_response,
            )
        else:
            proxy_response.on_close = None
            await proxy_response.read()
            # NEW CODE: send the proxy response headers to a callback
            if wrapper._self_headers_callback:
                wrapper._self_headers_callback(proxy_response.headers)
        return connection

class MyClassThatDoesStuff:

    latest_headers = None

    def after_session_is_set_up(self):
        https_proxy = self.httpx_session.proxies['https']
        self.proxy_wrapper = WrapperHTTPProxy(https_proxy)
        self.httpx_session.proxies['https'] = self.proxy_wrapper
        def cb(headers):
            self.latest_headers = headers
        self.proxy_wrapper._self_headers_callback = cb

On the face of it this doesn't immediately look to me like something that we ought to be supporting via our public API.

It's possible that I've made an incorrect assessment here. There's a couple of things that could change my mind on this...

Is this something that you'd be able to do more easily in requests/urllib3?
Do you have a concrete proposal for what the public API for this ought to look like.

In the meantime I'd suggest you support this by subclassing HTTPProxy, adding your custom behavior, and using your custom proxy with the client...

proxy = MyCustomProxy(proxy_url)
client = Client(proxies=proxy)

Thanks for considering this, Tom. I'm not familiar enough with the httpx API and the design principles behind it to offer an API suggestion at this point. As I use httpx more, I'll give this some thought.

I think reading the CONNECT response headers is also quite difficult in requests / urllib3. For example, there's this open issue on httplib:

https://bugs.python.org/issue24964

And, related to this, a stackoverflow question, with some background about why this might be a useful feature in some circumstances.

https://stackoverflow.com/questions/39068998/reading-connect-headers

Finally, an implementation in urllib3 showing that, indeed, it's not straightforward:

https://gist.github.com/bpartridge/9c758c5e70222bac6ce6e1db7bb4d8ea

This establishes that, while this is a niche need, there's at least one other person who wanted to access these headers... and that it's likely harder to do in urllib3 and requests than in httpx.

Thanks for the suggestion to subclass HTTPProxy. I'll give that a go.

Thanks for the suggestion to subclass HTTPProxy. I'll give that a go.

Great - probably worth following up here with how you get on there.

Perhaps there's something here that we could come back to, but at the moment it looks more likely that we should instead make sure that it's not too painful to do this with a custom proxy class.

Okay — subclassing HTTPProxy is significantly better. Rather than 90 lines of code copied, and using Wrapt, it's now more standard Python and 30 lines of code copied.

class ResponseHeaderCapturingProxy(httpx.HTTPProxy):
    def __init__(self, proxy_url, backend=None):
      super().__init__(proxy_url, backend=backend)
      self.response_headers = None


    def proxy_response_hook(self, response):
      self.response_headers = response.headers

    async def request_tunnel_proxy_connection(self, origin: httpx.Origin) -> httpx.HTTPConnection:
        """Creates an HTTPConnection by setting up a TCP tunnel"""
        proxy_headers = self.proxy_headers.copy()
        proxy_headers.setdefault("Accept", "*/*")
        proxy_request = httpx.AsyncRequest(
            method="CONNECT", url=self.proxy_url.copy_with(), headers=proxy_headers
        )
        proxy_request.url.full_path = f"{origin.host}:{origin.port}"

        await self.max_connections.acquire()

        connection = httpx.HTTPConnection(
            self.proxy_url.origin,
            verify=self.verify,
            cert=self.cert,
            timeout=self.timeout,
            backend=self.backend,
            http_versions=["HTTP/1.1"],  # Short-lived 'connection'
            trust_env=self.trust_env,
            release_func=self.release_connection,
        )
        self.active_connections.add(connection)

        # See if our tunnel has been opened successfully
        proxy_response = await connection.send(proxy_request)

        self.proxy_response_hook(proxy_response) # Hook inserted into my subclass

        if not (200 <= proxy_response.status_code <= 299):
            await proxy_response.read()
            raise httpx.ProxyError(
                f"Non-2XX response received from HTTP proxy "
                f"({proxy_response.status_code})",
                request=proxy_request,
                response=proxy_response,
            )
        else:
            proxy_response.on_close = None
            await proxy_response.read()

        return connection

It would be nice to be able to copy less code. Perhaps the hook I inserted into my subclass could be added to HTTPProxy along with a no-op implementation of proxy_response_hook(response)?

For my purposes, a synchronous method is fine, but for flexibility I guess an async hook method gives more choice.

One other thing I'd like to mention: while I was looking for others wanting to use this kind of feature in Python http libraries, I came across the idea of having an option on an HTTP Proxy to either

ignore headers from the proxy response
combine headers from the proxy response into the regular http response

That would have served my need in this case too.

Gotcha. I don't have much of a gauge on how useful this would likely be to other users or not. Is that a typical thing to do? I guess we might want to wait and see if a similar issue comes up again, and if so figure out if there's much commonality?

It's kind of niche, I guess.

It is needed by anyone using certain HTTP proxy services that do specific things like rotate IP addresses, where you want to give feedback about the proxying independently of the requesting. For example, https://luminati.io/ or https://www.proxymesh.com/

Navigation Menu

Repository navigation

access https proxy CONNECT response headers #523

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

access https proxy CONNECT response headers #523

Description

Activity

lovelydinosaur commented on Nov 30, 2019

SteveAlexander commented on Nov 30, 2019

lovelydinosaur commented on Nov 30, 2019

SteveAlexander commented on Nov 30, 2019

SteveAlexander commented on Nov 30, 2019

lovelydinosaur commented on Dec 2, 2019

SteveAlexander commented on Dec 2, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions