Allow the specifying of which system IP address to use for client connections.

Allow the specifying of which system IP address to use for client connections. If you look at http.client.HTTPConnection, it allows you pass a source address:

class http.client.HTTPConnection(host, port=None, [timeout, ]source_address=None, blocksize=8192)

https://docs.python.org/3/library/http.client.html

Thanks, that's a really interesting request.

I think we're likely at a stage in the project where we'd probably push back on new API unless we consider it absolutely clear-cut must have functionality, and that some things that don't fall into that category could go into a maybe later, but let's get more real-world feedback first, so my initial reaction would probably be to put this into the "interesting, maybe later" pile...

For a bit of context, what's the motivation / use-case here?
Are there other high-level Python HTTP clients that make this available?

Are there other high-level Python HTTP clients that make this available?

urllib3
requests
asks
tornado.httpclient
aiohttp
treq through supplying a Twisted Reactor

I can find it in urllib3, https://urllib3.readthedocs.io/en/latest/reference/#urllib3.connection.HTTPConnection but I can't find it in requests?

Closed as "out of scope" for requests, psf/requests#394 but available in the toolbelt... https://toolbelt.readthedocs.io/en/latest/adapters.html#sourceaddressadapter

I assumed for requests itself. But, yeah. Still available in semi-official capacity.

Edit: And the rest were from code searches, though they're not all spelled the same way

Thanks, that's a really interesting request.

I think we're likely at a stage in the project where we'd probably push back on new API unless we consider it absolutely clear-cut must have functionality, and that some things that don't fall into that category could go into a maybe later, but let's get more real-world feedback first, so my initial reaction would probably be to put this into the "interesting, maybe later" pile...

For a bit of context, what's the motivation / use-case here?
Are there other high-level Python HTTP clients that make this available?

The use case I have run into many times now is the need to simulate a pool of users, sourcing from unique IP addresses, for simulating users. This is not possible with requests and has been requested as an enhancement many times. Today I find users spawning off multiple containers to accomplish this vs. using a single machine, with multiple secondary IP addresses bound to the NIC, to accomplish this. To add more color to the use case: testing a web proxy, FW, wan optimizer, and network traffic probes, ...etc, whereby a single IP requesting many webpages.

I was just looking for this functionality in httpx, for a use case of sending DoH (DNS over HTTPS) requests, from dnspython. Currently, dnspython uses requests to send queries, and requests_toolbelt.adapters.source.SourceAddressAdapter (when necessary) to set the source address, but since DoH is supposed to use HTTP/2, it would be nice to use httpx instead.

I'm going to currently close this off as out of scope.

We might be able to consider a nicely done PR to httpcore that implements this, but I don't want to consider it a priority unless someone's willing to step up and take it on.

Since there is no proper documentation for this, here is how to use local address:

>>> import httpx
>>> from httpcore import SyncConnectionPool
>>> transport = SyncConnectionPool(local_address="xxxx:xxxx:xxxx:xxxx:f3d2:3ef7:28cd:7b60", http2=True)
>>> client = httpx.Client(transport=transport)
>>> r = client.get("https://api6.ipify.org")
>>> r.text
'xxxx:xxxx:xxxx:xxxx:f3d2:3ef7:28cd:7b60'

To answer earlier question: how to do it with requests: https://stackoverflow.com/questions/48776564/python-and-c-sharing-the-same-memory-resources

I'm trying to get an async version working:

import trio
import httpx

from httpcore import AsyncConnectionPool

URL = 'https://api4.ipify.org/'
IP = '...'  # my public IP4 address

async def run_client(ip):
    # works
    async with httpx.AsyncClient() as client:
        response = await client.get(URL)
        print(response.status_code, len(response.content))

    # fails
    async with AsyncConnectionPool(local_address=ip, http2=True) as transport:
        async with httpx.AsyncClient(transport=transport) as client:
            response = await client.get(URL)
            print(ip, response.status_code, len(response.content))

async def main():
    async with trio.open_nursery() as n:
        n.start_soon(run_client, IP)

trio.run(main)

... but getting an error:

root@test-host-1:~# python3 trio_scraper.py 
200 14
Traceback (most recent call last):
  File "trio_scraper.py", line 22, in <module>
    trio.run(main)
  File "/usr/local/lib/python3.8/dist-packages/trio/_core/_run.py", line 1932, in run
    raise runner.main_task_outcome.error
  File "trio_scraper.py", line 20, in main
    n.start_soon(run_client, '209.250.240.59')
  File "/usr/local/lib/python3.8/dist-packages/trio/_core/_run.py", line 815, in __aexit__
    raise combined_error_from_nursery
  File "trio_scraper.py", line 15, in run_client
    response = await client.get(URL)
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1736, in get
    return await self.request(
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1513, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1600, in send
    response = await self._send_handling_auth(
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1628, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1665, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1702, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.8/dist-packages/httpcore/_async/connection_pool.py", line 206, in handle_async_request
    scheme = request.url.scheme.decode()
AttributeError: 'str' object has no attribute 'decode'

You need to be using async with httpx.HTTPTransport(local_address=ip, http2=True) as transport:

Thanks!
However if I switch that out, I now get:

  File "trio_scraper.py", line 20, in run_client
    async with httpx.HTTPTransport(local_address=ip, http2=True) as transport:
AttributeError: __aexit__

@tomchristie

ok gottit!
Here's how to do async:

import trio
import httpx

URL = 'https://api4.ipify.org/'
IP = '...'  # use your box's public IPv4

async def run_client(ip):
    async with httpx.AsyncHTTPTransport(local_address=ip, http2=True) as transport:
        async with httpx.AsyncClient(transport=transport) as client:
            response = await client.get(URL)
            print(ip, response.status_code, len(response.content))

async def main():
    async with trio.open_nursery() as n:
        n.start_soon(run_client, IP)

trio.run(main)

Thanks @tomchristie for shaking it loose

Just a note that trying a different target URL I ran into ModuleNotFoundError: No module named 'h2' and needed pip install h2.
As per guidelines I floated it on https://gitter.im/encode/community rather than file an issue.

Since there is no proper documentation for this, here is how to use local address:

Thanks for that example. I ended on this issue after a quick search and I had to make some adjustments to that example for it to work:

import httpx

transport = httpx.HTTPTransport(local_address="MY_IP")

client = httpx.Client(transport=transport)

response = client.get("https://api6.ipify.org")

print(response.text)

This use case is also documented in https://www.python-httpx.org/advanced/transports/, but for some reason I found this issue first.

Navigation Menu

Repository navigation

Allow the specifying of which system IP address to use for client connections. #755

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow the specifying of which system IP address to use for client connections. #755

Description

Activity

lovelydinosaur commented on Jan 13, 2020

thebigmunch commented on Jan 13, 2020

lovelydinosaur commented on Jan 13, 2020

lovelydinosaur commented on Jan 13, 2020

thebigmunch commented on Jan 13, 2020

eparra commented on Jan 13, 2020

bwelling commented on May 7, 2020

lovelydinosaur commented on May 11, 2020

kotori2 commented on Feb 17, 2021

p-i- commented on Jan 17, 2022

p-i- commented on Jan 17, 2022

lovelydinosaur commented on Jan 17, 2022

p-i- commented on Jan 17, 2022

p-i- commented on Jan 17, 2022

p-i- commented on Jan 17, 2022

mimi89999 commented on Mar 23, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions