Repository navigation
Too many open files if keep changing proxy #824
Description
Here is the codes:
import os,httpx,asyncio
async def main():
proxies_list=[
{'all':'http://4.3.2.1:8080'},
{'all':'http://1.2.3.4:8080'},
]
for i in range(10):
proxies=proxies_list[i%2]
async with httpx.AsyncClient(proxies=proxies) as client:
r=await client.get('http://my-url.com/')
print('lsof -np',os.getpid())
for i in range(60):
print(i)
await asyncio.sleep(3)
asyncio.run(main())After sleep 5 seconds as defined in httpx.dispatch.connection_pool.ConnectionPool.KEEP_ALIVE_EXPIRY, I run this commend line:
lsof -np 18532 | grep 8080
Then I get this:
python 19034 root 6u IPv4 6658191 0t0 TCP 192.168.1.1:41922->4.3.2.1:8080(ESTABLISHED)
python 19034 root 7u IPv4 6658192 0t0 TCP 192.168.1.1:63358->1.2.3.4:8080(ESTABLISHED)
python 19034 root 8u IPv4 6658193 0t0 TCP 192.168.1.1:41926->4.3.2.1:8080(ESTABLISHED)
python 19034 root 9u IPv4 6658194 0t0 TCP 192.168.1.1:63362->1.2.3.4:8080(ESTABLISHED)
python 19034 root 8u IPv4 6658195 0t0 TCP 192.168.1.1:41926->4.3.2.1:8080(ESTABLISHED)
python 19034 root 9u IPv4 6658196 0t0 TCP 192.168.1.1:63362->1.2.3.4:8080(ESTABLISHED)
python 19034 root 10u IPv4 6658197 0t0 TCP 192.168.1.1:41926->4.3.2.1:8080(ESTABLISHED)
python 19034 root 11u IPv4 6658198 0t0 TCP 192.168.1.1:63362->1.2.3.4:8080(ESTABLISHED)
python 19034 root 12u IPv4 6658198 0t0 TCP 192.168.1.1:41926->4.3.2.1:8080(ESTABLISHED)
python 19034 root 13u IPv4 6658200 0t0 TCP 192.168.1.1:63362->1.2.3.4:8080(ESTABLISHED)
It means httpx dosn't close the connection correctly. If I keep changing proxy, I will get "too many open files" error at the end.
Activity
lovelydinosaur commentedon Feb 25, 2020
Good observation... We're closing the main dispatch class, but we're not also making sure to close the proxy dispatch classes...
httpx/httpx/_client.py
Line 864 in 50d337e
A mitigation would be to just use a single client wherever possible, or a single client per required configuration. If you do that then you're getting the benefits of the connection pooling that it provides.
However there is a valid issue here, related to how we deal with closing multiple mounted dispatch classes, rather than just the main dispatch class.
Jonney commentedon Feb 25, 2020
Actually, I am writing a web crawler, after runing 1-2 days it crashes because of "too many open files", this is how I get it.
I have no time to wait httpx to release a new version, sorry, I've got to use aiohttp to avoid this problem.