Repository navigation
safe_requests feature request #13
Description
The current implementation of safe_requests parses the hostname from the request URL and checks it against a fairly limited set of IP addresses (namely 198.168.1.1 and the AWS instance metadata URL). This should be fairly easy to bypass by registering a domain name (or finding one that already exists) either with an A record that resolves to one of these restricted addresses, or by hosting a web server that returns a redirect to the restricted address.
I propose the following procedure for sandboxing requests.
- Resolve the host to an IP address, ex.
socket.gethostbyname() - Check that the IP address falls within a set of public IP address ranges
- Swap the hostname of the URL with the resolved IP address
- Hand the request off to
requests
There are many edge cases to handle here, ex. tls hostname verification, proxies, redirects, and it may ultimately be easiest to add functionality to requests and/or urllib to support IP address allow/block lists.
Activity
tjs-intel commentedon Feb 22, 2024
@drdavella
drdavella commentedon Feb 22, 2024
Hi @tjs-intel thanks so much for opening this issue! I agree that ultimately upstreaming this to
requestsitself might be a good idea. However one of the main motivations for thesecurityproject is the fact that many commonly used libraries don't currently enable that kind of hardening.You've identified an interesting gap here and I think it's worth working through a potential solution. I'm going to ask @matt- to chime in since I think he has a few ideas about how we could accomplish what you're suggesting. One such option might be to hook into a lower-level layer of the
requestslibrary and perform validation there.tjs-intel commentedon Feb 22, 2024
I tried a quick POC of this concept yesterday by adding an IP address check here in
http.client:https://github.com/python/cpython/blob/b254de71d0d1eb0232ff4abd403a4977ca3f2fe7/Lib/http/client.py#L1010
matt- commentedon Feb 22, 2024
If we just swap out the host name with the resolved IP I think we may run into some issues with servers that check hostname header, or parse raw URL for some reason.
However I think we could hook into this by making the safe methods use a custom "Transport Adapter"
I will have to dig more, but it looks like we could grab access to the "connection" before the request is made which would give us the resolved IP
tjs-intel commentedon Feb 26, 2024
A relevant post on SO, showing how to resolve a hostname before making the connection: https://stackoverflow.com/a/22614367