Repository navigation
Use percent-encoding for / in query params, for better robustness. #2721
Description
Discussed in #2661
Originally posted by marban April 15, 2023
Haven't gotten into the details but by upgrading to 0.24 and using a regular dict as GET params that contains e.g. an url like {"q": "http://example.com"} will somehow break the request to, in this case, Twitter's API. If if urllib.parse.quote_plus the "http://example.com", everything works fine (The slash is the culprit).
If I urlencode the whole dict manually before passing it to httpx and call the r.url after the request the / is again not encoded, although it previously was.
https://api.twitter.com/1.1/search/tweets.json?q=url%3Ahttp%3A//google.com
I assume it has to do with
- Query parameter encoding switches from using + for spaces and %2F for forward slash, to instead using %20 for spaces and treating forward slash as a safe, unescaped character. This differs from requests, but is in line with browser behavior in Chrome, Safari, and Firefox. Both options are RFC valid. (Use '%20' for encoding spaces in query parameters. #2543)
Activity
lovelydinosaur commentedon May 26, 2023
Let me nudge you in the right direction here...
httpx/httpx/_urlparse.py
Line 264 in 7335950
Also we don't want to make it optional, we just want to change the behaviour.
Evidently percent-encoding
/in query params might not be spec-neccessary, but it is more robust behaviour.[-]Make forward slash encoding optional (to make it compatible with Twitter's API)[/-][+]Use percent-encoding for `/` in query params, for better robustness.[/+]BjoernPetersen commentedon Sep 23, 2023
@tomchristie just so you know: this broke more than the Twitter API. I have a project in which I want to request a signed URL to an image (from the OpenAI API) using httpx. Some signature query params contain forward slashes, and the server evidently doesn't accept it if those are encoded as
%2F.I want to note that the URL works in Firefox, Chrome, and using
requests.get(image_url), but httpx's encoding breaks it.It might be true that encoding the slashes is technically "more correct", but that doesn't really help if the server just won't accept it.
ttys0dev commentedon Oct 7, 2023
Yeah, hit this as well in #2883 when migrating from requests to httpx, unconditionally precent-encoding
/'s is clearly not the correct behavior IMO if servers treat it differently.alexmaurizio commentedon Nov 23, 2023
Jumping in to signal we also had to revert httpx from 0.25.0 to 0.24.1 due to this PR.
This change is causing issues with double encoding and a whole lot of services.