Repository navigation
urlparse goes wrong with IP:port without scheme #38644
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/orsenthil'
closed_at = <Date 2010-08-04.04:56:30.468>
created_at = <Date 2003-06-13.15:15:34.000>
labels = ['type-bug', 'library']
title = 'urlparse goes wrong with IP:port without scheme'
updated_at = <Date 2019-10-18.15:23:21.979>
user = 'https://bugs.python.org/kruegi'bugs.python.org fields:
activity = <Date 2019-10-18.15:23:21.979>
actor = 'orsenthil'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2010-08-04.04:56:30.468>
closer = 'orsenthil'
components = ['Library (Lib)']
creation = <Date 2003-06-13.15:15:34.000>
creator = 'kruegi'
dependencies = []
files = ['10752', '18362']
hgrepos = []
issue_num = 754016
keywords = ['patch']
message_count = 19.0
messages = ['16377', '16378', '16379', '16380', '16381', '16382', '67890', '68526', '68533', '68535', '68841', '69095', '85877', '110334', '112725', '112758', '354892', '354897', '354906']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'georg.brandl', 'facundobatista', 'jjlee', 'sjones', 'kruegi', 'orsenthil', 'dstanek', 'elachuni', 'miss-islington']
pr_nums = ['661', '16837', '16839']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue754016'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']
Activity
kruegi commentedon Jun 13, 2003
urlparse doesnt work if IP and port are given without
scheme:
should be:
sjones commentedon Jun 14, 2003
Logged In: YES
user_id=589306
urlparse.urlparse takes a url of the format:
<scheme>://<netloc>/<path>;<params>?<query>#<fragment>
And returns a 6-tuple of the format:
(scheme, netloc, path, params, query, fragment).
An example from the library refrence takes:
sjones commentedon Jun 14, 2003
Logged In: YES
user_id=589306
Sorry, previous comment got cut off...
urlparse.urlparse takes a url of the format:
<scheme>://<netloc>/<path>;<params>?<query>#<fragment>
And returns a 6-tuple of the format:
(scheme, netloc, path, params, query, fragment).
An example from the library refrence takes:
urlparse('http://www.cwi.nl:80/%7Eguido/Python.html')
And produces:
('http', 'www.cwi.nl:80', '/%7Eguido/Python.html', '',
'', '')
--------------------------------
Note that there isn't a field for the port number in the
6-tuple. Instead, it is included in the netloc. Urlparse
should handle your example as:
Instead, it gives the incorrect output as you indicated.
sjones commentedon Jun 14, 2003
Logged In: YES
user_id=589306
Ok, I researched this a bit, and the situation isn't as
simple as it first appears. The RFC that urlparse tries to
follow is at http://www.faqs.org/rfcs/rfc1808.html and
notice specifically sections 2.1 and 2.2.
It seems to me that the source code follows rfc1808
religiously, and in that sense it does the correct thing.
According to the RFC, the netloc should begin with a '//',
and since your example didn't include one then it technical
was an invalid URL. Here is another example where it seems
Python fails to do the right thing:
Note that it is putting 'python.org' as the path and not the
netloc. So the problem isn't limited to just when you use a
scheme parameter and/or a port number. Now if we put '//' at
the beginning, we get:
So here it does the correct thing.
There are two problems though. First, it is common for
browsers and other software to just take a URL without a
scheme and '://' and assume it is http for the user. While
the URL is technically not correct, it is still common
usage. Also, urlparse does take a scheme parameter. It seems
as though this parameter should be used with a scheme-less
URL to give it a default one like web browsers do.
So somebody needs to make a decision. Should urlparse follow
the RFC religiously and require '//' in front of netlocs? If
so, I think the documentation should give an example showing
this and showing how to use the 'scheme' parameter. Or
should urlparse use the more commonly used form of a URL
where '//' is omitted when the scheme is omitted? If so,
urlparse.py will need to be changed. Or maybe another
fuction should be added to cover whichever behaviour
urlparse doesn't cover.
In any case, you can temporarily solve your problem by
making sure that URL's without a scheme have '//' at the
front. So your example becomes:
facundobatista commentedon Dec 26, 2004
Logged In: YES
user_id=752496
The problem is still present in Py2.3.4.
IMO, it should support dirs without the "http://" or raise
an error if it's not present (never fail silently!).
birkenfeld commentedon Aug 25, 2005
Logged In: YES
user_id=1188172
Will look into it. Should be easy to fix.
orsenthil commentedon Jun 10, 2008
Attaching the patch to fix this issue. I deliberated upon this for a
while and came up with the approach to:
fix the port issue, wherein urlparse should technically recognize the
':' separator for port from ':' after scheme.
And Doc fix wherein, it is advised that in the absence of a scheme,
use the net_loc as //net_loc (following RCF 1808).
If we go for any other fix, like internally pre-pending // when user has
not specified the scheme (like in many pratical purpose), then we stand
at chance of breaking a number of tests ( cases where url is 'g'(path
only),';x' (path with params) and cases where relative url is g:h)
Let me know your thoughts on this.
6 remaining items
orsenthil commentedon Apr 11, 2009
Facundo, I re-looked at this issue (after a long time; sorry for that)
and I think that the patch is good to be applied as it is for this issue.
The Web-SIG discussion, which you pointed to in the comment
(http://mail.python.org/pipermail/web-sig/2008-June/003458.html)
suggests a more "general case" of thinking and parsing of the urls.
The suggestion is specifically for "//path" kind of urls, which is will
be parsed into authority (as per RFC2396) or into netloc as in the patch.
My suggestion is to fix this issue with patch and if the corner case
like "//path" comes up, then I shall look the modifications required as
there is No RFC Specifications and Tests defining those cases.
Your comments?
BreamoreBoy commentedon Jul 14, 2010
The patch will need to be reworked for the 2.7, 3.1 and 3.2 branches.
dstanek commentedon Aug 3, 2010
I've reworked the patch so that it applied against the py3k branch. It's been attached to this issue and is also available here: http://codereview.appspot.com/1910044.
orsenthil commentedon Aug 4, 2010
Fixed in revision 83700 (release27-maint). r83701(py3k) and r83702(release31-maint).
David, thanks for reworking on the patch. Couple of comments
I made change to the original patch where I checked 'https:' and 'http:' kind of url with url.endswith(':') instead of len(url) == i+1 // I had a hard time to figure out why I had this way in the first place.
In py3k, the urlparse.urlparse is changed to urllib.parse.urlparse. So that changes were required in the tests.
orsenthil commentedon Oct 18, 2019
New changeset 5a88d50 by Senthil Kumaran (Tim Graham) in branch 'master':
bpo-27657: Fix urlparse() with numeric paths (#661)
5a88d50
miss-islington commentedon Oct 18, 2019
New changeset 82b5f6b by Miss Islington (bot) in branch '3.7':
bpo-27657: Fix urlparse() with numeric paths (GH-661)
82b5f6b
orsenthil commentedon Oct 18, 2019
New changeset 0f3187c by Senthil Kumaran in branch '3.8':
[3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) (bpo-16839)
0f3187c