Repository navigation
Cannot override 'connection: close' in urllib2 headers #57058
Description
| BPO | 12849 |
|---|---|
| Nosy | @jcea, @orsenthil, @bitdancer, @vadmium |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-08-28.19:19:27.961>
labels = ['invalid', 'type-bug', '3.7', 'expert-IO']
title = "Cannot override 'connection: close' in urllib2 headers"
updated_at = <Date 2020-03-09.07:00:08.605>
user = 'https://bugs.python.org/shubhojeetghosh'bugs.python.org fields:
activity = <Date 2020-03-09.07:00:08.605>
actor = 'henrik242'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2011-08-28.19:19:27.961>
creator = 'shubhojeet.ghosh'
dependencies = []
files = []
hgrepos = []
issue_num = 12849
keywords = []
message_count = 8.0
messages = ['143120', '170476', '211387', '221006', '243879', '250285', '363544', '363697']
nosy_count = 8.0
nosy_names = ['jcea', 'orsenthil', 'r.david.murray', 'martin.panter', 'shubhojeet.ghosh', 'sanxiago', 's7v7nislands@gmail.com', 'henrik242']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue12849'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.7']
Activity
shubhojeetghosh commentedon Aug 28, 2011
There seems to be an issue with urllib2
The headers defined does not match with the physical data packet (from wireshark). Other header parameters such as User Agent, cookie works fine.
Here is an example of a failure:
Python Code:
import urllib2
Wireshark:
GET / HTTP/1.1
Accept-Encoding: identity
Connection: close
Host: www.python.org
User-Agent: Python-urllib/2.6
bitdancer commentedon Sep 14, 2012
I've closed bpo-15943 as a duplicate of this one. As I said there, I'm not sure that we (can?) support keep-alive in urllib, though we do in httplib (which is the http package in python3).
[-]urllib2 headers issue[/-][+]Cannot override 'connection: close' in urllib2 headers[/+]vadmium commentedon Feb 17, 2014
I suggest using setdefault() in urllib.request.AbstractHTTPHandler.do_open():
I am trying to work around a server that truncates its response when this header is sent, and this change would allow me to specify headers={"Connection", "Keep-Alive"} to get the same effect as dropping the Connection header. This is also consistent with the way the other headers (Accept-Encoding, User-Agent, Host) may be overridden.
demianbrecht commentedon Jun 19, 2014
The problem here as far as I can tell is that the underlying file object (addinfourl) blocks while waiting for a full response from the server. As detailed in section 8.1 of RFC 2616, requests and responses can be pipelined, meaning requests can be sent while waiting for full responses from a server.
The suggested change of overriding headers is only a partial solution as it doesn't allow for non-blocking pipelining.
@martin Panter: My suggestion for you would simply be to use http.client (httplib) as R. David Murray suggests, which doesn't auto-inject the Connection header. Also, a server truncating responses when "Connection: close" is sent sounds like a server-side bug to me. Unless you're a server maintainer (or have access to the developers), have you tried reaching out to them to request a fix?
vadmium commentedon May 23, 2015
So far the only reasons that have been given to override this header (mine and the one in bpo-15943) seem to be to work around buggy servers. It is already documented that HTTP 1.1 and “Connection: close” are used, so if this issue is only about working around buggy servers, the best thing might be to close this as being “not a Python bug”. The user can always still use the low-level HTTP client, or make a custom urllib.request handler class (which is what I did).
Shubhojeet: What was the reason you wanted to set a keep-alive header?
If this is about proper keep-alive (a.k.a persistent) connection support in urllib.request, perhaps have a look at bpo-9740.
vadmium commentedon Sep 9, 2015
Just closed bpo-25037 about a server that omits the chunk length headers when “Connection: closed” is used.
I wonder if it would be such a bad idea to just remove the “Connection: closed” flag. It was added in 2004 in revision 5e7455fb8db6, but I do not agree with the reason given in the commit message and comment. Adding the flag is only really a courtesy to the server, saying it can drop the connection once it sends the response. Removing it in theory shouldn’t change anything about how the client parses the HTTP response, but in practice it seems it may improve compatibility with buggy servers.
henrik242 commentedon Mar 6, 2020
That mandatory "Connection: close" makes it impossible to POST a data request to Solr, as described in https://bugs.python.org/issue39875
It would be very helpful if it could be made optional.
henrik242 commentedon Mar 9, 2020
Correction: My problem in bpo-39875 was not related to Connection: Close, but with weird POST handling in Solr.
serhiy-storchaka commentedon Jul 3, 2026
This was already resolved as "not a bug" on bpo: hop-by-hop headers are controlled by the library, and real keep-alive support is a separate feature request. Closing.