Skip to content

urllib does not send http/1.1 ALPN extension #85140

Closed
Closed
urllib does not send http/1.1 ALPN extension#85140
Labels
3.10 (EOL)end of life3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@pmenzel
pmenzel
mannequin
opened on Jun 13, 2020
BPO 40968
Nosy @orsenthil, @tiran
PRs
  • bpo-40968: Send http/1.1 ALPN extension #20959
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-06-13.11:10:14.395>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'urllib does not send http/1.1 ALPN extension'
updated_at = <Date 2020-11-13.15:38:00.535>
user = 'https://bugs.python.org/pmenzel'

    bugs.python.org fields:

    activity = <Date 2020-11-13.15:38:00.535>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-13.11:10:14.395>
creator = 'pmenzel'
dependencies = []
files = []
hgrepos = []
issue_num = 40968
keywords = ['patch']
message_count = 5.0
messages = ['371450', '371619', '371620', '371621', '380899']
nosy_count = 3.0
nosy_names = ['orsenthil', 'christian.heimes', 'pmenzel']
pr_nums = ['20959']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40968'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

    Activity

    1. pmenzel commented on Jun 13, 2020

      @pmenzel
      pmenzelmannequin
      on Jun 13, 2020
      MannequinAuthor

      Having the TURN server Coturn 1 set up in a Jitsi Meet installation, Python’s urllib requests fail, while it works with cURL and browsers.

      $ curl -I https://jitsi.molgen.mpg.de
HTTP/2 200 
server: nginx/1.14.2
date: Sat, 13 Jun 2020 11:09:19 GMT
content-type: text/html
vary: Accept-Encoding
strict-transport-security: max-age=63072000


      >>> import urllib.request
>>> response = urllib.request.urlopen('https://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
    2. pmenzel
      added
      3.8 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Jun 13, 2020

    3. tiran commented on Jun 16, 2020

      @tiran
      tiran
      on Jun 16, 2020
      Member

      It looks like the server is refusing requests that don't have an ALPN extension set in TLS layer. At first I though that the server may only support HTTP/2. The curl requests in your examples uses HTTP/2. urllib only supports HTTP/1.1 and HTTP/1.0.

      Then I tried "curl --http1.1 https://jitsi.molgen.mpg.de". The request also succeeds because curl is sending "ALPN, offering http/1.1".

      This request works for me:

      >>> import ssl
>>> import urllib.request
>>> ctx = ssl.create_default_context()
>>> ctx.set_alpn_protocols(['http/1.1'])
>>> urllib.request.urlopen('https://jitsi.molgen.mpg.de', context=ctx)
<http.client.HTTPResponse object at 0x6040009b7960>

      urllib could set the ALPN header by default when the user does not supply a custom context. It looks like curl always adds an ALPN extension. I don't see code in Python requests that sets ALPN extension. On the other hand Tom Christie's excellent httpx library does set ALPN extension just like curl.

    4. pmenzel commented on Jun 16, 2020

      @pmenzel
      pmenzelmannequin
      on Jun 16, 2020
      MannequinAuthor

      Wow, great job in figuring this out. Passing the context, it works here too.

      I just wanted to add, that I looked at this with Wireshark, and to not complicate things, first tried plain HTTP.

      $ curl http://jitsi.molgen.mpg.de
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

      With Python, there is also the connection lost error. But Wireshark shows that the 301 HTTP packet is send back by the server.

      $ python3
Python 3.8.3 (default, May 14 2020, 11:03:12) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> response = urllib.request.urlopen('http://jitsi.molgen.mpg.de')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    r = h.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

    5. tiran commented on Jun 16, 2020

      @tiran
      tiran
      on Jun 16, 2020
      Member

      Look closer at the traceback. You'll see that urllib follows redirects (http_response -> http_error_302 -> open -> HTTPSConnection). curl doesn't follow redirects by default.

    6. tiran
      added
      3.7 (EOL)end of life
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      on Jun 18, 2020
    7. tiran
      changed the title [-]urllib is unable to deal with TURN server infront[/-] [+]urllib does not send http/1.1 ALPN extension[/+] on Jun 18, 2020
    8. tiran
      added
      3.7 (EOL)end of life
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      on Jun 18, 2020
    9. tiran
      changed the title [-]urllib is unable to deal with TURN server infront[/-] [+]urllib does not send http/1.1 ALPN extension[/+] on Jun 18, 2020

    10. tiran commented on Nov 13, 2020

      @tiran
      tiran
      on Nov 13, 2020
      Member

      New changeset f97406b by Christian Heimes in branch 'master':
      bpo-40968: Send http/1.1 ALPN extension (bpo-20959)
      f97406b

    11. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    12. furkanonder commented on May 10, 2023

      @furkanonder
      furkanonder
      on May 10, 2023
      Contributor

      @tiran Issue seems to be closed. We can close the issue.

    13. orsenthil
      closed this as completedon May 10, 2023
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions