http.client._MAXHEADERS = 100 limit no longer sufficient

Bug report

Bug description:

This hard-coded sanity check for HTTP response headers is no longer sufficient to fetch a Microsoft 365 page.

Line 112 in 0a91456

_MAXHEADERS = 100

I do have a case open with Microsoft Support (TrackingID#2503130010002871), but it's not getting much traction as it's not causing problems with full web browsers.

Steps to reproduce:

import http.client
con = http.client.HTTPSConnection('outlook.office365.com')
con.request("GET", "/owa/example.edu")  # any domain seems to trigger
r = con.getresponse()

And that throws a HTTPException

>>> con = http.client.HTTPSConnection('outlook.office365.com')
>>> con.request("GET", "/owa/foo.bar")
>>> r = con.getresponse()
Traceback (most recent call last):
  File "<python-input-8>", line 1, in <module>
    r = con.getresponse()
  File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 1428, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 350, in begin
    self.headers = self.msg = parse_headers(self.fp)
                              ~~~~~~~~~~~~~^^^^^^^^^
  File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 248, in parse_headers
    headers = _read_headers(fp)
  File "C:\Users\jmacdone\AppData\Local\Programs\Python\Python313-arm64\Lib\http\client.py", line 226, in _read_headers
    raise HTTPException("got more than %d headers" % _MAXHEADERS)
http.client.HTTPException: got more than 100 headers
>>>

It seems to be just spilling over with 101 headers. Though, not consistently. Presumably it depends upon which load balancer node is responding.

$ curl --silent -D - 'https://outlook.office365.com/owa/example.edu' | grep -E "^[a-zA-Z-]+: " | wc -l

returns with 96, 99, 101, etc. headers, depending on ~~Microsoft's mood~~ unknown factors.

For background, it's common to use https://outlook.com/example.edu as a domain hint ("smart link") to go directly to a tenant's identity provider and avoid the "Please provide your email address" step. We have a nagios check for that, which broke recently as the number of Set-Cookie: OpenIdConnect.token.[...] variants continues to grow.

CPython versions tested on:

3.13, 3.9, 3.11

Operating systems tested on:

Linux, Windows

Linked PRs

gh-131724: Add a new max_response_headers param to HTTP/HTTPSConnection #136814

This seems to only restrict HTTPSConnection and HTTPConnection is not affected.

@StanFromIreland

Just to help clarify, picking a new arbitrary limit allows getresponse() to complete without an exception.

>>> import http.client
>>> http.client._MAXHEADERS = 120
>>> con = http.client.HTTPSConnection('outlook.office365.com')
>>> con.request("GET", "/owa/example.edu")  # any domain seems to trigger
>>> r = con.getresponse()
>>> r.read()
b'<html><head><title>Object moved</title> [...]'

@StanFromIreland I don't have an explicit test case, but I think this would affect both classes. HTTPSConnection.getresponse() is inherited from HTTPConnection.getresponse()

cpython/Lib/http/client.py

Line 1454 in 0a91456

class HTTPSConnection(HTTPConnection):

Maybe we should add a parameter to customize this limit?

I tried with both 3.11 and 3.15 and did not get an exception

I tried with both 3.11 and 3.15 and did not get an exception

@GGyll Confirmed. It does indeed appear to be just under the limit today

>>> len(r.headers)
95

And, for completeness, Microsoft closed my ticket a few weeks ago: "working as intended". They're happy they're setting 60+ cookies.

For the time being, my workaround is setting the module's "private" variable as show in #131724 (comment)

I suppose that could be a patch option. Rename _MAXHEADERS to MAXHEADERS and document it as part of the module's public interface. Or, maybe it's not worth bothering with just to accommodate one page from one vendor?

Maybe we should add a parameter to customize this limit?

I mean adding a parameter to HTTPConnection and HTTPSConnection constructors. If not specified, the default value would be http.client._MAXHEADERS.

Adding a parameter sounds like a good approach. I can have a look at this.

pythongh-131724: Add a new max_headers param to HTTP/HTTPSConnection …

pythongh-131724: Add a new max_headers param to HTTP/HTTPSConnection

Merge branch 'main' into pythongh-131724

gh-131724: Add a new max_response_headers param to HTTP/HTTPSConnecti…

@aeurielesn

Thank you for the feature, @aeurielesn!

pythongh-131724: Add a new max_response_headers param to HTTP/HTTPSCo…

Navigation Menu

Repository navigation

http.client._MAXHEADERS = 100 limit no longer sufficient #131724

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

http.client._MAXHEADERS = 100 limit no longer sufficient #131724

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

StanFromIreland commented on Mar 25, 2025

jmacdone commented on Mar 28, 2025

vstinner commented on Mar 28, 2025

gostak-dd commented on May 20, 2025

jmacdone commented on May 20, 2025

vstinner commented on May 20, 2025

aeurielesn commented on Jul 19, 2025

encukou commented on Jul 22, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions