Repository navigation
socket module calls with long host names can fail with idna codec error #77139
Description
| BPO | 32958 |
|---|---|
| Nosy | @gpshead, @bitdancer, @bdarnell, @jhasapp, @agnosticdev, @alexmv |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2018-02-26.19:52:35.417>
labels = ['3.8', '3.7', 'library']
title = 'socket module calls with long host names can fail with idna codec error'
updated_at = <Date 2022-01-25.00:36:47.690>
user = 'https://bugs.python.org/ablack'bugs.python.org fields:
activity = <Date 2022-01-25.00:36:47.690>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-02-26.19:52:35.417>
creator = 'ablack'
dependencies = []
files = []
hgrepos = []
issue_num = 32958
keywords = []
message_count = 10.0
messages = ['312947', '313163', '313164', '313323', '372865', '373006', '374207', '391990', '393323', '411539']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'r.david.murray', 'Ben.Darnell', 'joseph.hackman', 'agnosticdev', 'ablack', 'sdbowman', 'midopa', 'alexmv']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue32958'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']
Activity
ablack commentedon Feb 26, 2018
While working on a custom conda channel with authentication, I ran into the following UnicodeError:
The error can be consistently reproduced when the first substring of the url hostname is greater than 64 characters long, as in "0123456789012345678901234567890123456789012345678901234567890123.example.com". This wouldn't be a problem, except that it doesn't seem to separate out credentials from the first substring of the hostname so the entire "[user]:[secret]@xxx" section must be less than 65 characters long. This is problematic for services that use longer API keys and expect their submission over basic auth.
ned-deily commentedon Mar 2, 2018
Thanks for the report. The behavior you see can be further isolated to socket.gethostbyname:
Other socket module calls accepting host names fail similarly, such as getaddrinfo.
[-]Urllib proxy_bypass crashes for urls containing long basic auth strings[/-][+]socket module calls with long host names can fail with idna codec error[/+]ablack commentedon Mar 2, 2018
Just to be clear, I don't know if the socket needs to support 64 character long host name sections, so here's an example url that is at the root of my problem that I'm pretty sure it should support:
agnosticdev commentedon Mar 6, 2018
Using Ubuntu 16.04 with the 3.6.0 tag I was also able to reproduce the same error reported:
It looks like the hostname being 64 characters long is the issue in that it cannot be encoded. Thus falling into the UnicodeError being raised in idna.py:
# ASCII name: fast path
labels = result.split(b'.')
for label in labels[:-1]:
if not (0 < len(label) < 64):
raise UnicodeError("label empty or too long")
if len(labels[-1]) >= 64:
raise UnicodeError("label too long")
return result, len(input)
I did some work on this to try and resolve this, but ultimately it was not worth committing so I wanted to report my findings.
sdbowman commentedon Jul 2, 2020
When will this issue be fixed? Thanks!
jhasapp commentedon Jul 5, 2020
According to the DNS standard, hostnames with more than 63 characters per label (the sections between .) are not allowed [https://tools.ietf.org/html/rfc1035#section-2.3.1].
That said, enforcing that at the codec level might be the wrong choice. I threw together a quick patch moving the limits up to 250, and nothing blew up. It's unclear what the general usefulness of such a change would be, since DNS servers probably couldn't handle those requests anyway.
As for the original issue, if anybody is still doing something like that, could they provide a full example URL? I was unable to reproduce on HTTP (failed in a different place), or FTP.
ablack commentedon Jul 24, 2020
joseph.hackman
I don't think that the 63 character limit on a label is the problem specifically, merely it's application.
The crux of my issue was that credentials passed with the url in a basic-authy fashion (as some services require) count against the label length. For example, this would trigger the error:
Since the first label would be treated as:
"ablack:very_long_api_key_0123456789012345678901234567890123456789012345678901234567890123@www"
My specific issue goes away if any text up to / including an "@" in the first label section is not included in the label validation. I don't know off hand if that information is supposed to be included per the label in the DNS spec though.
alexmv commentedon Apr 26, 2021
It seems reasonable to fail on hostnames that are too long -- but it feels like the weirdness is that it is categorized as a UnicodeError, and not as, say, a ValueError.
Would a re-categorization as ValueError seem like a reasonable adjustment here?
bdarnell commentedon May 9, 2021
[I'm coming here from https://github.com/tornadoweb/tornado/pull/3010)
UnicodeError is a subclass of ValueError, so I don't see what value that change would provide. The thing that's surprising to me is that it's not a
socket.herror(orgaierrorfor socket.getaddrinfo). I guess the docs don't formally say thatherror/gaierroris the only possible error from these functions, butgaierrorwas the only error I was catching so the unexpected UnicodeError escaped the layer that was intended to handle it.I do think that in the special case of
getaddrinfowith theAI_NUMERICHOSTflag it should be handled differently: in that mode there is no network access necessary and it's reasonable to assume that the only possible error is agaierrorwithEAI_NONAME.I'd like to at least see better documentation about what errors are possible from this family of functions.
gpshead commentedon Jan 25, 2022
ablack: the basic auth username:password@ part of the string is not part of a hostname. What code are you seeing that is trying to send that to a name resolver rather than stripping the obviously private info up through the @ sign?
eikesauer commentedon Oct 25, 2024
Still getting a crash on long API keys six years after the original report...
o-oconnell commentedon Oct 21, 2025
Just wanted to +1 this issue and mention it's causing all of my certbot renewals to fail. Extremely painful.
Certbot specific thread: https://community.letsencrypt.org/t/attempting-to-generate-cert-getting-unicodeerror-encoding-with-idna-codec-failed/61675/4
Original Python issue from 2018 (!!!): https://bugs.python.org/issue32958
gpshead commentedon Oct 23, 2025
per prior comments, the same question remains: What code is passing the basicauth username:password@ prefix in to the DNS resolver as a part of the hostname? That is where the bug comes from. Nobody has provided example code that can reproduce this issue.
Individual parts of a FQDN (a "Label" in original DNS RFC terms) such as the hostname MUST be <= 63 bytes in length. Per the DNS spec.
The above examples people have posted are all examples of "yes, the length limit is enforced is the spec requires" or "this wasn't actually a valid name to lookup but we passed it to the resolver anyways".
idna refusing to encode something longer than 63 bytes is by design per the spec it is intended to use for. The exception happens when people give the codec more data than they strictly should. forgetting to remove basic:auth@ prefix, or forgetting to remove a trailing :portnumber for example (based on linked issues such as this one).
Python is doing the right thing with the idna codec and resolver, exposing bugs higher in your stacks.
I'm closing this as nobody has provided a reproducer showing that Python stdlib itself is doing anything out of spec. (they could be situations in our own standard library url handling and http libraries that get this wrong... if you find those, please re-open with specific examples)