Repository navigation
email.message_from_binary_file creates invalid RFC2047 encodings #142424
Description
Bug report
Bug description:
The following illustrates the issue.
>>> # Here's an example file
>>> with open('badx2.eml') as fp:
... print(fp.read())
...
From: "Sweepstakes Alert–Dyson Vacuum" <DysonVacuumSweeps@stellaryx.space>
body
>>> # Note that the `–` above is a unicode En Dash
>>> # Make a message object
>>> from email import message_from_binary_file
>>> with open('badx2.eml', 'rb') as fp:
... msg = message_from_binary_file(fp)
...
>>> # and look at it
>>> print(msg.as_string())
From: =?unknown-8bit?q?=22Sweepstakes_Alert=E2=80=93Dyson_Vacuum=22_=3CDysonVacuumSweeps=40stellaryx=2Espace=3E?=
body
>>> # Note that the email address is improperly included in the encoded wordCPython versions tested on:
3.13
Operating systems tested on:
Linux
Activity
fatelei commentedon Dec 11, 2025
UTF‑8 headers (requires server support)
print(msg.as_string(policy=policy.SMTPUTF8))
ASCII‑safe (works everywhere)
print(msg.as_string(policy=policy.SMTP))
try
medmunds commentedon Dec 13, 2025
I believe this is working as expected, and should be closed as not a bug.
A (utf-8 encoded, 8-bit) en dash is not a valid character in an email header, unless the message is being handled by an SMTP server that specifically advertises support for the smtputf8 extension (see RFC 6531 and 6532). If you are processing a message with RFC 6532 8-bit headers, you must share that knowledge with Python's email package by using a policy with
EmailPolicy.utf8set True. Example:Without that knowledge, the email parser has no way of knowing if the 8-bit characters in the header are utf-8, or iso-8859-1, or some other encoding. So it identifies them as using an "unknown-8bit" charset.
msapiro commentedon Dec 13, 2025
It is not working as expected. The issue is not that the en dash is encoded as unknown-8bit. The issue is that the entire header including the email address is RFC 2047 encoded. Email addresses MUST not be RFC 2047 encoded.
medmunds commentedon Dec 13, 2025
That is absolutely correct. Sorry I overlooked that in your report.
But this is essentially garbage in, garbage out (invalid header in → different invalid header out). Given that it's occurring in the legacy compat32 email parser and generator, I wouldn't expect it to be fixed.
When you call message_from_bytes() without specifying a policy, you're getting the legacy
compat32policy. The legacy parser does not support 8-bit headers. And the legacy generator's as_string() dates from the Python 2 days and largely unaware of Unicode strings.Note that you can get the original (invalid) header back from the legacy generator's as_bytes():
Switching to the modern
email.policy.default(which is confusingly not the default policy) provides more tolerant handling of the invalid input, plus RFC 2047 compliant output:msapiro commentedon Dec 13, 2025
I understand that.
The example I posted was a contrived minimal example. The actual issue arises in Mailman 3.
A much larger message with that From: header is posted to a list (we have also seen other messages with non-ascii display names). It is converted to a mailman.email.message.Message object (a subclass of Python's email.Message class) using compat32 policy. This message object is processed completely okay through Mailman, but it is converted to bytes by the as_bytes() method and passed to the HyperKitty archiver which converts it to an email.EmailMessage object using policy default, and it is there that the issue manifests and causes archiving to fail.
For various historical reasons, we are reluctant to change this process.
Granted, this may not be a serious issue as the original messages are non- compliant, but in any case, the email package should not be creating invalid encodings. At a minimum it should flag this with a defect in the message object.
medmunds commentedon Dec 13, 2025
Hmm, is Mailman also using compat32 as_string() somewhere in the process?
compat32 as_bytes() does not seem to apply rfc2047 to the addr-spec—see example in my previous comment. Which is why I'm speculating some other operations must be involved. (Does Mailman override any header field methods?)
The modern email parser does flag it as a defect—example in my previous comment. (Field-level defects are not propagated up to the message object, which is bug #106144. And I don't think the legacy Message object has any way to access field-level defects.)
msapiro commentedon Dec 14, 2025
I know that as_bytes() does not rfc2047 encode the header in my 3 line example, but in the case of a real message it does.