email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default

Bug report

Bug description:

When using email.parser with the modern email.policy.default, the parser incorrectly inserts a space between adjacent rfc2047 encoded-words that are separated by ~~folding~~ whitespace. This can result in splitting words or names in unexpected places in the parsed headers.

Example:

Python 3.13.1 (main, Dec  3 2024, 17:59:52) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
>>> from email import message_from_bytes, policy
>>> from email.message import EmailMessage

>>> address = "Bérénice-Amélie Rosemonde Dûbois-Bénard <rose@example.com>"
>>> message = EmailMessage()
>>> message["From"] = address
>>> message_bytes = message.as_bytes()

>>> default_parsed = message_from_bytes(message_bytes, policy=policy.default)
>>> default_parsed_from = default_parsed["From"].addresses[0]
>>> assert default_parsed_from == address
Traceback (most recent call last):
  File "<python-input-22>", line 1, in <module>
    assert default_parsed_from == address
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError

>>> print(default_parsed_from); print(address)
Bérénice-Amélie Rosemonde Dûbois-Béna rd <rose@example.com>
Bérénice-Amélie Rosemonde Dûbois-Bénard <rose@example.com>

>>> print(default_parsed["From"].addresses[0].display_name)
Bérénice-Amélie Rosemonde Dûbois-Béna rd

Notice the unexpected space in the parsed version. (I have more-problematic examples involving variations of the Scunthorpe problem, but they're not suitable for GitHub. This example uses Python's modern email API to generate the message, but the bug applies to parsing messages from any source.)

This seems to be caused by two consecutive rfc2047 encoded-words [edit: whether or not they cross a fold, see later comment]:

>>> print(message_bytes.decode())
From: =?utf-8?b?QsOpcsOpbmljZS1BbcOpbGllIFJvc2Vtb25kZSBEw7tib2lzLULDqW5h?=
 =?utf-8?q?rd?= <rose@example.com>

RFC 2047 section 6.2 requires that space that separates adjacent encoded-words be ignored. (Specifically to allow splitting encoded-words at any character—behavior that the policy.default email.generator relies on.)

Note that the legacy compat32 parser did not have this problem. (The legacy parser requires separately calling legacy email.header.decode_header() to decode rfc2047.)

>>> from email.header import decode_header
>>> compat32_parsed = message_from_bytes(message_bytes, policy=policy.compat32)
>>> compat32_parsed_from = "".join(
...     segment if charset is None and isinstance(segment, str)
...     else segment.decode(charset or "ascii")
...     for segment, charset in decode_header(compat32_parsed["From"])
... )
>>> assert compat32_parsed_from == address  # (success)
>>> print("original:", address); print("compat32:", compat32_parsed_from); print(" default:", default_parsed_from)
original: Bérénice-Amélie Rosemonde Dûbois-Bénard <rose@example.com>
compat32: Bérénice-Amélie Rosemonde Dûbois-Bénard <rose@example.com>
 default: Bérénice-Amélie Rosemonde Dûbois-Béna rd <rose@example.com>

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux, macOS

Linked PRs

@medmunds

@medmunds you are working on fixing this ? If not, I am working on it

@srinivasreddy

@srinivasreddy I already have a bunch of email-related PRs that have been stuck awaiting review for months, so I'm not working on new PRs until those move forward. If you have a fix for this issue, that's great!

On further investigation, it seems parsing with policy.default fails to remove "any 'linear-white-space' that separates a pair of adjacent 'encoded-word's" whether or not folding is involved:

>>> message_from_bytes(b"From: =??q?ab?= =??q?cd?= <foo@example>\n", policy=policy.default)["From"]
'ab cd <foo@example>'
# expected: 'abcd <foo@example>'

I started the work on it.

Also, it looks like the problem only affects address headers:

>>> # Subject header is correct:
>>> message_from_bytes(b"Subject: =??q?ab?= =??q?cd?=\n", policy=policy.default)["Subject"]
'abcd'
>>> # Address header has the bug:
>>> message_from_bytes(b"To: =??q?ab?= =??q?cd?= <foo@example>\n", policy=policy.default)["To"]
'ab cd <foo@example>'

Until this gets fixed, here's a workaround that can repair an EmailMessage with misparsed headers. And a short check to see if you need the workaround.

@srinivasreddy

@srinivasreddy are you still working on this?

@medmunds

@medmunds apologies . I didn’t get enough time for this . This ticket is up for grabs .

pythongh-128110: Fix rfc2047 handling in email parser address headers

Also, it looks like the problem only affects address headers:

Not sure if related or not, but Message-ID also has a whitespace problem:

#134812

Refs #35581 -- Added workaround for Python bug in mail tests.

I think the issue affects Subject as well:
[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost
gets encoded with add_header() as:
b'Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje \n =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= zranitelnost\n\n'
which gets decoded as aktivnězneužívanou instead of aktivně zneužívanou.

When I change the string to aaaaaaaaaaa[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost, the issue changes and it produces 2 spaces between opravuje and aktivně.

@multiflexi

I think the issue affects Subject as well:
[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost
gets encoded with add_header() as:
b'Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje \n =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= zranitelnost\n\n'
which gets decoded as aktivnězneužívanou instead of aktivně zneužívanou.

@multiflexi I can't reproduce that in Python 3.14.4 (or 3.15.0a8). The space between the two words seems to be correctly encoded as _ inside a single encoded-word:

>>> from email import policy, message_from_bytes
>>> from email.message import EmailMessage
>>> msg = EmailMessage()
>>> subject = "[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost"
>>> msg.add_header("Subject", subject)
>>> print(msg.as_bytes().decode())
Subject: [TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje
 =?utf-8?q?aktivn=C4=9B_zneu=C5=BE=C3=ADvanou?= zranitelnost
>>> ### The space gets ^ encoded as '_'
>>> parsed = message_from_bytes(msg.as_bytes(), policy=policy.default)
>>> parsed["Subject"]
'[TLP:CLEAR] [Max CVSS 10.0] Cisco Catalyst SD-WAN opravuje aktivně zneužívanou zranitelnost'
>>> parsed["Subject"] == subject
True

If you're still seeing it, please open a new issue with the exact Python version you're using and sample code to reproduce the problem. If the subject you show gets encoded as ... =?utf-8?q?aktivn=C4=9B?= =?utf-8?q?zneu=C5=BE=C3=ADvanou?= ..., that's a bug in the email generator. (That encoded subject would correctly decode as ... aktivnězneužívanou ... because the space between the encoded-words should be ignored per rfc2047.)

gh-128110: Fix rfc2047 whitespace handling in email parser address he…

[3.15] gh-128110: Fix rfc2047 whitespace handling in email parser add…

[3.14] gh-128110: Fix rfc2047 whitespace handling in email parser add…

[3.13] gh-128110: Fix rfc2047 whitespace handling in email parser add…

Navigation Menu

Repository navigation

email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default #128110

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

2 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

email.parser can insert extraneous spaces when parsing rfc2047 headers with policy.default #128110

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

srinivasreddy commented on Dec 20, 2024

medmunds commented on Dec 20, 2024

medmunds commented on Dec 22, 2024

srinivasreddy commented on Dec 23, 2024

medmunds commented on Dec 23, 2024

medmunds commented on Dec 24, 2024

medmunds commented on Feb 27, 2025

srinivasreddy commented on Feb 28, 2025

jakajancar commented on May 27, 2025

2 remaining items

multiflexi commented on Feb 26, 2026

medmunds commented on May 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions