AddressHeader problem when FriendlyName is improperly quoted and contains comma

@smith

Bug report

Bug description:

Given an address text of the form: John Smith, Jr <johnsmith@example.com>
Because of the unquoted comma, the text is parsed as two addresses:
"John" <@smith>, "Jr" <johnsmith@example.com>
This is of greatest concern if the object is supposed to be a SingleAddressHeader.

I have also encountered TO and CC strings that have incomplete address specifications, containing only a Friendly Name. For these messages, I am only interested in the valid addresses, although others may want the existing behavior.

To work around both the parsing problem and the invalid address terms, I created a sub-class which discards any address where username or domain are not strings. My workaround does not correct for the incomplete Friendly Name, but it ensures a correct count of valid addresses. It would be desirable to implement a trap for unquoted commas, but there are so many challenges to parsing address lists correctly, I wonder if fixing the unquoted comma problem will create other headaches.

Problem observed on Python 3.11.3150.0 and Python 3.8.1

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Please show us a reprocible example. Is it the email module?

Yes, I am referring to the email module, and the AddressHeader object with its variants.

Second issue first:
I observed a long CC list which was apparently truncated by Outlook due to length. It ended with:
, "john doe"
The final term was parsed correctly as a DisplayName with no address. It is a philosophical question of "what do we want the software to do with an email address object that lacks an email address?" For my purposes, I want to discard it, but the addresses object is a tuple, not a list, so the only way to alter the result is to clone it into a list object. I don't have a specific recommendation for development, but wanted to document the issue that I observed.

The first problem is also caused by malformed content, but it is more common. Display Names are not required to be quoted, and often are not. But if a comma appears in the DisplayName, it should be quoted so that it is not treated as an address term delimiter. Unfortunately, this simple rule is often violated. The email module splits the list at the comma, then attempts to construct two address headers from the results. The result is one address object with no email address, and another address with a portion of the intended DisplayName text

As far as an example, I don't know what you need beyond the text that I provided. You should be able to drop it into any EML file as a From header or Reply-To header for testing. I don't have freedom to post an actual email message in a public forum.

Please share a reproducer with fake addresses then. Issues only with a wall of text are hard to understand because we do not have any real information (what is the current output (traceback, error message, etc) and what is the expected one?) or what is your workaround?

You should be able to drop it into any EML file as a From header or Reply-To header for testing.

This is your reponsibilty to produce such example. We do not have the bandwidth to investigate cases that are not described without exactly knowing what happened.

I will close the issue if no further explanation that we can use to help you is given.

testparse1.py
addresslist.txt

The AddressList.Txt file has the same content for both TO and CC headers
The TO header is parsed using my child object which discards results with no email address. It returns on addresslist entry:
header: to sequence: 1 displayname: Jr user: johnsmith domain: example.com

The CC header is parsed using the standard address header and returns three address entries:
header: cc sequence: 1 displayname: user: John Smith domain:
header: cc sequence: 2 displayname: Jr user: johnsmith domain: example.com
header: cc sequence: 3 displayname: user: John Doe domain:

Could you show the contents of the file directly by using <details> and block quotes as well please? Readers (including me) do not like downloading contents like that. It is also better for posterity to have it directly in the issue description. TiA.

Trying again

Test data for addresslist.txt file:
to: John Smith, Jr johnsmith@example.com, "John Doe"
cc: John Smith, Jr johnsmith@example.com, "John Doe"

Sample program:

Details

from email import _header_value_parser as headparser

from email.message import EmailMessage
from email.parser import BytesParser, Parser
from email.policy import default
from email.policy import EmailPolicy
from email.policy import SMTP
from email.headerregistry import HeaderRegistry

from email.header import decode_header

from email.headerregistry import UnstructuredHeader
from email.headerregistry import UniqueAddressHeader
from email.headerregistry import SingleAddressHeader
from email.headerregistry import UniqueSingleAddressHeader
from email.headerregistry import AddressHeader
from email.headerregistry import DateHeader
from email.headerregistry import UniqueDateHeader
from email.generator import BytesGenerator

from email.utils import parsedate_to_datetime

from email.headerregistry import Address

from email.utils import parsedate_tz
from email.utils import mktime_tz
import re

class MyAddressHeader(AddressHeader):

    def init(self,*args,**kw):
        super().init(*args, **kw)
        self.addresslist = []
        for addr in self.addresses:
            self.addresslist.append(addr)
        itms = len(self.addresslist)
        for itm in range(itms):
            idx = itms - itm - 1
            addr = self.addresslist[idx]
            username = addr._username
            domain = addr._domain
            displayname = addr._display_name
            if isinstance(username,str) and isinstance(domain,str):
                if len(username) > 0 and len(domain) > 2:
                    pass
                else:           # Discard invalid addresses
                    self.addresslist.pop(itm)
        # Bottom of for address list loop
# End of Class definition

def CreatePolicy():
    MyReg = HeaderRegistry()
    MyReg.map_to_type('to',MyAddressHeader)
    kw = {"header_factory" : MyReg , "max_line_length" : 990}
    MyPol = default.clone(**kw)
    return MyPol, MyReg

def LoadMsg(filename,MyPol):
    fileobj = open(filename,"rb")
    BP = BytesParser(policy=MyPol)
    emailmsg = BP.parse(fileobj)
    fileobj.close()
    return emailmsg

MyPol, MyReg = CreatePolicy()
emlfile = 'addresslist.txt'
emailmsg = LoadMsg(emlfile,MyPol)
print(emailmsg)
for hdritem in emailmsg._headers:
    Pattern2 = "[\s*\\r*\\n]+\\s+"
    hdrname = hdritem[0].lower()
    hdrtext = hdritem[1]
    hdrtext = re.sub(Pattern2," ",hdrtext)
    hdrobj = MyPol.header_fetch_parse(hdrname,hdrtext)
    addrcount = 0
    if hdrname == 'to':
        for addr in hdrobj.addresslist:
            addrcount = addrcount + 1
            print("header:", hdrname, "sequence:", addrcount, "displayname:", addr.display_name, "user:", addr.username, "domain:", addr.domain)
    if hdrname == 'cc':
        for addr in hdrobj.addresses:
            addrcount = addrcount + 1
            print("header:", hdrname, "sequence:", addrcount, "displayname:", addr.display_name, "user:", addr.username, "domain:", addr.domain)

@bitdancer

Thanks. This looks lile you are accessing implementation details to fix the issue but I also wonder whether those address lists are valid in the first place (that is, are they RFC compliant?)

We need to know whether this is a bug in the parser (ie, legitimate emails that are documented as accepted failed to be parsed) or if it is a case of "garbage in/garbage out" (GIGO). As I am currently travelling I will ping the email maintainer cc @bitdancer

I feel it could be both a bug and a feature. A bug because we fail to handle "bad" common cases but also a feature because those cases are already invalid in the first place. I do not know if it is also possible to avoid splitting on a comma that was not quoted (is the quoting of comma an RFC requierement? if so I do not think we should be smarter and consider this as GIGO case). How common are those cases? Outlook seems to truncate things but does it truncate names with unquoted commas? (that is what happens with your example of "Jr" with Outlook for instance?)

An alternative is that we consume as many characters as we can until the email address and consider it as the display name but... it may be very backward incompatible. I will leave the design and feasibility questions to @bitdancer

I thought that I had been clear that this is GIGO. Since comma is a specified delimiter, the display name should be quoted.

I think the root problem is that better products will always quote the display name. while inferior implementations never apply quotes. From a coding standpoint, there is no reason for a developer to create conditional logic when quoting is always acceptable. However, the presence of a comma in the displayname is determined by the user, not by the software, so the combination of an inferior product and a creative user produces the GIGO result.

Frequency: Not particularly common, but the offenders will tend to repeat.
From a reference set of 235,000 accepted (non-spam) email messages, I have:
10 unique examples of mangled to/cc from messages because long lists of recipients were not handled cleanly by the originating email client.
2 unique examples of Sender strings with unquoted commas,
2 unique examples of Reply-to with unquoted comma,, and
2 unique examples of Reply-to with invalid content because substitution tokens were not substituted with valid data.

I do not collect this level of detail on blocked messages.

History: I have been dumping all of my parsed addresses into a database. A review of database contents exposed these records with invalid email addresses. After investigating to see if there was a bug in my parser, I determined that the problem was with the email module's difficulty when parsing non-compliant text. As indicated in the first note and in my example, I have implemented a workaroud.

In this case I do not think we will change the module. I do not see it therefore as a bug and considering the feature request is niche, it is not worth a change on our side. I actually do not understand the exact purpose of the issue if you know that it is GIGO:

do you want to share your implementation? in this case it should be done in PyPI.
do you want us to support this bad case? in this case it is a feature request but this is too niche IMO and thus not worth it in the stdlib. It should be done as a PyPI package as well.
do you want to report it as a bug? as you said it is GIGO and GIGO cases are not considered as bugs.

So I do not think CPython has any actionable buttons for this issue (supporting it means maintaining it, but not all features are meant to be included in the stdlib).

While everything picnixz says is correct, in fact the current parser does have at least a version of the information you are looking for, it just isn't currently exposed. This appears to be an oversight on my part, since I fully intended to expose it. We can do so by adding a 'valid' attribute to Address, and changing the 'parse' method of AddressHeader to include valid=mb.token_type=='mailbox' in the list comprehension that creates the Address object. Invalid mailboxes have the token_type 'invalid-mailbox'.

We could also add a valid_addresses attribute that only returned addresses that were valid, though after the change above writing [x for x in hdr.addresses if x.valid] would be pretty easy. I think I originally intended for there to be such an attribute and it got lost somewhere along the way.

Of course, since this feature could never be used to this point, we may well have cases where valid mailboxes are detected as invalid and vice versa...in fact I know that this is true for at least a couple corner cases, mostly involving encoded words. I have a PR in the works that will among other things fix the cases I have discovered, but it may be a while before it arrives.

As far as "a trap for unquoted commas", I have mixed feelings about that. You could have a special rule for single address headers, but on balance I think it is probably better to go for consistency in parsing. You can still access the list of addresses and look for a valid one once we add this 'valid' flag.

Ignoring the parsing problem is reasonable, although a future documentation note would be useful. Between the flexibility allowed by IETF specifications and the imperfections that occur in the wild, the parsing problem seems overwhelming:

any or all of the text may be UTF-8 encoded Unicode.
Quotes, angle brackets, commas, and "at" symbol are all delimiters, but any delimiter may be included in a quoted string.
Username/local-part may be a quoted string that contains any of the other delimiters.
Domain names may be punycode or Unicode.
Real-world implementations may omit delimiters that they deem optional.

I have been impressed that this is the only problem I have detected, although mercifully I do not usually deal with internationalied text. I was surprised yesterday when my review showed that the problem had not occurred on a From header, only on Sender and Reply-to, where display-name is minimally important.

I detected limited use of "Reply-to: <>", which is apparently intended to force no-reply. I wonder if that format would benefit from special handling, since it is expected to produce different behavior than absence of a Reply-To.

To distinguish between valid and invalid addresses, I favor creating a second property that contains only valid addresses. The beauty of a well-designed object is that developers don't need to track supplemental information about each instance of the object. The valid/invalid indicator is also a good idea.

Overall, I don't have a strong emotion about any of your options, since I have found a solution for myself. I believe in solving my own problems rather than whining because someone else has not solved them for me. But it did seem like something that should be reported.

Navigation Menu

Repository navigation

AddressHeader problem when FriendlyName is improperly quoted and contains comma #143938

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

AddressHeader problem when FriendlyName is improperly quoted and contains comma #143938

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Activity

picnixz commented on Jan 17, 2026

fostervbv commented on Jan 17, 2026

picnixz commented on Jan 18, 2026

picnixz commented on Jan 18, 2026

fostervbv commented on Jan 19, 2026

picnixz commented on Jan 19, 2026

fostervbv commented on Jan 20, 2026

picnixz commented on Jan 20, 2026

fostervbv commented on Jan 20, 2026

picnixz commented on Jan 20, 2026

bitdancer commented on Jan 20, 2026

fostervbv commented on Jan 21, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions