Skip to content

email: multiline encoded headers #91957

Open

Description

@mdierksen

Bug report

There seems to be a problem with multiline encoded headers, for example subject headers.

from email import message_from_string, policy
import base64

e = """From: <test@test.com>
To: <test@test.com>
Subject: =?UTF-8?B?0JfQsNGB0YLRgNCw0YXQvtCy0LrQsCAi0JDQu9C40LDQvdGGINCX0LTRgNCw0LLQ?=
 =?UTF-8?B?tSIg4oCTINCY0L3QtNC40LLQuNC00YPQsNC70LXQvSDQvdC+0LzQtdGAINC90LAg?=
 =?UTF-8?B?0YPQstC10LTQvtC80LvQtdC90LjQtQ==?=

test"""

e = message_from_string(e, policy=policy.default)

s1 = e["Subject"]

try:
    e.as_bytes()
except Exception:
    print("error")
else:
    print("ok")

s = (base64.b64decode("0JfQsNGB0YLRgNCw0YXQvtCy0LrQsCAi0JDQu9C40LDQvdGGINCX0LTRgNCw0LLQ") + base64.b64decode("tSIg4oCTINCY0L3QtNC40LLQuNC00YPQsNC70LXQvSDQvdC+0LzQtdGAINC90LAg")
    + base64.b64decode("0YPQstC10LTQvtC80LvQtdC90LjQtQ==")).decode("utf-8")

del e["Subject"]
e["Subject"] = s

s2 = e["Subject"]

try:
    e.as_bytes()
except Exception:
    print("error")
else:
    print("ok")

print(s1 == s2)

e = """From: <test@test.com>
To: <test@test.com>
Subject: =?UTF-8?B?0JfQsNGB0YLRgNCw0YXQvtCy0LrQsCAi0JDQu9C40LDQvdGGINCX0LTRgNCw0LLQtSIg4oCTINCY0L3QtNC40LLQuNC00YPQsNC70LXQvSDQvdC+0LzQtdGAINC90LAg0YPQstC10LTQvtC80LvQtdC90LjQtQ==?=

test"""

e = message_from_string(e, policy=policy.default)

s3 = e["Subject"]

try:
    e.as_bytes()
except Exception:
    print("error")
else:
    print("ok")

print(s2 == s3)

Your environment

Tested on Python 3.10.4 and 3.11.0a7 on Arch Linux

Activity

  1. wbolster commented on May 9, 2022

    @wbolster
    Contributor

    it would help if you described what you believe the problem is, @mdierksen (without forcing others to reverse engineer your example code to try to understand)

  2. mdierksen commented on May 10, 2022

    @mdierksen
    Author

    The first e.as_bytes() causes this exception:

    Traceback (most recent call last):
      File "/home/dierksen/test4.py", line 16, in <module>
        e.as_bytes()
        ^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/email/message.py", line 178, in as_bytes
        g.flatten(self, unixfrom=unixfrom)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/email/generator.py", line 116, in flatten
        self._write(msg)
        ^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/email/generator.py", line 199, in _write                                                                                                                                                              
        self._write_headers(msg)                                                                                                                                                                                                            
        ^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                            
      File "/usr/local/lib/python3.11/email/generator.py", line 422, in _write_headers                                                                                                                                                      
        self._fp.write(self.policy.fold_binary(h, v))                                                                                                                                                                                       
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                        
      File "/usr/local/lib/python3.11/email/policy.py", line 200, in fold_binary                                                                                                                                                            
        folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')                                                                                                                                                               
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                               
      File "/usr/local/lib/python3.11/email/policy.py", line 214, in _fold                                                                                                                                                                  
        return self.header_factory(name, ''.join(lines)).fold(policy=self)                                                                                                                                                                  
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                  
      File "/usr/local/lib/python3.11/email/headerregistry.py", line 253, in fold                                                                                                                                                           
        return header.fold(policy=policy)                                                                                                                                                                                                   
               ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                   
      File "/usr/local/lib/python3.11/email/_header_value_parser.py", line 156, in fold                                                                                                                                                     
        return _refold_parse_tree(self, policy=policy)                                                                                                                                                                                      
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                      
      File "/usr/local/lib/python3.11/email/_header_value_parser.py", line 2825, in _refold_parse_tree                                                                                                                                      
        last_ew = _fold_as_ew(tstr, lines, maxlen, last_ew,                                                                                                                                                                                 
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                 
      File "/usr/local/lib/python3.11/email/_header_value_parser.py", line 2913, in _fold_as_ew
        encoded_word = _ew.encode(to_encode_word, charset=encode_as)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.11/email/_encoded_words.py", line 222, in encode
        bstring = string.encode('ascii', 'surrogateescape')
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-10: ordinal not in range(128)
    

    The problem is with the subject header. Extracting the subject as a string works however.

    Manually decoding the subject header and replacing it in the email object fixes the problem. Extracting the subject as a string yields the same string as before.

    The problem does not occur for encoded headers in a single line, but only for multiline headers that are split up in a certain way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytopic-emailtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions