Skip to content

imaplib: must not replace LF or CR by CRLF in literals #49680

Closed
Closed
imaplib: must not replace LF or CR by CRLF in literals#49680
Labels
3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeeasystdlibStandard Library Python modules in the Lib/ directorytopic-emailtype-bugAn unexpected behavior, bug, or error

Description

@memeplex
memeplex
mannequin
opened on Mar 6, 2009
BPO 5430
Nosy @warsaw, @mcepl, @bitdancer, @rduplain, @JulienPalard, @csabella
PRs
  • bpo-5430: Preservation LF, CR in IMAP4 append method. #10901
    		•
    Files
  • test_imaplib_cr_lf.diff: an attempt at testing IMAP4.append for CR, LF preservation
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2009-03-06.07:51:15.737>
labels = ['easy', 'type-bug', '3.8', 'expert-email', '3.10', 'library', '3.9']
title = 'imaplib: must not replace LF or CR by CRLF in literals'
updated_at = <Date 2020-11-07.01:52:35.851>
user = 'https://bugs.python.org/memeplex'

    bugs.python.org fields:

    activity = <Date 2020-11-07.01:52:35.851>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2009-03-06.07:51:15.737>
creator = 'memeplex'
dependencies = []
files = ['13789']
hgrepos = []
issue_num = 5430
keywords = ['patch', 'easy']
message_count = 8.0
messages = ['83241', '86572', '86602', '94578', '315575', '315577', '316170', '352292']
nosy_count = 8.0
nosy_names = ['barry', 'mcepl', 'r.david.murray', 'memeplex', 'ron.duplain', 'rajeshsr', 'mdk', 'cheryl.sabella']
pr_nums = ['10901']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5430'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    Linked PRs

    Activity

    1. memeplex commented on Mar 6, 2009

      @memeplex
      memeplexmannequin
      on Mar 6, 2009
      MannequinAuthor

      For example, after that "normalization", quoted printable encoded
      headers (as described at rfc 2047) longer than 76 characters are
      splitted in two different ill-formed headers because the soft LF line
      break becomes a "hard" CRLF one. This is clearly wrong.

      rfc 2060 specifically allows CR and LF inside literals:

      """
      A literal is a sequence of zero or more octets (including CR and LF),
      prefix-quoted with an octet count in the form of an open brace ("{"),
      the number of octets, close brace ("}"), and CRLF.
      """

    2. memeplex
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Mar 6, 2009
    3. memeplex
      changed the title [-]Must not replace LF or CR by CRLF in literals[/-] [+]imaplib: must not replace LF or CR by CRLF in literals[/+] on Mar 6, 2009
    4. devdanzin
      added
      easy
      on Apr 22, 2009

    5. rduplain commented on Apr 25, 2009

      @rduplain
      rduplainmannequin
      on Apr 25, 2009
      Mannequin

      It looks like the IMAP4.append method is responsible for the CRLF
      substitution (trunk/Lib/imaplib.py).

      # defined near top of module:
MapCRLF = re.compile(r'\r\n|\r|\n')

# in append method:
self.literal = MapCRLF.sub(CRLF, message)

      I'll work on a test for it this evening.

      -Ron

    6. rduplain commented on Apr 26, 2009

      @rduplain
      rduplainmannequin
      on Apr 26, 2009
      Mannequin

      Module imaplib has pretty sparse test code. There is only 1 test case,
      for imaplib.Time2Internaldate.
      trunk/Lib/test/test_imaplib.py

      The attached patch tests for LF, CR preservation with regard to the
      IMAP4.append method, but more testing is necessary to make sure LF, CR
      are preserved in IMAP literals throughout imaplib and that the
      respective fix doesn't break anything. Mock IMAP interaction may be the
      way to go for better test cases. Is anyone working on expanding imaplib
      test coverage?

      -Ron

    7. rajeshsr commented on Oct 27, 2009

      @rajeshsr
      rajeshsrmannequin
      on Oct 27, 2009
      Mannequin

      Am quite new here; just searching hard to contribute, would like to
      patch this, if I can go ahead.

      Don't we need to patch the original imaplib code also?

      just remove the line:
      self.literal = MapCRLF.sub(CRLF, message)
      and have:
      self.literal = message
      Or am I missing something completely?

    8. bitdancer
      added
      topic-email
      on Jan 23, 2013

    9. mcepl commented on Apr 21, 2018

      @mcepl
      mceplmannequin
      on Apr 21, 2018
      Mannequin

      Module imaplib has pretty sparse test code.

      On that note: would anybody be willing to express in form of (simplified) test case, what's the problem reported here? I am not sure, I follow.

    10. mcepl commented on Apr 21, 2018

      @mcepl
      mceplmannequin
      on Apr 21, 2018
      Mannequin

      Oh, this is 2.6 bug. This should be closed.

    11. csabella commented on May 4, 2018

      @csabella
      csabella
      on May 4, 2018
      Contributor

      I'm not sure if this is a 2.6/2.7 only issue because the code mentioned in msg86572 still exists, although with a change for bpo-21800 (RFC 6855/UTF-8) applied.

      For reference, bpo-25591 expanded the test coverage for the imaplib. However, I don't see any tests related to CR or LF in literals.

    12. JulienPalard commented on Sep 13, 2019

      @JulienPalard
      JulienPalard
      on Sep 13, 2019
      Member

      It looks like a revert of 47404ff
      Which is the fix for https://bugs.python.org/issue723962 which I'm currently reading.

    13. iritkatriel
      added
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      3.10 (EOL)end of life
      on Nov 7, 2020
    14. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    15. mcepl commented on Dec 6, 2024

      @mcepl
      mcepl
      on Dec 6, 2024
      Contributor

      It looks like a revert of 47404ff Which is the fix for https://bugs.python.org/issue723962 which I'm currently reading.

      Yes, because that commit was too hard hitting, and it made some messages non-conforming to RFCs (e.g., by corrupting headers).

    16. savagemechanic commented on May 18, 2026

      @savagemechanic
      savagemechanic
      on May 18, 2026
      Contributor

      I traced this on current main. IMAP4.append() still normalizes the APPEND message literal with MapCRLF.sub(CRLF, message) before assigning self.literal, so caller-provided bytes containing bare CR or LF are sent as different bytes.

      I also looked at the old PR #10901 and the concern about the older bpo-723962 change that introduced this canonicalization. The distinction that seems important is command framing vs literal payload: _command() already sends the literal length, waits for continuation, sends the literal bytes, and then sends the protocol CRLF after the literal. So it seems like append() should preserve the message literal exactly, while the protocol framing remains handled outside the literal.

      Would a narrowly scoped PR be welcome that:

      • preserves the APPEND message bytes exactly for the literal payload,
      • updates the existing APPEND tests that currently expect normalization,
      • adds a regression test with CR, LF, and CRLF in the message body to assert the fake server receives the exact payload?

      I have not opened a PR yet; I wanted to check that this framing/payload distinction is the right way to resolve the bpo-723962 concern.

    17. harjothkhara
      added 3 commits that reference this issue on Jun 16, 2026

      pythongh-49680: Add Harjoth Khara to ACKS

      4805d9f

      pythongh-49680: Don't rewrite CR/LF in imaplib APPEND literals

      0d83725

      pythongh-49680: Add Harjoth Khara to ACKS

      7e6bd8c
    18. serhiy-storchaka
      added 5 commits that reference this issue on Jul 2, 2026

      pythongh-49680: Test exact APPEND and email message round-trip

      ccf900a

      gh-49680: Add translate_line_endings parameter to imaplib.IMAP4.append (

      ddb6539

      [3.15] gh-49680: Test imaplib.IMAP4.append line-ending normalization (G…

      6759fd8

      [3.14] gh-49680: Test imaplib.IMAP4.append line-ending normalization (G…

      c98e263

      [3.13] gh-49680: Test imaplib.IMAP4.append line-ending normalization (G…

      f7431eb

    19. serhiy-storchaka commented on Jul 2, 2026

      @serhiy-storchaka
      serhiy-storchaka
      on Jul 2, 2026
      Member

      Fixed in #152775: append() now accepts translate_line_endings=False to send the message literal as exact octets, without rewriting bare CR or LF to CRLF. The useful test additions from @harjothkhara's original PR (#151557) were also backported to 3.13, 3.14 and 3.15.

    20. serhiy-storchaka
      closed this as completedon Jul 2, 2026
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeeasystdlibStandard Library Python modules in the Lib/ directorytopic-emailtype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions