Skip to content

Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError #85287

Closed
Closed
Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError#85287
Labels
3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeeasystdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@pitrou
pitrou
opened on Jun 25, 2020
BPO 41115
Nosy @malemburg, @doerwalter, @pitrou, @benjaminp, @ezio-melotti, @serhiy-storchaka, @srinivasreddy, @eamanu, @utkarsh261
PRs
  • gh-85287: Convert UnicodeError to UnicodeEncodeError| UnicodeDecodeError  #21165
  • bpo-41115: Modified src to raise rather Unicode{Encode, Decode}Error rather than plain UnicodeError #21170
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-06-25.12:49:30.420>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.7', 'library']
title = 'Codecs should raise precise UnicodeDecodeError or UnicodeEncodeError'
updated_at = <Date 2020-06-26.16:59:17.544>
user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2020-06-26.16:59:17.544>
actor = 'utk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-25.12:49:30.420>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 41115
keywords = ['patch', 'easy']
message_count = 6.0
messages = ['372367', '372368', '372369', '372373', '372431', '372433']
nosy_count = 9.0
nosy_names = ['lemburg', 'doerwalter', 'pitrou', 'benjamin.peterson', 'ezio.melotti', 'serhiy.storchaka', 'thatiparthy', 'eamanu', 'utk']
pr_nums = ['21165', '21170']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41115'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

    Linked PRs

    Activity

    1. pitrou commented on Jun 25, 2020

      @pitrou
      pitrou
      on Jun 25, 2020
      MemberAuthor

      A number of codecs raise bare UnicodeError, rather than Unicode{Decode,Encode}Error. Example:

      File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/encodings/utf_16.py", line 67, in _buffer_decode
      raise UnicodeError("UTF-16 stream does not start with BOM")

      A more complete list can be found here:
      https://gist.github.com/pitrou/60594b28d8e47edcdb97d9b15d5f9866

    2. pitrou
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      easy
      type-bugAn unexpected behavior, bug, or error
      on Jun 25, 2020

    3. srinivasreddy commented on Jun 25, 2020

      @srinivasreddy
      srinivasreddymannequin
      on Jun 25, 2020
      Mannequin

      This looks like an easy task. Shall I create a PR?

    4. eamanu commented on Jun 25, 2020

      @eamanu
      eamanumannequin
      on Jun 25, 2020
      Mannequin

      Hi,

      IMO this can be mark as an easy issue.

      @thatiparthy please, go ahead

    5. 7 remaining items

    6. mblahay commented on Apr 26, 2023

      @mblahay
      mblahay
      on Apr 26, 2023
      Contributor

      @SahillMulani, I am looking to perhaps work on this issue, but will avoid it if you are. Please let me know what your status is.

    7. mblahay commented on Apr 26, 2023

      @mblahay
      mblahay
      on Apr 26, 2023
      Contributor

      @pitrou Regarding the start and end values that are used to communicate where the bad values are, what should be done if the absolute location cannot be determined? There are portions of code, such as with punycode, where the encoded byes is divided up for processing and the only positional data available at the time of exception is the relative position within the given segment. Should 0,0 be used in a situation like this?

    8. mblahay commented on Apr 26, 2023

      @mblahay
      mblahay
      on Apr 26, 2023
      Contributor

      @methane I was told you may be an interested party that could answer questions about unicode exception handling. Regarding my question above, what are you expectations with regards to the reported start and end positions?

    9. methane commented on May 25, 2023

      @methane
      methane
      on May 25, 2023
      Member

      It seems this issue is not fixed yet.

      c

      > rg -t c -w PyExc_UnicodeError

Modules/cjkcodecs/multibytecodec.c
831:            PyErr_SetString(PyExc_UnicodeError,
863:            PyErr_SetString(PyExc_UnicodeError, "pending buffer overflow");
944:            PyErr_SetString(PyExc_UnicodeError, "pending buffer too large");
984:        PyErr_SetString(PyExc_UnicodeError, "pending buffer too large");
1271:        PyErr_SetString(PyExc_UnicodeError, "pending buffer too large");

      python:

      ~/w/p/cpython (main)> rg -t py 'raise UnicodeError'
Lib/test/support/os_helper.py
104:            raise UnicodeError

Lib/test/test_array.py
1042:                raise UnicodeError
1047:            raise UnicodeError

Lib/urllib/parse.py
1047:            raise UnicodeError("URL " + repr(url) +

Lib/encodings/punycode.py
137:                raise UnicodeError("incomplete punicode string")
145:            raise UnicodeError("Invalid extended code point '%s'"
174:                raise UnicodeError("Invalid character U+%x" % char)
206:            raise UnicodeError("Unsupported error handling "+errors)
217:            raise UnicodeError("Unsupported error handling "+self.errors)

Lib/encodings/utf_16.py
67:                raise UnicodeError("UTF-16 stream does not start with BOM")
141:            raise UnicodeError("UTF-16 stream does not start with BOM")

Lib/encodings/utf_32.py
62:                raise UnicodeError("UTF-32 stream does not start with BOM")
136:            raise UnicodeError("UTF-32 stream does not start with BOM")

Lib/encodings/undefined.py
19:        raise UnicodeError("undefined encoding")
22:        raise UnicodeError("undefined encoding")
26:        raise UnicodeError("undefined encoding")
30:        raise UnicodeError("undefined encoding")

Lib/encodings/idna.py
38:            raise UnicodeError("Invalid character %r" % c)
50:            raise UnicodeError("Violation of BIDI requirement 2")
56:            raise UnicodeError("Violation of BIDI requirement 3")
71:        raise UnicodeError("label empty or too long")
86:        raise UnicodeError("label empty or too long")
90:        raise UnicodeError("Label starts with ACE prefix")
101:    raise UnicodeError("label empty or too long")
113:        raise UnicodeError("label way too long")
130:            raise UnicodeError("Invalid character in IDN label")
147:        raise UnicodeError("IDNA does not round-trip", label, label2)
159:            raise UnicodeError("unsupported error handling "+errors)
173:                    raise UnicodeError("label empty or too long")
175:                raise UnicodeError("label too long")
195:            raise UnicodeError("Unsupported error handling "+errors)
230:            raise UnicodeError("unsupported error handling "+errors)
264:            raise UnicodeError("Unsupported error handling "+errors)
    10. jjsloboda
      added a commit that references this issue on Jan 3, 2024

      fix issue pythongh-85287

      7339989

    11. jjsloboda commented on Jan 3, 2024

      @jjsloboda
      jjsloboda
      on Jan 3, 2024
      Contributor

      Decided to give this a shot as I see it's still unresolved on the main branch. Made some decisions about how to do certain things just so I could ship something, but very much open to discussion on other approaches.

      The biggest question is whether it's worth having the codec functions save their original arguments just so they can be shown if an exception happens. It might not even be extra memory overhead since the caller is probably still holding a reference to the argument object in most cases.

      Also this is my first time looking at the CPython codebase and my first time using the C API so please let me know if I'm making any beginner mistakes.

    12. methane
      added a commit that references this issue on Mar 17, 2024

      gh-85287: Change codecs to raise precise UnicodeEncodeError and Unico…

      649857a
    13. vstinner
      added a commit that references this issue on Mar 20, 2024

      pythongh-85287: Change codecs to raise precise UnicodeEncodeError and…

      1ccbc3d
    14. adorilson
      added a commit that references this issue on Mar 25, 2024

      pythongh-85287: Change codecs to raise precise UnicodeEncodeError and…

      8d804eb
    15. diegorusso
      added a commit that references this issue on Apr 17, 2024

      pythongh-85287: Change codecs to raise precise UnicodeEncodeError and…

      7c53561

    16. nineteendo commented on May 19, 2024

      @nineteendo
      nineteendo
      on May 19, 2024
      Contributor

      Isn't this fixed?

    17. methane
      closed this as completedon May 19, 2024
    18. github-project-automation
      moved this to Done in Codecs and encodings issueson May 19, 2024
    19. StanFromIreland
      marked codecs should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just UnicodeError #70068 as a duplicate of this issue on Jul 10, 2025
    20. StanFromIreland
      marked Unexpected UnicodeError instead of UnicodeDecodeError within codec.readline() only for 'utf-16' encoding #112812 as a duplicate of this issue on Jul 12, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifeeasystdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions