Repository navigation
codecs should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just UnicodeError #70068
Closed as duplicate
Description
| BPO | 25880 |
|---|---|
| Nosy | @malemburg, @loewis, @vstinner, @ezio-melotti, @bitdancer, @serhiy-storchaka, @spaceone |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2015-12-16.08:03:52.505>
labels = ['expert-unicode', '3.11']
title = 'codecs should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just UnicodeError'
updated_at = <Date 2021-11-27.00:00:23.381>
user = 'https://github.com/spaceone'bugs.python.org fields:
activity = <Date 2021-11-27.00:00:23.381>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2015-12-16.08:03:52.505>
creator = 'spaceone'
dependencies = []
files = []
hgrepos = []
issue_num = 25880
keywords = []
message_count = 12.0
messages = ['256514', '256519', '256520', '256521', '256594', '256605', '256606', '256608', '256609', '256700', '256701', '256704']
nosy_count = 8.0
nosy_names = ['lemburg', 'loewis', 'vstinner', 'ezio.melotti', 'r.david.murray', 'SilentGhost', 'serhiy.storchaka', 'spaceone']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'resolved'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue25880'
versions = ['Python 3.11']
Activity
spaceone commentedon Dec 16, 2015
→ I was expecting that this raises either not at all or UnicodeEncodeError.
The error message is also messed up which wasn't the case in python 2.7. It could be cleaned up.
bitdancer commentedon Dec 16, 2015
The error message is accurate. That string has empty label segments in it, which RFC 5890 defines as an error on encoding. There is no such error defined for decoding, so that doesn't raise an error.
I don't see anything wrong with the error message, it includes the same one as raised in python2. Perhaps you are confused by the error chaining introduced in Python3? The second part of the traceback is coming from the encoding machinery, while the first part lets you know where in the encoder the error was raised. In this case having both doesn't provide much additional information, but if one was debugging a codec or the error were coming from inside an application, it would.
spaceone commentedon Dec 16, 2015
But why is the error UnicodeError instead of UnicodeEncodeError?
bitdancer commentedon Dec 16, 2015
Why does it matter? If you want to suggest changing it, you could propose a patch. Maybe in reading the code you'll find out why it is the way it is now. I haven't looked at that code in a while myself, so I don't remember if there is a reason or not :)
spaceone commentedon Dec 17, 2015
It makes error handling really hard.
Here is a patch:
https://github.com/python/cpython/compare/master...spaceone:idna?expand=1
bitdancer commentedon Dec 17, 2015
Can you explain why it makes error handling hard? I'm still not seeing the use case. I've always viewed UnicodeEncodeError vs UnicodeDecodeError as "extra" information for the consumer of the error message, not something that matters in code (I just catch UnicodeError).
I'm not objecting to the change, but it might be nice to know why Martin chose plain UnicodeError, if he's got the time to answer.
spaceone commentedon Dec 17, 2015
Because i need to do everywhere where I use this:
try:
user_input.encode(encoding)
except UnicodeDecodeError:
raise
except (UnicodeError, UnicodeEncodeError):
do_my_error_handling()
instead of
try:
user_input.encode(encoding)
except UnicodeEncodeError:
do_my_error_handling()
SilentGhost commentedon Dec 17, 2015
I think what David was trying to say is that you could do
try:
user_input.encode(encoding)
except UnicodeError:
do_my_error_handling()
since UnicodeError is a super class of UnicodeDecodeError and UnicodeEncodeError.
spaceone commentedon Dec 17, 2015
I know that UnicodeEncodeError is a subclass of UnicodeError. The problem here is that UnicodeError would also catch UnicodeDecodeError.
This is especially disturbing if you catch errors of a whole function.
If you e.g. use python2.7 you might want to catch only UnicodeEncodeError if you encode something and don't want to catch UnicodeDecodeError.
(Read that code carefully!!! It's not something which should ever be done but might happen in the world)
Especially if you are writing python2+3 compatible applications.
serhiy-storchaka commentedon Dec 18, 2015
The bare UnicodeError is raised also by following codecs: utf_16, utf_32, punycode, undefined, and East-Asian multibyte codecs, and by undocumented an unused function urllib.urlparse.to_bytes().
I think it would be nice to be more specific if possible.
bitdancer commentedon Dec 18, 2015
I wonder if we originally only had UnicodeError and it got split later but these codecs were never updated. The codecs date back to the start of unicode support in python2, I think.
Adding MAL, he's likely to have an opinion on this ;)
Oh, right. The more likely possibility is that there was (in python2) no way to know if the operation was (from the user's POV) encoding or decoding when the codec was called. In python3 we do know, when the codec is called via encode/decode, but the codecs are still generic in principle. So yeah, we need MAL's opinion. (Or, I could be completely confused, since I always found encode/decode confusing in python2 :)
malemburg commentedon Dec 18, 2015
On 18.12.2015 20:25, R. David Murray wrote:
UnicodeDecodeError and UnicodeEncodeError were added in Python 2.3
as part of the more flexible error handlers.
There's a clear direction with codecs:
Take e.g. the hex codec. It encodes data into hex format and
decodes from hex format back into the original data.
The IDNA codecs transforms Unicode domains into the IDNA format
(.encode()) and back to Unicode again (.decode()).
It was added in Python 2.3 as well, so I guess it was just
an overlap/oversight that it was not adapted to the new error
classes.
[-]u'..'.encode('idna') → UnicodeError: label empty or too long[/-][+]codecs should raise specific UnicodeDecodeError/UnicodeEncodeError rather than just UnicodeError[/+]StanFromIreland commentedon Jul 10, 2025
A duplicate was created: #85287, and all instances were fixed in 649857a