Repository navigation
Incorrect handling of start and end values in codecs error handlers #126004
Description
Crash report
What happened?
./python -c "import codecs; codecs.xmlcharrefreplace_errors(UnicodeEncodeError('bad', '', 0, 1, 'reason'))"
python: ./Include/cpython/unicodeobject.h:339: PyUnicode_READ_CHAR: Assertion `index >= 0' failed.
Aborted (core dumped)./python -c "import codecs; codecs.backslashreplace_errors(UnicodeDecodeError('utf-8', b'00000', 9, 2, 'reason'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
SystemError: Negative size passed to PyUnicode_New./python -c "import codecs; codecs.replace_errors(UnicodeTranslateError('000', 1, -7, 'reason'))"
python: Python/codecs.c:743: PyCodec_ReplaceErrors: Assertion `PyUnicode_KIND(res) == PyUnicode_2BYTE_KIND' failed.
Aborted (core dumped)See #123378 for the root cause. Since we are still wondering how to fix the getters and setters, I suggest we first fix the crash by adding the checks inside at the handler's level (for now). I'm not sure if the handler itself is handling corner cases correctly as well.
Activity
[-]Incorrect handling of `start` and `end` values in `codecs.xmlcharrefreplace_errors` handler[/-][+]Incorrect handling of `start` and `end` values in `codecs` error handlers[/+]picnixz commentedon Nov 29, 2024
FTR: I prepared three branches that fix (and cleanup) the error handlers.
Those branches include #123380 and thus I'll wait for the latter to be merged since otherwise there will be a crash even before the actual "bad" code happens.
The fixes are actually quite straightforward: there is actually no check on the consistency of
startandendafter they've been retrieved via*GetStartand*GetEnd, and this leads to OOB issues or other kind of issues. So, even if we fix the getters, we still need to handle the consistency of the bounds.cc @encukou
picnixz commentedon Dec 6, 2024
Since #123378 is only 3.14+ (not backported), the fixes for those handlers will also be 3.14+ only.
gh-126004: Fix positions handling in `codecs.xmlcharrefreplace_errors` (
gh-126004: Fix positions handling in `codecs.replace_errors` (#127674)
gh-126004: Fix positions handling in `codecs.backslashreplace_errors` (…
gh-126004: Remove redundant safeguards for codecs handlers tests (#12…