Skip to content

python3.12 introduces numerous memory leaks (as reported by ASAN) #113190

Closed
Closed
python3.12 introduces numerous memory leaks (as reported by ASAN)#113190
Labels
topic-subinterpreterstype-bugAn unexpected behavior, bug, or error

Description

@Mekk
Mekk
opened on Dec 15, 2023

Bug report

Bug description:

Let example illustrate:

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.10 -c 'print("Hello")'
Hello

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
  PYTHONMALLOC=malloc \
  /usr/bin/python3.11 -c 'print("Hello")'
Hello

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'print("Hello")'
Hello

=================================================================
==3317995==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 55877 byte(s) in 1063 object(s) allocated from:
    #0 0x7f5d145847ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 30732 byte(s) in 614 object(s) allocated from:
    #0 0x7f5d145847ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 86609 byte(s) leaked in 1677 allocation(s).

CPython versions tested on:

3.12.1

Operating systems tested on:

Linux

Linked PRs

Activity

  1. Mekk
    added
    type-bugAn unexpected behavior, bug, or error
    on Dec 15, 2023

  2. Mekk commented on Dec 15, 2023

    @Mekk
    Mekk
    on Dec 15, 2023
    Author

    I assume those are various „global” objects and to a degree the report is spurious, nevertheless using sanitizer while maintaining binary python extensions and code embedding python turned out to be invaluable for me many times (esp. considering new/borrow reference semantics is often poorly documented and related code error-prone).

    On py3.10 I had nice, clear state, on py3.12 the same code reports plenty of leaks.

    Example above is only a start, slightly more complicated code (clear and sane on 3.10) starts reporting much more leaks, mostly mentioning _PyUnicode_New, _PyUnicode_JoinArray , _PyUnicode_FromASCII (all rooted somewhere inside python3.12 binary)

    Tested on python 3.12.1 from deadsnakes.

  3. Mekk commented on Dec 15, 2023

    @Mekk
    Mekk
    on Dec 15, 2023
    Author

    Maybe one more example:

    $ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
  PYTHONMALLOC=malloc \
  /usr/bin/python3.10 -c 'import pytest'

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'import pytest'

=================================================================
==3331693==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 591154 byte(s) in 11373 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 111806 byte(s) in 2108 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 815 byte(s) in 16 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x549a29 in _PyUnicode_FromASCII (/usr/bin/python3.12+0x549a29) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 390 byte(s) in 6 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 104 byte(s) in 2 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x584ce0 in PyUnicode_Concat (/usr/bin/python3.12+0x584ce0) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 704269 byte(s) leaked in 13505 allocation(s).

  4. Mekk commented on Dec 15, 2023

    @Mekk
    Mekk
    on Dec 15, 2023
    Author

    Staying in stdlib:

    $ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.10 -c 'import re'

$ LD_PRELOAD=/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so \
   PYTHONMALLOC=malloc \
   /usr/bin/python3.12 -c 'import re'

=================================================================
==3333908==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 74271 byte(s) in 1472 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5573b9  (/usr/bin/python3.12+0x5573b9) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 57512 byte(s) in 1095 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x539ddf  (/usr/bin/python3.12+0x539ddf) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

Direct leak of 260 byte(s) in 4 object(s) allocated from:
    #0 0x7f34a15807ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

SUMMARY: AddressSanitizer: 132043 byte(s) leaked in 2571 allocation(s).

  5. neonene commented on Dec 15, 2023

    @neonene
    neonene
    on Dec 15, 2023
    Contributor

    If your cases come from the commit 67807cf (3.12.0 alpha 1), I'm waiting for the memory release to be implemented at the finalization phase, and #113055 may be partially related.

  6. Mekk commented on Dec 19, 2023

    @Mekk
    Mekk
    on Dec 19, 2023
    Author

    I can't comment about reasons, but the bug you mention claims that some leak existed, and 3.12 only increased it's scale in case multiple interpreters are in use ( #113055 (comment) ).

    In my case (a) there is single interpreter in use and (b) 3.10 and 3.11 don't report any leaks (neither in simple examples above, nor in more complicated real code).

  7. vstinner commented on Dec 20, 2023

    @vstinner
    vstinner
    on Dec 20, 2023
    Member

    Valgrind doesn't see any leak in the current 3.12 branch:

    $ PYTHONMALLOC=malloc valgrind --leak-check=full --show-leak-kinds=all --log-file=valgrind.log --num-callers=20 ./python -c 'print("Hello")'
Hello

    valgrind.log:

    (...)
==1662842== HEAP SUMMARY:
==1662842==     in use at exit: 0 bytes in 0 blocks
==1662842==   total heap usage: 31,703 allocs, 31,703 frees, 4,419,701 bytes allocated
==1662842== 
==1662842== All heap blocks were freed -- no leaks are possible
(...)

    Direct leak of 390 byte(s) in 6 object(s) allocated from:
    #0 0x7f47b09917ee in __interceptor_malloc (/usr/lib/llvm-14/lib/clang/14.0.0/lib/linux/libclang_rt.asan-x86_64.so+0xcd7ee) (BuildId: a6105a816e63299474c1078329a59ed80f244fbf)
    #1 0x5a0cfd in _PyUnicode_JoinArray (/usr/bin/python3.12+0x5a0cfd) (BuildId: aba53fb0246abeea0af6e1c463dcb32848eb6c27)

    I'm not sure why this traceback is truncated, but without a more complete traceback, I don't see how to debug this issue.

  8. vstinner commented on Dec 20, 2023

    @vstinner
    vstinner
    on Dec 20, 2023
    Member

    It's possible to build Python with clang -fsanitize=memory:

    ./configure CC=clang LD=clang --with-memory-sanitizer && make

    The problem is that all dependencies of Python, such as OpenSSL, should be built with the same option. Otherwise, clang can emit false alarms. Example in OpenSSL:

      Uninitialized value was created by a heap allocation
    #0 0x484596 in malloc (/home/vstinner/python/3.12/python+0x484596) (BuildId: 0f3caceddd1581f15aab325a45ae12048629d15a)
    #1 0x7f28271d064a in OBJ_txt2obj (/lib64/libcrypto.so.3+0x1d064a) (BuildId: 5d012dcc6f62f35dabb8a129d641e3df7b731f11)
    #2 0x7f2827790c1c in _ssl_txt2obj_impl /home/vstinner/python/3.12/./Modules/_ssl.c:5285:11
    #3 0x7f2827790c1c in _ssl_txt2obj /home/vstinner/python/3.12/./Modules/clinic/_ssl.c.h:1351:20
    #4 0x72f535 in cfunction_vectorcall_FASTCALL_KEYWORDS /home/vstinner/python/3.12/Objects/methodobject.c:438:24

    Right now on my Fedora 39, even with detect_leaks=0, I fail to build Python successfully :-(

    ASAN_OPTIONS="detect_leaks=0:allocator_may_return_null=1:handle_segv=0" make
  9. ericsnowcurrently
    added
    topic-subinterpreters
    on Feb 27, 2024
  10. github-project-automation
    added this to Subinterpreterson Feb 27, 2024
  11. github-project-automation
    moved this to Todo in Subinterpreterson Feb 27, 2024

  12. WillAyd commented on Aug 2, 2024

    @WillAyd
    WillAyd
    on Aug 2, 2024
    Contributor

    I'm not sure why this traceback is truncated, but without a more complete traceback, I don't see how to debug this issue.

    In case it is helpful at all I can replicate the issue with a slightly more complete traceback from 3.12.4:

    Direct leak of 111239 byte(s) in 2183 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90dc618 in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90dc618 in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90dc618 in _PyUnicode_FromUCS1 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:2036
    #4 0x5f50f90dc618 in _PyUnicode_FromUCS1 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:2022

Direct leak of 58060 byte(s) in 1111 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90c419b in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90c419b in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90c419b in unicode_decode_utf8 /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:4693

Direct leak of 459 byte(s) in 7 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f911318a in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f911318a in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f911318a in _PyUnicode_JoinArray /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:9661

Direct leak of 50 byte(s) in 1 object(s) allocated from:
    #0 0x73eaeb6b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x5f50f90d1da1 in PyObject_Malloc /usr/local/src/conda/python-3.12.4/Objects/obmalloc.c:801
    #2 0x5f50f90d1da1 in PyUnicode_New /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1251
    #3 0x5f50f90d1da1 in _PyUnicode_FromASCII /usr/local/src/conda/python-3.12.4/Objects/unicodeobject.c:1998
  13. encukou
    added a commit that references this issue on Aug 15, 2024

    gh-113190: Reenable non-debug interned string cleanup (GH-113601)

    3203a74
  14. blhsing
    added a commit that references this issue on Aug 22, 2024

    pythongh-113190: Reenable non-debug interned string cleanup (pythonGH…

    b32350a
  15. hugovk
    added 2 commits that reference this issue on Feb 26, 2025

    [3.13] pythongh-113190: Reenable non-debug interned string cleanup (p…

    0355566

    [3.12] pythongh-113190: Reenable non-debug interned string cleanup (p…

    dccc2ed

  16. hugovk commented on Feb 26, 2025

    @hugovk
    hugovk
    on Feb 26, 2025
    Member

    I think we can close this? #113601 has been merged in main and we've decided not to backport: #113601 (comment).

  17. hugovk
    closed this as completedon Feb 26, 2025
  18. github-project-automation
    moved this from Todo to Done in Subinterpreterson Feb 26, 2025
  19. jcfr
    added a commit that references this issue on Sep 4, 2025

    ci(tests): suppress CPython 3.12 interned-unicode leak (PyUnicode_New…

    8871ee9
  20. mrbean-bremen
    added a commit that references this issue on Sep 4, 2025

    ci(tests): suppress CPython 3.12 interned-unicode leak (PyUnicode_New…

    b4fc6ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic-subinterpreterstype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions