A lot of leaks in the test suite on the JIT build

Bug report

Bug description:

List of failed tests(I'm used ./python.exe -m test -R 3:3 to run the test suite):

319 tests failed:
    test.test_asyncio.test_base_events
    test.test_asyncio.test_buffered_proto
    test.test_asyncio.test_eager_task_factory
    test.test_asyncio.test_events test.test_asyncio.test_futures
    test.test_asyncio.test_futures2 test.test_asyncio.test_locks
    test.test_asyncio.test_pep492
    test.test_asyncio.test_proactor_events
    test.test_asyncio.test_queues test.test_asyncio.test_runners
    test.test_asyncio.test_selector_events
    test.test_asyncio.test_sendfile test.test_asyncio.test_server
    test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
    test.test_asyncio.test_sslproto test.test_asyncio.test_staggered
    test.test_asyncio.test_streams test.test_asyncio.test_subprocess
    test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
    test.test_asyncio.test_threads test.test_asyncio.test_timeouts
    test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait
    test.test_inspect.test_inspect
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads
    test.test_multiprocessing_spawn.test_manager
    test.test_multiprocessing_spawn.test_misc
    test.test_multiprocessing_spawn.test_processes
    test.test_multiprocessing_spawn.test_threads
    test.test_pydoc.test_pydoc test___all__ test__interpchannels
    test__interpreters test__opcode test__osx_support test_abc
    test_argparse test_array test_asdl_parser test_ast test_asyncgen
    test_audit test_base64 test_baseexception test_bdb test_bigmem
    test_binascii test_binop test_bisect test_buffer test_bufio
    test_builtin test_bytes test_bz2 test_c_locale_coercion
    test_calendar test_call test_capi test_class test_clinic
    test_cmath test_cmd test_cmd_line test_cmd_line_script test_code
    test_code_module test_codeccallbacks test_codecencodings_cn
    test_codecencodings_hk test_codecencodings_iso2022
    test_codecencodings_jp test_codecencodings_kr
    test_codecencodings_tw test_codecs test_codeop test_collections
    test_colorsys test_compare test_compile test_compileall
    test_compiler_codegen test_complex test_configparser test_context
    test_contextlib test_contextlib_async test_coroutines test_csv
    test_ctypes test_dataclasses test_datetime test_dbm test_dbm_dumb
    test_dbm_sqlite3 test_decimal test_deque test_descr test_descrtut
    test_dict test_difflib test_dis test_doctest test_docxmlrpc
    test_email test_embed test_ensurepip test_enum test_errno
    test_except_star test_exception_group test_exceptions test_extcall
    test_faulthandler test_filecmp test_float test_fnmatch test_format
    test_fractions test_frame test_free_threading test_fstring
    test_ftplib test_functools test_gc test_generated_cases
    test_generators test_genericalias test_genexps test_getopt
    test_getpass test_getpath test_gettext test_glob test_grammar
    test_graphlib test_grp test_gzip test_hashlib test_heapq test_hmac
    test_html test_htmlparser test_http_cookiejar test_http_cookies
    test_httplib test_httpservers test_idle test_imaplib test_import
    test_int test_interpreters test_io test_ipaddress test_iterlen
    test_itertools test_json test_list test_listcomps test_locale
    test_logging test_long test_lzma test_mailbox test_marshal
    test_math test_memoryio test_metaclass test_mimetypes test_minidom
    test_mmap test_module test_modulefinder test_monitoring
    test_multiprocessing_main_handling test_named_expressions
    test_netrc test_ntpath test_numeric_tower test_opcache
    test_operator test_optparse test_ordered_dict test_os test_pathlib
    test_patma test_pdb test_peepholer test_pep646_syntax test_pkg
    test_pkgutil test_platform test_plistlib test_poll test_poplib
    test_positional_only_arg test_posix test_posixpath test_pow
    test_pprint test_property test_pstats test_pty test_pulldom
    test_pwd test_py_compile test_pyclbr test_pyexpat test_queue
    test_quopri test_random test_range test_re test_regrtest
    test_reprlib test_richcmp test_rlcompleter test_robotparser
    test_runpy test_sax test_script_helper test_selectors test_set
    test_setcomps test_shelve test_shlex test_shutil test_signal
    test_site test_slice test_smtplib test_socket test_sort
    test_sqlite3 test_ssl test_stable_abi_ctypes test_stat
    test_statistics test_str test_strftime test_string
    test_string_literals test_strptime test_strtod test_struct
    test_structseq test_subprocess test_super test_support
    test_symtable test_syntax test_sys test_sys_settrace test_tabnanny
    test_tarfile test_tcl test_tempfile test_textwrap test_thread
    test_threadedtempfile test_threading test_threading_local
    test_time test_tokenize test_tools test_traceback test_tracemalloc
    test_tuple test_type_aliases test_type_annotations test_type_cache
    test_type_comments test_type_params test_types test_typing
    test_ucn test_unicode_file_functions test_unicodedata
    test_unittest test_univnewlines test_unpack test_unpack_ex
    test_unparse test_urllib test_urllib2 test_urllib2_localnet
    test_urlparse test_userdict test_userlist test_userstring
    test_uuid test_venv test_warnings test_wave test_weakref
    test_weakset test_webbrowser test_wsgiref test_xml_etree
    test_xml_etree_c test_xmlrpc test_yield_from test_zipapp
    test_zipfile test_zipimport test_zipimport_support test_zlib
    test_zoneinfo

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

gh-120501: Fix reference leak in JIT build #120649

Oh, I'm sorry, this also happens on non-jit build (just a --with-pydebug option).

Uhm, sorry again 😄 It's actually related to JIT build. I cannot reproduce it in pure debug build anymore. But on the JIT build, I can.

I'm not able to recreate this on Linux build on main - maybe Mac specific? (Well, a few skipped tests from missing extras, but not 300+ failures):

./make distclean && ./configure --enable-experimental-jit --with-pydebug && make -j8
./python -VV # Python 3.14.0a0 (heads/main:6af190f8d0, Jun 14 2024, 08:56:48) [GCC 11.4.0]
./python -m test -R 3:3 -j8

== Tests result: SUCCESS ==

12 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_android test_devpoll
    test_kqueue test_launcher test_msvcrt test_startfile test_winapi
    test_winconsoleio test_winreg test_wmi

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

455 tests OK.

I'm not able to recreate this on Linux build on main (well, a few skipped tests from missing extras, but not 300+ failures):

./make distclean && ./configure --enable-experimental-jit --with-pydebug && make -j8
./python -VV # Python 3.14.0a0 (heads/main:6af190f8d0, Jun 14 2024, 08:56:48) [GCC 11.4.0]
./python -m test -R 3:3 -j8

== Tests result: SUCCESS ==

12 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test_android test_devpoll
    test_kqueue test_launcher test_msvcrt test_startfile test_winapi
    test_winconsoleio test_winreg test_wmi

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

455 tests OK.

Interesting. I can reproduce this on both of my WSL and macOS setups.
I'm using the command below to check it for leaks.

git clean -fdx && ./configure --with-pydebug --enable-experimental-jit && make -j && ./python.exe -m test -R 3:3 test___all__

Additionally, I can reproduce it on my Windows setup, using this command:

 pcbuild/build.bat -c Debug --experimental-jit ; ./python -m test -R 3:3 test___all__
 ...many lines of build logs
 Running Debug|x64 interpreter...
Using random seed: 2230268839
0:00:00 Run 1 test sequentially in a single process
0:00:00 [1/1] test___all__
beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more)
123:456
XXX XXX
test___all__ leaked [11, 11, 11] references, sum=33
test___all__ leaked [11, 11, 11] memory blocks, sum=33
test___all__ failed (reference leak)

== Tests result: FAILURE ==

1 test failed:
    test___all__

Total duration: 19.5 sec
Total tests: run=1
Total test files: run=1/1 failed=1
Result: FAILURE

Agreed, I'm running the full windows tests on a native build now and seeing tons of leaking tests. I'll be curious if the failure list is the same as on your original Mac build or different.

Just ran the Linux tests again sequentially (i.e. without -j8) to see if that made a difference. Again all tests pass on Linux (or skip).

@markshannon

Bisected to f6fab21
cc @markshannon

On Windows build, 314 tests failed:

./PCBuild/amd64/python_d.exe -VV #
./PCBuild/amg64/python_d.exe -m test -R 3:3 -j8

Total test files: run=468/478 failed=314 skipped=41 resource_denied=10

I would like to test on my machine but it incorrectly assumes that I need clang-18 instead of gcc. I'm using OpenSUSE 15.5 and my clang is 15.0.7. How can I actually force to use the gcc for JIT? (and is there any support for it actually?)

I would like to test on my machine but it incorrectly assumes that I need clang-18 instead of gcc. I'm using OpenSUSE 15.5 and my clang is 15.0.7. How can I actually force to use the gcc for JIT? (and is there any support for it actually?)

I think there's no support for gcc since our JIT is based on LLVM

Oh, I thought that the JIT support explained in PEP 744 was for all Tier 1 platforms (especially "x86_64-unknown-linux-gnu/gcc") but I still need LLVM and Clang. Ok, I'll install local versions for that.

@picnixz

@picnixz Actually we have a README for JIT, which is located at Tools/jit/README.md 🙂

(Yes, I've seen the README after actually. I first looked at the PEP to see whether my platform was supported or not and was surprised)

Is this rather new?

 ======================================================================
FAIL: test_strftime_y2k (test.datetimetester.TestSubclassDateTime_Fast.test_strftime_y2k) (year=999, specifier='F')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/datetimetester.py", line 1728, in test_strftime_y2k
    self.assertEqual(d.strftime(f"%{specifier}"), expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '999-01-01' != '0999-01-01'
- 999-01-01
+ 0999-01-01
? +


======================================================================
FAIL: test_strftime_y2k (test.datetimetester.TestSubclassDateTime_Fast.test_strftime_y2k) (year=999, specifier='C')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/datetimetester.py", line 1728, in test_strftime_y2k
    self.assertEqual(d.strftime(f"%{specifier}"), expected)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '9' != '09'
- 9
+ 09
? +


----------------------------------------------------------------------
Ran 8 tests in 0.567s

FAILED (failures=48)
test test_datetime failed
1 test failed again:
    test_datetime

== Tests result: FAILURE then FAILURE ==

38 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test_android test_bz2 test_ctypes test_dbm_gnu test_dbm_ndbm
    test_dbm_sqlite3 test_devpoll test_free_threading test_gzip
    test_idle test_ioctl test_kqueue test_launcher test_lzma
    test_msvcrt test_readline test_smtpnet test_sqlite3 test_ssl
    test_stable_abi_ctypes test_startfile test_tcl test_tkinter
    test_ttk test_ttk_textonly test_turtle test_winapi
    test_winconsoleio test_winreg test_wmi test_zlib

8 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_socketserver
    test_urllib2net test_urllibnet test_winsound test_zipfile64

5 tests run no tests:
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test_external_inspection

1 re-run test:
    test_datetime

1 test failed:
    test_datetime

Log: https://github.com/sobolevn/cpython/actions/runs/10603373267/job/29387830804

I think it was #120713.

This is actually a tier-2 leaks, not a JIT leaks, so I am removing the topic-JIT label here.
(You can reproduce reference/memory block leaks by using the --enable-experimental-jit=interpreter configure option)

Navigation Menu

Repository navigation

A lot of leaks in the test suite on the `JIT` build #120501

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

A lot of leaks in the test suite on the JIT build #120501

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

Eclips4 commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

JeffersGlass commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

JeffersGlass commented on Jun 14, 2024

JeffersGlass commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

JeffersGlass commented on Jun 14, 2024

picnixz commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

picnixz commented on Jun 14, 2024

Eclips4 commented on Jun 14, 2024

picnixz commented on Jun 14, 2024

sobolevn commented on Aug 28, 2024

picnixz commented on Aug 29, 2024

Eclips4 commented on Jan 9, 2025

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

A lot of leaks in the test suite on the `JIT` build #120501