Skip to content

Tests fail if --disable-test-modules is supplied #116303

Closed
Closed
Tests fail if --disable-test-modules is supplied#116303
Labels
testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

Description

@chaosflaws
chaosflaws
opened on Mar 4, 2024

Bug report

Bug description:

I compiled python 3.12 on a Debian system with --enable-optimizations and --disable-test-modules, and during the profile-guided optimization tests, multiple tests failed (see below).

Compiling python without --disable-test-modules makes the tests pass, and the output indicates that the failures are due to the test API not being available.

I suppose it's not too big of a deal, but this might normalize seeing errors when building python, leading users to ignore actual test failures.

Details 
# Next, run the profile task to generate the profile information.
LD_LIBRARY_PATH=/home/chaosflaws/Python-3.12.2 ./python -m test --pgo --timeout=1200 || true
Using random seed: 3095374302
0:00:00 load avg: 1.00 Run 44 tests sequentially (timeout: 20 min)
0:00:00 load avg: 1.00 [ 1/44] test_array
0:00:05 load avg: 1.00 [ 2/44] test_base64
0:00:07 load avg: 1.00 [ 3/44] test_binascii
0:00:08 load avg: 1.00 [ 4/44] test_binop
0:00:08 load avg: 1.00 [ 5/44] test_bisect
0:00:09 load avg: 1.00 [ 6/44] test_bytes
0:00:09 load avg: 1.00 [ 7/44] test_bz2 -- test_bytes skipped
0:00:12 load avg: 1.08 [ 8/44] test_cmath
0:00:13 load avg: 1.08 [ 9/44] test_codecs
0:00:20 load avg: 1.07 [10/44] test_collections
0:00:25 load avg: 1.07 [11/44] test_complex
0:00:27 load avg: 1.06 [12/44] test_dataclasses
0:00:29 load avg: 1.06 [13/44] test_datetime
0:00:44 load avg: 1.05 [14/44] test_decimal
0:01:08 load avg: 1.03 [15/44] test_difflib
0:01:14 load avg: 1.03 [16/44] test_embed
test test_embed failed
0:01:28 load avg: 1.02 [17/44] test_float -- test_embed failed (37 errors, 3 failures)
0:01:30 load avg: 1.02 [18/44] test_fstring
0:01:34 load avg: 1.02 [19/44] test_functools
0:01:37 load avg: 1.02 [20/44] test_generators
0:01:39 load avg: 1.02 [21/44] test_hashlib
0:01:41 load avg: 1.02 [22/44] test_heapq
0:01:45 load avg: 1.02 [23/44] test_int
test test_int failed
0:01:47 load avg: 1.01 [24/44] test_itertools -- test_int failed (2 errors)
0:02:15 load avg: 1.01 [25/44] test_json
0:02:26 load avg: 1.08 [26/44] test_long
0:02:44 load avg: 1.14 [27/44] test_lzma
0:02:45 load avg: 1.14 [28/44] test_math
0:03:02 load avg: 1.10 [29/44] test_memoryview
0:03:05 load avg: 1.10 [30/44] test_operator
0:03:05 load avg: 1.10 [31/44] test_ordered_dict
0:03:13 load avg: 1.08 [32/44] test_patma
0:03:14 load avg: 1.08 [33/44] test_pickle
0:03:48 load avg: 1.05 [34/44] test_pprint -- test_pickle passed in 34.1 sec
0:03:50 load avg: 1.05 [35/44] test_re
0:03:54 load avg: 1.04 [36/44] test_set
0:04:24 load avg: 1.02 [37/44] test_sqlite3
Failed to import test module: test.test_sqlite3.test_dbapi
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_dump
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dump.py", line 5, in <module>
    from .test_dbapi import memory_database
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_hooks
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_hooks.py", line 29, in <module>
    from test.test_sqlite3.test_dbapi import memory_database, cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_regression
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_regression.py", line 31, in <module>
    from test.test_sqlite3.test_dbapi import memory_database, cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_transactions
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_transactions.py", line 30, in <module>
    from test.test_sqlite3.test_dbapi import memory_database
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

Failed to import test module: test.test_sqlite3.test_userfunctions
Traceback (most recent call last):
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 394, in _find_test_path
    module = self._get_module_from_name(name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chaosflaws/Python-3.12.2/Lib/unittest/loader.py", line 337, in _get_module_from_name
    __import__(name)
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_userfunctions.py", line 35, in <module>
    from test.test_sqlite3.test_dbapi import cx_limit
  File "/home/chaosflaws/Python-3.12.2/Lib/test/test_sqlite3/test_dbapi.py", line 37, in <module>
    from _testcapi import INT_MAX, ULLONG_MAX
ModuleNotFoundError: No module named '_testcapi'

0:04:25 load avg: 1.02 [38/44] test_statistics -- test_sqlite3 failed (uncaught exception)
0:04:38 load avg: 1.02 [39/44] test_struct
0:04:41 load avg: 1.02 [40/44] test_tabnanny
0:04:43 load avg: 1.02 [41/44] test_time
0:04:46 load avg: 1.02 [42/44] test_unicode
0:04:59 load avg: 1.01 [43/44] test_xml_etree
0:05:02 load avg: 1.01 [44/44] test_xml_etree_c

Total duration: 5 min 8 sec
Total tests: run=8,196 failures=3 skipped=267
Total test files: run=44/44 failed=3 skipped=1
Result: FAILURE

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

Activity

  1. chaosflaws
    added
    type-bugAn unexpected behavior, bug, or error
    on Mar 4, 2024
  2. encukou
    added
    testsTests in the Lib/test dir
    topic-sqlite3
    on Mar 4, 2024

  3. erlend-aasland commented on Mar 4, 2024

    @erlend-aasland
    erlend-aasland
    on Mar 4, 2024
    Contributor

    The test fails because _testcapi is not available. This should fix it:

    Details 
    diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py
index 588272448b..e46c62cce8 100644
--- a/Lib/test/test_sqlite3/test_dbapi.py
+++ b/Lib/test/test_sqlite3/test_dbapi.py
@@ -31,11 +31,10 @@
 
 from test.support import (
     SHORT_TIMEOUT, check_disallow_instantiation, requires_subprocess,
-    is_apple, is_emscripten, is_wasi
+    is_apple, is_emscripten, is_wasi, import_helper
 )
 from test.support import gc_collect
 from test.support import threading_helper
-from _testcapi import INT_MAX, ULLONG_MAX
 from os import SEEK_SET, SEEK_CUR, SEEK_END
 from test.support.os_helper import TESTFN, TESTFN_UNDECODABLE, unlink, temp_dir, FakePath
 
@@ -1200,6 +1199,7 @@ def test_blob_seek_and_tell(self):
         self.assertEqual(self.blob.tell(), 40)
 
     def test_blob_seek_error(self):
+        testcapi = import_helper.import_module("_testcapi")
         msg_oor = "offset out of blob range"
         msg_orig = "'origin' should be os.SEEK_SET, os.SEEK_CUR, or os.SEEK_END"
         msg_of = "seek offset results in overflow"
@@ -1217,9 +1217,9 @@ def test_blob_seek_error(self):
         # Force overflow errors
         self.blob.seek(1, SEEK_SET)
         with self.assertRaisesRegex(OverflowError, msg_of):
-            self.blob.seek(INT_MAX, SEEK_CUR)
+            self.blob.seek(testcapi.INT_MAX, SEEK_CUR)
         with self.assertRaisesRegex(OverflowError, msg_of):
-            self.blob.seek(INT_MAX, SEEK_END)
+            self.blob.seek(testcapi.INT_MAX, SEEK_END)
 
     def test_blob_read(self):
         buf = self.blob.read()
@@ -1374,13 +1374,14 @@ def test_blob_mapping_invalid_index_type(self):
             self.blob["a"] = b"b"
 
     def test_blob_get_item_error(self):
+        testcapi = import_helper.import_module("_testcapi")
         dataset = [len(self.blob), 105, -105]
         for idx in dataset:
             with self.subTest(idx=idx):
                 with self.assertRaisesRegex(IndexError, "index out of range"):
                     self.blob[idx]
         with self.assertRaisesRegex(IndexError, "cannot fit 'int'"):
-            self.blob[ULLONG_MAX]
+            self.blob[testcapi.ULLONG_MAX]
 
         # Provoke read error
         self.cx.execute("update test set b='aaaa' where rowid=1")
  4. erlend-aasland
    added a commit that references this issue on Mar 4, 2024

    pythongh-116303: Skip some sqlite3 tests if testcapi is unavailable

    9b3f0f7

  5. erlend-aasland commented on Mar 4, 2024

    @erlend-aasland
    erlend-aasland
    on Mar 4, 2024
    Contributor

    We should have a buildbot for --disable-test-modules.

  6. erlend-aasland commented on Mar 4, 2024

    @erlend-aasland
    erlend-aasland
    on Mar 4, 2024
    Contributor

    test_stable_abi_ctypes may also have the same problem.

  7. erlend-aasland commented on Mar 4, 2024

    @erlend-aasland
    erlend-aasland
    on Mar 4, 2024
    Contributor

    Actually, there's more than a handful of tests failing (around 35 tests). I'll update the PR to cover all cases, but it will take a lot of time as it is tedious work.

  8. erlend-aasland
    added a commit that references this issue on Mar 4, 2024

    pythongh-116303: Skip some sqlite3 tests if testcapi is unavailable

    fc76e3b
  9. erlend-aasland
    changed the title [-]PGO tests fail if `disable-test-modules` is suplied[/-] [+]Tests fail if `--disable-test-modules` is suplied[/+] on Mar 5, 2024
  10. brettcannon
    changed the title [-]Tests fail if `--disable-test-modules` is suplied[/-] [+]Tests fail if `--disable-test-modules` is supplied[/+] on Mar 5, 2024
  11. erlend-aasland
    added 3 commits that reference this issue on Mar 7, 2024

    pythongh-116303: Explicitly check for the _testsinglephase module in …

    93eb35d

    pythongh-116303: Handle disabled test modules in test.support

    c5fa18b

    gh-116303: Explicitly check for the _testsinglephase module in config…

    d864b00
  12. adorilson
    added a commit that references this issue on Mar 25, 2024

    pythongh-116303: Explicitly check for the _testsinglephase module in …

    ff4b157

  13. 5 remaining items

  14. NGRsoftlab
    added a commit that references this issue on Apr 2, 2024

    Squashed commit of the following:

    a016a54
  15. erlend-aasland
    added 3 commits that reference this issue on Apr 3, 2024

    gh-116303: Skip test module dependent tests if test modules are unava…

    ea94b3b

    pythongh-116303: Don't build xxlimited and xxlimited_35 if --disable-…

    2b48f3f

    gh-116303: Don't build xxlimited and xxlimited_35 if --disable-test-m…

    9c1dfe2
  16. encukou
    added a commit that references this issue on Apr 8, 2024

    gh-116303: Skip tests if C recursion limit is unavailable (GH-117368)

    ca62ffd

  17. erlend-aasland commented on Apr 8, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 8, 2024
    Contributor

    @chaosflaws, I think we've got all --disable-test-modules issues covered now. Could you verify that it works for you?

  18. chaosflaws commented on Apr 8, 2024

    @chaosflaws
    chaosflaws
    on Apr 8, 2024
    Author

    Just built it using main, looks good to me :)

    Edit, for posterity:
    Commit was ca62ffd
    Command executed was ./configure --enable-optimizations --disable-test-modules --prefix=[some-local-path]

    # Next, run the profile task to generate the profile information.
./python -m test --pgo --timeout=
Using random seed: 72227414
0:00:00 load avg: 1.07 Run 44 tests sequentially
0:00:00 load avg: 1.07 [ 1/44] test_array
0:00:01 load avg: 1.07 [ 2/44] test_base64
0:00:02 load avg: 1.07 [ 3/44] test_binascii
0:00:02 load avg: 1.07 [ 4/44] test_binop
0:00:02 load avg: 1.07 [ 5/44] test_bisect
0:00:02 load avg: 1.07 [ 6/44] test_bytes
0:00:03 load avg: 1.07 [ 7/44] test_bz2 -- test_bytes skipped
0:00:04 load avg: 1.07 [ 8/44] test_cmath
0:00:04 load avg: 1.07 [ 9/44] test_codecs
0:00:06 load avg: 1.07 [10/44] test_collections
0:00:08 load avg: 1.07 [11/44] test_complex
0:00:08 load avg: 1.07 [12/44] test_dataclasses
0:00:09 load avg: 1.07 [13/44] test_datetime
0:00:16 load avg: 0.98 [14/44] test_decimal
0:00:24 load avg: 0.98 [15/44] test_difflib
0:00:26 load avg: 0.98 [16/44] test_embed
0:00:30 load avg: 0.98 [17/44] test_float
0:00:30 load avg: 0.98 [18/44] test_fstring
0:00:33 load avg: 0.98 [19/44] test_functools
0:00:35 load avg: 0.98 [20/44] test_generators
0:00:35 load avg: 0.98 [21/44] test_hashlib
0:00:36 load avg: 0.98 [22/44] test_heapq
0:00:38 load avg: 0.98 [23/44] test_int
0:00:38 load avg: 0.98 [24/44] test_itertools
0:00:46 load avg: 0.99 [25/44] test_json
0:00:53 load avg: 0.99 [26/44] test_long
0:00:57 load avg: 0.99 [27/44] test_lzma
0:00:58 load avg: 0.99 [28/44] test_math
0:01:02 load avg: 0.99 [29/44] test_memoryview
0:01:03 load avg: 0.99 [30/44] test_operator
0:01:03 load avg: 0.99 [31/44] test_ordered_dict
0:01:06 load avg: 0.99 [32/44] test_patma
0:01:06 load avg: 0.99 [33/44] test_pickle
0:01:16 load avg: 0.99 [34/44] test_pprint
0:01:16 load avg: 0.99 [35/44] test_re
0:01:18 load avg: 0.99 [36/44] test_set
0:01:26 load avg: 0.99 [37/44] test_sqlite3
0:01:27 load avg: 0.99 [38/44] test_statistics
0:01:40 load avg: 1.00 [39/44] test_str
0:01:43 load avg: 1.00 [40/44] test_struct
0:01:45 load avg: 1.00 [41/44] test_tabnanny
0:01:46 load avg: 1.00 [42/44] test_time
0:01:48 load avg: 1.00 [43/44] test_xml_etree
0:01:50 load avg: 1.08 [44/44] test_xml_etree_c

Total duration: 1 min 53 sec
Total tests: run=8,909 skipped=328
Total test files: run=44/44 skipped=1
Result: SUCCESS

  19. erlend-aasland commented on Apr 8, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 8, 2024
    Contributor

    Great, thanks for verifying! :)

    Closing this as completed.

  20. erlend-aasland
    closed this as completedon Apr 8, 2024
  21. erlend-aasland
    removed
    topic-sqlite3
    on Apr 8, 2024
  22. diegorusso
    added 5 commits that reference this issue on Apr 17, 2024

    pythongh-116303: Explicitly check for the _testsinglephase module in …

    45cce92

    pythongh-116303: Handle disabled test modules in test.support helpers (…

    c28a6fc

    pythongh-116303: Skip test module dependent tests if test modules are…

    8b7aa61

    pythongh-116303: Don't build xxlimited and xxlimited_35 if --disable-…

    a54f64f

    pythongh-116303: Skip tests if C recursion limit is unavailable (pyth…

    18ac5a6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    testsTests in the Lib/test dirtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions