Skip to content

Weird PGO build issue on macOS 14.3 #117752

Closed
Closed
Weird PGO build issue on macOS 14.3#117752
Labels
OS-macbuildThe build process and cross-build

Description

@corona10
corona10
opened on Apr 11, 2024
  • Xcode: 15.3 (22618)
  • Instruments: 15.3 (64565.111)

With ./configure --enable-optimizations, the PGO build itself is a success on my machine, but the status is weird.

Following build warnings are emitted at the first build

clang: warning: argument unused during compilation: '-fno-semantic-interposition' [-Wunused-command-line-argument]
...

During the profiling stage, several tests are not passed as before.

# Next, run the profile task to generate the profile information.
LLVM_PROFILE_FILE="code-%p.profclangr"  ./python.exe -m test --pgo --timeout=
Using random seed: 3569111910
Raised RLIMIT_NOFILE: 256 -> 1024
0:00:00 load avg: 4.25 Run 44 tests sequentially
0:00:00 load avg: 4.25 [ 1/44] test_array
0:00:00 load avg: 4.25 [ 2/44] test_base64
0:00:00 load avg: 4.25 [ 3/44] test_binascii -- test_base64 failed (env changed)
0:00:00 load avg: 4.25 [ 4/44] test_binop
0:00:00 load avg: 4.25 [ 5/44] test_bisect
0:00:00 load avg: 4.25 [ 6/44] test_bytes
0:00:02 load avg: 4.25 [ 7/44] test_bz2 -- test_bytes failed (env changed)
0:00:02 load avg: 4.25 [ 8/44] test_cmath
0:00:02 load avg: 4.25 [ 9/44] test_codecs
0:00:03 load avg: 4.25 [10/44] test_collections
0:00:03 load avg: 4.25 [11/44] test_complex
0:00:03 load avg: 4.25 [12/44] test_dataclasses
0:00:04 load avg: 4.25 [13/44] test_datetime
0:00:06 load avg: 4.63 [14/44] test_decimal
------------------------------------ NOTICE ------------------------------------
test_decimal may generate "malloc can't allocate region"
warnings on macOS systems. This behavior is known. Do not
report a bug unless tests are also failing.
See https://github.com/python/cpython/issues/85100
--------------------------------------------------------------------------------
0:00:08 load avg: 4.63 [15/44] test_difflib
0:00:09 load avg: 4.63 [16/44] test_embed
0:00:12 load avg: 4.50 [17/44] test_float -- test_embed failed (env changed)
0:00:12 load avg: 4.50 [18/44] test_fstring
0:00:13 load avg: 4.50 [19/44] test_functools
0:00:14 load avg: 4.50 [20/44] test_generators
0:00:14 load avg: 4.50 [21/44] test_hashlib
0:00:14 load avg: 4.62 [22/44] test_heapq
0:00:15 load avg: 4.62 [23/44] test_int
0:00:15 load avg: 4.62 [24/44] test_itertools
0:00:16 load avg: 4.62 [25/44] test_json -- test_itertools failed (env changed)
0:00:18 load avg: 4.62 [26/44] test_long -- test_json failed (env changed)
0:00:19 load avg: 4.41 [27/44] test_lzma
0:00:20 load avg: 4.41 [28/44] test_math
0:00:21 load avg: 4.41 [29/44] test_memoryview
0:00:21 load avg: 4.41 [30/44] test_operator
0:00:21 load avg: 4.41 [31/44] test_ordered_dict
0:00:21 load avg: 4.41 [32/44] test_patma
0:00:22 load avg: 4.41 [33/44] test_pickle
0:00:24 load avg: 4.41 [34/44] test_pprint
0:00:24 load avg: 4.41 [35/44] test_re
0:00:24 load avg: 4.38 [36/44] test_set -- test_re failed (env changed)
0:00:26 load avg: 4.38 [37/44] test_sqlite3
0:00:26 load avg: 4.38 [38/44] test_statistics -- test_sqlite3 failed (env changed)
0:00:29 load avg: 4.11 [39/44] test_str
0:00:30 load avg: 4.11 [40/44] test_struct -- test_str failed (env changed)
0:00:30 load avg: 4.11 [41/44] test_tabnanny -- test_struct failed (env changed)
0:00:30 load avg: 4.11 [42/44] test_time -- test_tabnanny failed (env changed)
0:00:32 load avg: 4.11 [43/44] test_xml_etree
0:00:33 load avg: 4.11 [44/44] test_xml_etree_c

Total duration: 33.8 sec
Total tests: run=9,203 skipped=182
Total test files: run=44/44 env_changed=10
Result: SUCCESS

Also, writing profile files is in a weird status.

/usr/bin/xcrun llvm-profdata merge -output=code.profclangd *.profclangr
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory
LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory

cc @ned-deily @ronaldoussoren @erlend-aasland

Linked PRs

Retargeted PRs:

Activity

  1. corona10
    added
    OS-mac
    on Apr 11, 2024
  2. erlend-aasland
    added
    buildThe build process and cross-build
    on Apr 11, 2024
  3. erlend-aasland
    changed the title [-]Weired PGO build on macOS 14.3[/-] [+]Weird PGO build issue on macOS 14.3[/+] on Apr 11, 2024

  4. erlend-aasland commented on Apr 11, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 11, 2024
    Contributor

    Related to -fno-semantic-interposition:

  5. erlend-aasland commented on Apr 11, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 11, 2024
    Contributor

    Related issues:

    UPDATE: I kept #96398 open and closed the other as duplicates or superseded. Let's not forget this issue.

  6. erlend-aasland commented on Apr 11, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 11, 2024
    Contributor

    Following build warnings are emitted at the first build [...]

    clang: warning: argument unused during compilation: '-fno-semantic-interposition' [-Wunused-command-line-argument]

    This clang warning is also mentioned in issue #110577. Some of our $CC checks in configure.ac are inaccurate1. However, IIUC c6d7e82 should have take care of that.

    For example, applying this patch (as hinted to in #110577) gets rid of the warning:

    Details 
    diff --git a/configure.ac b/configure.ac
index e195e15b39..325b5b973b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1796,10 +1796,12 @@ if test "$Py_OPT" = 'true' ; then
   DEF_MAKE_RULE="build_all"
   case $CC in
     *gcc*)
-      AX_CHECK_COMPILE_FLAG([-fno-semantic-interposition],[
-      CFLAGS_NODIST="$CFLAGS_NODIST -fno-semantic-interposition"
-      LDFLAGS_NODIST="$LDFLAGS_NODIST -fno-semantic-interposition"
-      ])
+      if test -n "${cc_is_clang}"; then
+        AX_CHECK_COMPILE_FLAG([-fno-semantic-interposition],[
+        CFLAGS_NODIST="$CFLAGS_NODIST -fno-semantic-interposition"
+        LDFLAGS_NODIST="$LDFLAGS_NODIST -fno-semantic-interposition"
+        ])
+      fi
       ;;
   esac
 elif test "$ac_sys_system" = "Emscripten" -o "$ac_sys_system" = "WASI"; then

    Footnotes

    1. which is also mentioned in some of the other related issues I linked to]. ↩

  7. erlend-aasland
    added a commit that references this issue on Apr 11, 2024

    pythongh-117752: Autoconf: fix -fno-semantic-interposition check

    a04fb5c

  8. erlend-aasland commented on Apr 11, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 11, 2024
    Contributor

    Regarding this:

    LLVM Profile Error: Failed to write file "code-75991.profclangr": No such file or directory

    Try this patch:

    diff --git a/configure.ac b/configure.ac
index 60c4141a3d..2570fc5f32 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2013,7 +2013,7 @@ case "$CC_BASENAME" in
     PGO_PROF_GEN_FLAG="-fprofile-instr-generate"
     PGO_PROF_USE_FLAG="-fprofile-instr-use=code.profclangd"
     LLVM_PROF_MERGER="${LLVM_PROFDATA} merge -output=code.profclangd *.profclangr"
-    LLVM_PROF_FILE="LLVM_PROFILE_FILE=\"code-%p.profclangr\""
+    LLVM_PROF_FILE="LLVM_PROFILE_FILE=\"\$(shell pwd)/code-%p.profclangr\""
     if test $LLVM_PROF_FOUND = not-found
     then
       LLVM_PROF_ERR=yes
@@ -2029,7 +2029,7 @@ case "$CC_BASENAME" in
         PGO_PROF_GEN_FLAG="-fprofile-instr-generate"
         PGO_PROF_USE_FLAG="-fprofile-instr-use=code.profclangd"
         LLVM_PROF_MERGER="${LLVM_PROFDATA} merge -output=code.profclangd *.profclangr"
-        LLVM_PROF_FILE="LLVM_PROFILE_FILE=\"code-%p.profclangr\""
+        LLVM_PROF_FILE="LLVM_PROFILE_FILE=\"\$(shell pwd)/code-%p.profclangr\""
         if test "${LLVM_PROF_FOUND}" = "not-found"
         then
           LLVM_PROF_ERR=yes

    This should also get rid of the "env changed" warnings, as profile data will now not interfere with the test environment.

  9. erlend-aasland commented on Apr 11, 2024

    @erlend-aasland
    erlend-aasland
    on Apr 11, 2024
    Contributor

    Moreover, we should consider compiling with -Wno-profile-instr-unprofiled (and perhaps also -Wno-profile-instr-out-of-date) when PGO is enabled. There is no need to warn about the files that lack profile data.

  10. erlend-aasland
    added 2 commits that reference this issue on Apr 12, 2024

    pythongh-117752: Autoconf: store all LLVM profile data in the build d…

    b35b8d4

    gh-117752: Autoconf: store all LLVM profile data in the build directo…

    396b831
  11. miss-islington
    added a commit that references this issue on Apr 12, 2024

    pythongh-117752: Autoconf: store all LLVM profile data in the build d…

    c903fc8
  12. erlend-aasland
    added 3 commits that reference this issue on Apr 12, 2024

    pythongh-117752: Autoconf: silence some clang warnings for PGO builds

    4238280

    [3.12] gh-117752: Autoconf: store all LLVM profile data in the build …

    c64b65e

    gh-117752: Autoconf: fix -fno-semantic-interposition check (#117789)

    49fc141
  13. miss-islington
    added a commit that references this issue on Apr 12, 2024

    pythongh-117752: Autoconf: fix -fno-semantic-interposition check (pyt…

    0f8e7da
  14. erlend-aasland
    added 3 commits that reference this issue on Apr 12, 2024

    [3.12] gh-117752: Autoconf: fix -fno-semantic-interposition check (GH…

    3f2f0cf

    pythongh-117752: Autoconf: fix PGO builds for 'make -C build' incanta…

    754312a

    gh-117752: Autoconf: fix PGO builds for 'make -C build' incantations (#…

    1ddbeae
  15. miss-islington
    added a commit that references this issue on Apr 12, 2024

    pythongh-117752: Autoconf: fix PGO builds for 'make -C build' incanta…

    0cb6d2f
  16. erlend-aasland
    added a commit that references this issue on Apr 12, 2024

    [3.12] gh-117752: Autoconf: fix PGO builds for 'make -C build' incant…

    a39e01d
  17. erlend-aasland
    closed this as completedon Apr 12, 2024
  18. diegorusso
    added 3 commits that reference this issue on Apr 17, 2024

    pythongh-117752: Autoconf: store all LLVM profile data in the build d…

    826ff92

    pythongh-117752: Autoconf: fix -fno-semantic-interposition check (pyt…

    4a2bf5f

    pythongh-117752: Autoconf: fix PGO builds for 'make -C build' incanta…

    8280f2b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-macbuildThe build process and cross-build

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions