Skip to content

"wasm32-wasi 3.x" buildbot times out on building Python #151929

Closed
Closed
"wasm32-wasi 3.x" buildbot times out on building Python#151929
Labels
OS-wasibuildThe build process and cross-buildtype-bugAn unexpected behavior, bug, or error

Description

@vstinner
vstinner
opened on Jun 22, 2026

Bug report

The wasm32-wasi 3.x buildbot worker seems to be affected by this LLVM (clang) performance issue: llvm/llvm-project#179695. Sometimes, the "Compile build Python" step is killed by a timeout.

Example of failure: the "Compile build Python" times out after 35 minutes 10 seconds.

clang -shared      Modules/pyexpat.o -lm Modules/expat/libexpat.a  -o Modules/pyexpat.cpython-316d-x86_64-linux-gnu.so
clang -shared      Modules/_elementtree.o   -o Modules/_elementtree.cpython-316d-x86_64-linux-gnu.so

command timed out: 1200 seconds without output running [b'python3', b'Tools/wasm/wasi.py', b'make-build-python'], attempting to kill
process killed by signal 9
program finished with exit code -1

"Compile build Python" uses clang -Og.

Linked PRs

Activity

  1. vstinner
    added
    type-bugAn unexpected behavior, bug, or error
    buildThe build process and cross-build
    on Jun 22, 2026

  2. vstinner commented on Jun 22, 2026

    @vstinner
    vstinner
    on Jun 22, 2026
    MemberAuthor

    cc @brettcannon

  3. StanFromIreland
    added
    OS-wasi
    on Jun 22, 2026

  4. vstinner commented on Jun 22, 2026

    @vstinner
    vstinner
    on Jun 22, 2026
    MemberAuthor

    I wrote cpython-devcontainers PRs to upgrade Fedora from 43 to 44, but stick to LLVM 21:

  5. brettcannon commented on Jun 22, 2026

    @brettcannon
    brettcannon
    on Jun 22, 2026
    Member

    It might also be the buildbot itself. A relaunch of the buildbot container got it to completion, but perf is still bad. I'll try a reboot after the 3.15.0b3 release.

  6. brettcannon commented on Jun 23, 2026

    @brettcannon
    brettcannon
    on Jun 23, 2026
    Member

    Reboot helped a little, but a full run has still gone from 00:30 to 1:14.

  7. vstinner commented on Jun 24, 2026

    @vstinner
    vstinner
    on Jun 24, 2026
    MemberAuthor

    Reboot helped a little, but a full run has still gone from 00:30 to 1:14.

    The C compiled used to compile host Python is 4.6x slower and seems to produce 3.4x slower WASI code. Did you change the WASI toolchain in the last 5 days?

    • Configure build Python: 50 sec => 1 min 03 sec (similar)
    • Compile build Python: 11 min 39 sec => 7 min 17 sec (1.6x faster) -- LLVM (clang) was upgraded from LLVM 21 to LLVM 22
    • Configure host Python: 57 sec => 5 min 02 sec (5.3x slower)
    • Compile host Python: 4 min 22 sec => 19 min 56 sec (4.6x slower)
    • test: 11 min 50 sec => 40 min 21 sec (3.4x slower)

    Build 234 (before): 10 slowest tests:

    • test_zipfile: 1 min 28 sec
    • test_fstring: 42.0 sec
    • test_tarfile: 40.9 sec
    • test_pickle: 39.6 sec
    • test_bytes: 27.8 sec
    • test_decimal: 23.9 sec
    • test_str: 22.6 sec
    • test_statistics: 21.4 sec
    • test_set: 20.2 sec
    • test___all__: 19.7 sec

    Build 250: 10 slowest tests:

    • test_zipfile: 5 min 27 sec
    • test_tarfile: 2 min 33 sec
    • test_fstring: 2 min 24 sec
    • test_bytes: 2 min 24 sec
    • test_pickle: 2 min 23 sec
    • test_decimal: 1 min 8 sec
    • test_set: 1 min 4 sec
    • test_userstring: 52.5 sec
    • test_str: 51.3 sec
    • test_unicodedata: 50.6 sec

    So for example, test_zipfile became 3.7x slower.

  8. vstinner commented on Jun 24, 2026

    @vstinner
    vstinner
    on Jun 24, 2026
    MemberAuthor

    Right now, /opt/wasi-sdk-33.0-x86_64-linux/bin/clang is clang 22.1:

    # /opt/wasi-sdk-33.0-x86_64-linux/bin/clang --version
clang version 22.1.0-wasi-sdk (https://github.com/llvm/llvm-project 4434dabb69916856b824f68a64b029c67175e532)
Target: wasm32-unknown-wasip1
Thread model: posix
InstalledDir: /opt/wasi-sdk-33.0-x86_64-linux/bin
Configuration file: /opt/wasi-sdk-33.0-x86_64-linux/bin/clang.cfg

  9. vstinner commented on Jun 24, 2026

    @vstinner
    vstinner
    on Jun 24, 2026
    MemberAuthor

    I built Python for WASI on my Fedora 44 laptop (12 CPUs):

    • Configure build Python: 40 sec (--with-pydebug)
    • Compile build Python: 57 sec (clang -Og)
    • Configure host Python: 44 sec
    • Compile host Python: 1 min 29 sec (/opt/wasi-sdk-33.0-x86_64-linux/bin/clang -O3)
    • test: 6 min 57 sec

    Timings are quite good, better than "wasm32-wasi 3.15" worker.

    So maybe something is wrong on "wasm32-wasi 3.15" worker?

  10. brettcannon commented on Jun 24, 2026

    @brettcannon
    brettcannon
    on Jun 24, 2026
    Member

    So maybe something is wrong on "wasm32-wasi 3.15" worker?

    That's what I'm starting to wonder, although I don't know what since it's a VM in Azure that does nothing else but run as a buildbot.

  11. vstinner
    added 4 commits that reference this issue on Jun 24, 2026

    gh-151929: Get boot ID, machine ID and uptime in test.pythoninfo (#15…

    3db3bba

    [3.14] gh-151929: Get boot ID, machine ID and uptime in test.pythonin…

    6e61e3c

    [3.15] gh-151929: Get boot ID, machine ID and uptime in test.pythonin…

    7d5e2dd

    [3.13] gh-151929: Get boot ID, machine ID and uptime in test.pythonin…

    7ca8f36

  12. 4 remaining items

  13. vstinner
    added 13 commits that reference this issue on Jun 25, 2026

    [3.14] gh-151929: Get machine ID and uptime on Windows in pythoninfo (#…

    be65a38

    gh-151929: Get uptime on BSD/macOS in pythoninfo (#152189)

    b6d89ed

    [3.13] gh-151929: Get machine ID and uptime on Windows in pythoninfo (#…

    5b5f372

    [3.14] gh-151929: Get uptime on BSD/macOS in pythoninfo (#152189) (#1…

    d25cf02

    [3.15] gh-151929: Get uptime on BSD/macOS in pythoninfo (GH-152189) (#…

    1926cf3

    [3.13] gh-151929: Get uptime on BSD/macOS in pythoninfo (#152189) (#1…

    704c4c7

    gh-151929: Add pythoninfo-build command to Platforms/emscripten (#152210

    7676427

    [3.15] gh-151929: Add pythoninfo-build command to Platforms/emscripten (

    25c7b0c

    [3.14] gh-151929: Add pythoninfo-build command to Platforms/emscripten (

    8162f61

    gh-151929: Add pythoninfo commands to Platforms/WASI (#152136)

    7c81637

    [3.15] gh-151929: Add pythoninfo commands to Platforms/WASI (GH-152136)…

    59e54f5

    [3.14] gh-151929: Add pythoninfo commands to Tools/wasm/wasi (#152136) (

    6426887

    [3.13] gh-151929: Add pythoninfo commands to Tools/wasm/wasi.py (#152136

    d740050

  14. brettcannon commented on Jun 29, 2026

    @brettcannon
    brettcannon
    on Jun 29, 2026
    Member

    As of https://buildbot.python.org/all/#/builders/2062/builds/295, the runs are back to taking around 20 minutes.

  15. brettcannon
    closed this as completedon Jun 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS-wasibuildThe build process and cross-buildtype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions