Skip to content

Optimizing BOLT flags #128514

Open
Open
Optimizing BOLT flags#128514
Assignees
zaniebcorona10
Labels
buildThe build process and cross-buildperformancePerformance or resource usagetype-featureA feature request or enhancement

Description

@zanieb
zanieb
opened on Jan 5, 2025

Feature or enhancement

This is a tracking issue for discussion on determining the optimal flags for BOLT to improve performance.

Tuning the flags is mentioned in #101525, but doesn't feel like a blocker for stabilization.

Linked PRs

Activity

  1. zanieb
    added
    type-featureA feature request or enhancement
    performancePerformance or resource usage
    buildThe build process and cross-build
    on Jan 5, 2025

  2. zanieb commented on Jan 5, 2025

    @zanieb
    zanieb
    on Jan 5, 2025
    ContributorAuthor

    There was a talk in March 2024 at the LLVM Performance Workshop; I can't find a copy of the talk online but the slides are available at https://llvm.org/devmtg/2024-03/slides/practical-use-of-bolt.pdf

    It includes the following suggestions:

    • Function splitting: -split-functions, -split-strategy=cdsplit
    • Function reordering: -reorder-functions=cdsort
    • Block reordering: -reorder-blocks=ext-tsp
    • Use THP pages for hot text: -hugify
    • PLT optimization: -plt
    • More aggressive ICF: -icf
    • Indirect Call Promotion: -indirect-call-promotion

    We're currently using:

    cpython/configure.ac

    Lines 2199 to 2212 in b60044b

    -reorder-blocks=ext-tsp
    -reorder-functions=cdsort
    -split-functions
    -icf=1
    -inline-all
    -split-eh
    -reorder-functions-use-hot-size
    -peepholes=none
    -jump-tables=aggressive
    -inline-ap
    -indirect-call-promotion=all
    -dyno-stats
    -use-gnu-stack
    -frame-opt=hot

    Suggesting we should explore -split-strategy=cdsplit, -hugify, and -plt

  3. zanieb commented on Jan 5, 2025

    @zanieb
    zanieb
    on Jan 5, 2025
    ContributorAuthor

    There's some commentary in #124948 (comment)

    python-build-standalone recently added -hugify and -split-strategy=cdsplit (astral-sh/python-build-standalone#462), though the performance benefits were not validated.

    My intent is to do some benchmarking for each flag.

  4. liusy58 commented on Jan 7, 2025

    @liusy58
    liusy58
    on Jan 7, 2025

    Ask me everything if you need. And I am now working on BOLT, I also want to contribute to Python!

  5. liusy58 commented on Jan 7, 2025

    @liusy58
    liusy58
    on Jan 7, 2025

    By the way, I wonder how you collect profiles? By instrumentation or perf ?

  6. liusy58 commented on Jan 7, 2025

    @liusy58
    liusy58
    on Jan 7, 2025

    I strongly recommend that --split-all-cold should be added.

  7. zanieb commented on Jan 7, 2025

    @zanieb
    zanieb
    on Jan 7, 2025
    ContributorAuthor

    I was going to collect benchmarks with https://github.com/python/pyperformance (i.e., not instrumentation) on my Linux machine.

    I think I can also post branches and ask the faster-cpython team to run benchmarks https://github.com/faster-cpython/benchmarking-public

    I have a few commits ready

  8. liusy58 commented on Jan 8, 2025

    @liusy58
    liusy58
    on Jan 8, 2025

    Profiles are key to BOLT. You are on x86, right? I remember cdsplit is not supported on AArch64.

  9. zanieb commented on Jan 8, 2025

    @zanieb
    zanieb
    on Jan 8, 2025
    ContributorAuthor

    I have machines with both architectures.

    Are you suggesting an alternative approach to measuring the effect?

  10. liusy58 commented on Jan 8, 2025

    @liusy58
    liusy58
    on Jan 8, 2025

    Yeah, maybe aarch64 can get more performance.

  11. zanieb commented on Jan 10, 2025

    @zanieb
    zanieb
    on Jan 10, 2025
    ContributorAuthor

    As an update, I set up an x86-64 bare metal machine with LLVM 19 and am running benchmarks for the flags I described above. I'm not including LTO in the baseline, should I?

  12. zanieb commented on Jan 11, 2025

    @zanieb
    zanieb
    on Jan 11, 2025
    ContributorAuthor

    Here are my initial results https://gist.github.com/zanieb/eee897b77597f93d0d3e0c2081dbc712
    Script at https://github.com/zanieb/cpython-bench/tree/34680389f94795b7b42d8c8dc784be34f0b14ce2

  13. corona10 commented on Jan 15, 2025

    @corona10
    corona10
    on Jan 15, 2025
    Member

    By the way, I wonder how you collect profiles? By instrumentation or perf ?

    FYI, We are getting BOLTed binary through instrumentation, not perf, when we actually build.

  14. corona10 commented on Jan 15, 2025

    @corona10
    corona10
    on Jan 15, 2025
    Member

    As an update, I set up an x86-64 bare metal machine with LLVM 19 and am running benchmarks for the flags I described above. I'm not including LTO in the baseline, should I?

    I belive that we don't have to, if only the difference between baseline is flag :)

  15. corona10
    assigned
    corona10
    and
    zanieb
    on Jan 15, 2025

  16. corona10 commented on Jan 15, 2025

    @corona10
    corona10
    on Jan 15, 2025
    Member

    (I am adding myself as assignee to catch up)

  17. zanieb commented on Jan 16, 2025

    @zanieb
    zanieb
    on Jan 16, 2025
    ContributorAuthor

    A second round of benchmarks with more samples comes out a little different https://gist.github.com/zanieb/8614bcb40b0db24dd678f2983146fb43

    The effect depends on the workload.

  18. zanieb commented on Feb 9, 2025

    @zanieb
    zanieb
    on Feb 9, 2025
    ContributorAuthor

    Following up with more analysis here

    hugify

    All benchmarks:
===============

Slower (23):
- sqlglot_normalize: 102 ms +- 1 ms -> 279 ms +- 3 ms: 2.74x slower
- python_startup_no_site: 6.52 ms +- 0.03 ms -> 8.53 ms +- 0.04 ms: 1.31x slower
- python_startup: 9.30 ms +- 0.03 ms -> 11.3 ms +- 0.0 ms: 1.21x slower
- spectral_norm: 93.0 ms +- 0.8 ms -> 98.5 ms +- 1.0 ms: 1.06x slower
- richards_super: 49.7 ms +- 0.4 ms -> 51.2 ms +- 0.4 ms: 1.03x slower
- generators: 25.3 ms +- 0.2 ms -> 26.1 ms +- 0.3 ms: 1.03x slower
- richards: 43.6 ms +- 0.4 ms -> 44.8 ms +- 0.4 ms: 1.03x slower
- coroutines: 20.4 ms +- 0.3 ms -> 21.0 ms +- 0.3 ms: 1.03x slower
- scimark_sor: 114 ms +- 1 ms -> 116 ms +- 1 ms: 1.02x slower
- mako: 10.2 ms +- 0.1 ms -> 10.4 ms +- 0.2 ms: 1.02x slower
- regex_dna: 147 ms +- 3 ms -> 150 ms +- 4 ms: 1.02x slower
- genshi_text: 19.6 ms +- 0.2 ms -> 19.9 ms +- 0.2 ms: 1.02x slower
- comprehensions: 15.3 us +- 0.1 us -> 15.5 us +- 0.2 us: 1.01x slower
- deltablue: 2.91 ms +- 0.03 ms -> 2.95 ms +- 0.03 ms: 1.01x slower
- xml_etree_iterparse: 78.7 ms +- 0.6 ms -> 79.7 ms +- 0.7 ms: 1.01x slower
- xml_etree_parse: 114 ms +- 1 ms -> 115 ms +- 1 ms: 1.01x slower
- raytrace: 256 ms +- 3 ms -> 258 ms +- 3 ms: 1.01x slower
- logging_simple: 5.25 us +- 0.07 us -> 5.29 us +- 0.10 us: 1.01x slower
- asyncio_tcp: 429 ms +- 4 ms -> 432 ms +- 4 ms: 1.01x slower
- pyflate: 380 ms +- 3 ms -> 382 ms +- 3 ms: 1.01x slower
- genshi_xml: 46.1 ms +- 0.6 ms -> 46.4 ms +- 0.6 ms: 1.01x slower
- go: 110 ms +- 1 ms -> 110 ms +- 1 ms: 1.00x slower
- asyncio_tcp_ssl: 1.48 sec +- 0.01 sec -> 1.48 sec +- 0.00 sec: 1.00x slower

Faster (39):
- logging_silent: 107 ns +- 1 ns -> 100 ns +- 1 ns: 1.07x faster
- regex_effbot: 2.47 ms +- 0.03 ms -> 2.33 ms +- 0.06 ms: 1.06x faster
- scimark_sparse_mat_mult: 4.21 ms +- 0.11 ms -> 3.98 ms +- 0.09 ms: 1.06x faster
- json_loads: 22.9 us +- 0.2 us -> 21.9 us +- 0.3 us: 1.05x faster
- json_dumps: 10.1 ms +- 0.2 ms -> 9.65 ms +- 0.16 ms: 1.04x faster
- regex_v8: 22.1 ms +- 0.2 ms -> 21.2 ms +- 0.2 ms: 1.04x faster
- telco: 6.60 ms +- 0.10 ms -> 6.38 ms +- 0.08 ms: 1.03x faster
- unpack_sequence: 37.4 ns +- 1.9 ns -> 36.2 ns +- 0.8 ns: 1.03x faster
- deepcopy: 250 us +- 2 us -> 242 us +- 2 us: 1.03x faster
- deepcopy_reduce: 2.60 us +- 0.03 us -> 2.52 us +- 0.04 us: 1.03x faster
- scimark_monte_carlo: 59.3 ms +- 0.4 ms -> 57.6 ms +- 0.6 ms: 1.03x faster
- deepcopy_memo: 29.6 us +- 0.2 us -> 28.7 us +- 0.3 us: 1.03x faster
- typing_runtime_protocols: 153 us +- 3 us -> 148 us +- 3 us: 1.03x faster
- scimark_fft: 313 ms +- 3 ms -> 305 ms +- 6 ms: 1.03x faster
- nqueens: 74.4 ms +- 0.7 ms -> 72.6 ms +- 0.5 ms: 1.02x faster
- chaos: 58.0 ms +- 0.8 ms -> 56.7 ms +- 0.7 ms: 1.02x faster
- pprint_pformat: 1.38 sec +- 0.01 sec -> 1.35 sec +- 0.02 sec: 1.02x faster
- scimark_lu: 112 ms +- 1 ms -> 110 ms +- 1 ms: 1.02x faster
- tomli_loads: 1.93 sec +- 0.02 sec -> 1.89 sec +- 0.03 sec: 1.02x faster
- pickle: 11.1 us +- 0.4 us -> 10.9 us +- 0.2 us: 1.02x faster
- pathlib: 19.2 ms +- 0.1 ms -> 18.8 ms +- 0.1 ms: 1.02x faster
- unpickle: 12.1 us +- 0.2 us -> 11.9 us +- 0.2 us: 1.02x faster
- logging_format: 5.82 us +- 0.13 us -> 5.74 us +- 0.12 us: 1.01x faster
- pprint_safe_repr: 667 ms +- 6 ms -> 657 ms +- 6 ms: 1.01x faster
- xml_etree_process: 54.6 ms +- 0.4 ms -> 53.8 ms +- 0.4 ms: 1.01x faster
- xml_etree_generate: 78.3 ms +- 0.5 ms -> 77.3 ms +- 0.6 ms: 1.01x faster
- pidigits: 160 ms +- 1 ms -> 158 ms +- 1 ms: 1.01x faster
- regex_compile: 118 ms +- 1 ms -> 117 ms +- 1 ms: 1.01x faster
- pickle_list: 3.84 us +- 0.07 us -> 3.80 us +- 0.16 us: 1.01x faster
- mdp: 2.33 sec +- 0.02 sec -> 2.30 sec +- 0.02 sec: 1.01x faster
- pickle_pure_python: 298 us +- 3 us -> 295 us +- 3 us: 1.01x faster
- meteor_contest: 89.6 ms +- 0.4 ms -> 89.0 ms +- 0.5 ms: 1.01x faster
- gc_traversal: 2.78 ms +- 0.06 ms -> 2.76 ms +- 0.05 ms: 1.01x faster
- html5lib: 60.5 ms +- 0.6 ms -> 60.1 ms +- 0.6 ms: 1.01x faster
- create_gc_cycles: 954 us +- 5 us -> 948 us +- 5 us: 1.01x faster
- dulwich_log: 60.0 ms +- 0.3 ms -> 59.7 ms +- 0.3 ms: 1.01x faster
- fannkuch: 372 ms +- 3 ms -> 370 ms +- 3 ms: 1.01x faster
- float: 66.4 ms +- 1.1 ms -> 66.0 ms +- 1.1 ms: 1.00x faster
- hexiom: 5.63 ms +- 0.06 ms -> 5.61 ms +- 0.05 ms: 1.00x faster

Benchmark hidden because not significant (11): 2to3, async_generators, asyncio_websockets, bench_mp_pool, bench_thread_pool, crypto_pyaes, docutils, nbody, pickle_dict, unpickle_list, unpickle_pure_python

Geometric mean: 1.01x slower

    cdsplit

    All benchmarks:
===============

Slower (41):
- sqlglot_normalize: 102 ms +- 1 ms -> 283 ms +- 2 ms: 2.78x slower
- deltablue: 2.91 ms +- 0.03 ms -> 3.20 ms +- 0.05 ms: 1.10x slower
- pyflate: 380 ms +- 3 ms -> 411 ms +- 4 ms: 1.08x slower
- scimark_fft: 313 ms +- 3 ms -> 329 ms +- 4 ms: 1.05x slower
- generators: 25.3 ms +- 0.2 ms -> 26.4 ms +- 0.2 ms: 1.04x slower
- html5lib: 60.5 ms +- 0.6 ms -> 62.7 ms +- 0.5 ms: 1.04x slower
- mako: 10.2 ms +- 0.1 ms -> 10.6 ms +- 0.1 ms: 1.04x slower
- go: 110 ms +- 1 ms -> 114 ms +- 1 ms: 1.03x slower
- logging_format: 5.82 us +- 0.13 us -> 6.00 us +- 0.13 us: 1.03x slower
- logging_simple: 5.25 us +- 0.07 us -> 5.41 us +- 0.12 us: 1.03x slower
- comprehensions: 15.3 us +- 0.1 us -> 15.8 us +- 0.1 us: 1.03x slower
- async_generators: 376 ms +- 4 ms -> 387 ms +- 4 ms: 1.03x slower
- genshi_xml: 46.1 ms +- 0.6 ms -> 47.3 ms +- 0.5 ms: 1.03x slower
- scimark_sparse_mat_mult: 4.21 ms +- 0.11 ms -> 4.32 ms +- 0.07 ms: 1.03x slower
- chaos: 58.0 ms +- 0.8 ms -> 59.4 ms +- 0.8 ms: 1.02x slower
- spectral_norm: 93.0 ms +- 0.8 ms -> 95.1 ms +- 0.9 ms: 1.02x slower
- scimark_sor: 114 ms +- 1 ms -> 116 ms +- 1 ms: 1.02x slower
- coroutines: 20.4 ms +- 0.3 ms -> 20.8 ms +- 0.3 ms: 1.02x slower
- richards: 43.6 ms +- 0.4 ms -> 44.4 ms +- 0.4 ms: 1.02x slower
- fannkuch: 372 ms +- 3 ms -> 378 ms +- 3 ms: 1.02x slower
- richards_super: 49.7 ms +- 0.4 ms -> 50.6 ms +- 0.5 ms: 1.02x slower
- json_dumps: 10.1 ms +- 0.2 ms -> 10.2 ms +- 0.3 ms: 1.02x slower
- regex_dna: 147 ms +- 3 ms -> 149 ms +- 3 ms: 1.02x slower
- genshi_text: 19.6 ms +- 0.2 ms -> 19.9 ms +- 0.2 ms: 1.02x slower
- asyncio_tcp: 429 ms +- 4 ms -> 436 ms +- 5 ms: 1.01x slower
- meteor_contest: 89.6 ms +- 0.4 ms -> 90.7 ms +- 0.5 ms: 1.01x slower
- float: 66.4 ms +- 1.1 ms -> 67.2 ms +- 1.1 ms: 1.01x slower
- typing_runtime_protocols: 153 us +- 3 us -> 154 us +- 3 us: 1.01x slower
- pprint_safe_repr: 667 ms +- 6 ms -> 674 ms +- 7 ms: 1.01x slower
- tomli_loads: 1.93 sec +- 0.02 sec -> 1.95 sec +- 0.02 sec: 1.01x slower
- hexiom: 5.63 ms +- 0.06 ms -> 5.68 ms +- 0.04 ms: 1.01x slower
- xml_etree_process: 54.6 ms +- 0.4 ms -> 55.0 ms +- 0.4 ms: 1.01x slower
- 2to3: 230 ms +- 1 ms -> 232 ms +- 1 ms: 1.01x slower
- regex_compile: 118 ms +- 1 ms -> 119 ms +- 1 ms: 1.01x slower
- xml_etree_iterparse: 78.7 ms +- 0.6 ms -> 79.2 ms +- 0.7 ms: 1.01x slower
- pprint_pformat: 1.38 sec +- 0.01 sec -> 1.39 sec +- 0.01 sec: 1.00x slower
- mdp: 2.33 sec +- 0.02 sec -> 2.34 sec +- 0.01 sec: 1.00x slower
- raytrace: 256 ms +- 3 ms -> 257 ms +- 3 ms: 1.00x slower
- asyncio_tcp_ssl: 1.48 sec +- 0.01 sec -> 1.48 sec +- 0.01 sec: 1.00x slower
- docutils: 2.22 sec +- 0.01 sec -> 2.23 sec +- 0.01 sec: 1.00x slower
- scimark_monte_carlo: 59.3 ms +- 0.4 ms -> 59.5 ms +- 0.5 ms: 1.00x slower

Faster (22):
- regex_effbot: 2.47 ms +- 0.03 ms -> 2.31 ms +- 0.05 ms: 1.07x faster
- scimark_lu: 112 ms +- 1 ms -> 105 ms +- 1 ms: 1.06x faster
- regex_v8: 22.1 ms +- 0.2 ms -> 20.8 ms +- 0.2 ms: 1.06x faster
- json_loads: 22.9 us +- 0.2 us -> 21.7 us +- 0.2 us: 1.06x faster
- logging_silent: 107 ns +- 1 ns -> 102 ns +- 1 ns: 1.05x faster
- unpack_sequence: 37.4 ns +- 1.9 ns -> 36.4 ns +- 0.8 ns: 1.03x faster
- deepcopy_reduce: 2.60 us +- 0.03 us -> 2.55 us +- 0.03 us: 1.02x faster
- unpickle: 12.1 us +- 0.2 us -> 11.9 us +- 0.1 us: 1.02x faster
- deepcopy_memo: 29.6 us +- 0.2 us -> 29.1 us +- 0.3 us: 1.02x faster
- pidigits: 160 ms +- 1 ms -> 158 ms +- 1 ms: 1.01x faster
- gc_traversal: 2.78 ms +- 0.06 ms -> 2.74 ms +- 0.02 ms: 1.01x faster
- create_gc_cycles: 954 us +- 5 us -> 944 us +- 6 us: 1.01x faster
- pathlib: 19.2 ms +- 0.1 ms -> 19.0 ms +- 0.1 ms: 1.01x faster
- bench_thread_pool: 936 us +- 39 us -> 926 us +- 35 us: 1.01x faster
- deepcopy: 250 us +- 2 us -> 248 us +- 2 us: 1.01x faster
- pickle_pure_python: 298 us +- 3 us -> 295 us +- 2 us: 1.01x faster
- unpickle_pure_python: 210 us +- 1 us -> 209 us +- 2 us: 1.01x faster
- unpickle_list: 4.36 us +- 0.07 us -> 4.33 us +- 0.09 us: 1.01x faster
- nbody: 88.3 ms +- 0.6 ms -> 87.8 ms +- 0.6 ms: 1.01x faster
- python_startup: 9.30 ms +- 0.03 ms -> 9.27 ms +- 0.03 ms: 1.00x faster
- python_startup_no_site: 6.52 ms +- 0.03 ms -> 6.49 ms +- 0.03 ms: 1.00x faster
- nqueens: 74.4 ms +- 0.7 ms -> 74.2 ms +- 0.6 ms: 1.00x faster

Benchmark hidden because not significant (10): asyncio_websockets, bench_mp_pool, crypto_pyaes, dulwich_log, pickle, pickle_dict, pickle_list, telco, xml_etree_parse, xml_etree_generate

Geometric mean: 1.02x slower

    cdsplit

    All benchmarks:
===============

Slower (41):
- sqlglot_normalize: 102 ms +- 1 ms -> 283 ms +- 2 ms: 2.78x slower
- deltablue: 2.91 ms +- 0.03 ms -> 3.20 ms +- 0.05 ms: 1.10x slower
- pyflate: 380 ms +- 3 ms -> 411 ms +- 4 ms: 1.08x slower
- scimark_fft: 313 ms +- 3 ms -> 329 ms +- 4 ms: 1.05x slower
- generators: 25.3 ms +- 0.2 ms -> 26.4 ms +- 0.2 ms: 1.04x slower
- html5lib: 60.5 ms +- 0.6 ms -> 62.7 ms +- 0.5 ms: 1.04x slower
- mako: 10.2 ms +- 0.1 ms -> 10.6 ms +- 0.1 ms: 1.04x slower
- go: 110 ms +- 1 ms -> 114 ms +- 1 ms: 1.03x slower
- logging_format: 5.82 us +- 0.13 us -> 6.00 us +- 0.13 us: 1.03x slower
- logging_simple: 5.25 us +- 0.07 us -> 5.41 us +- 0.12 us: 1.03x slower
- comprehensions: 15.3 us +- 0.1 us -> 15.8 us +- 0.1 us: 1.03x slower
- async_generators: 376 ms +- 4 ms -> 387 ms +- 4 ms: 1.03x slower
- genshi_xml: 46.1 ms +- 0.6 ms -> 47.3 ms +- 0.5 ms: 1.03x slower
- scimark_sparse_mat_mult: 4.21 ms +- 0.11 ms -> 4.32 ms +- 0.07 ms: 1.03x slower
- chaos: 58.0 ms +- 0.8 ms -> 59.4 ms +- 0.8 ms: 1.02x slower
- spectral_norm: 93.0 ms +- 0.8 ms -> 95.1 ms +- 0.9 ms: 1.02x slower
- scimark_sor: 114 ms +- 1 ms -> 116 ms +- 1 ms: 1.02x slower
- coroutines: 20.4 ms +- 0.3 ms -> 20.8 ms +- 0.3 ms: 1.02x slower
- richards: 43.6 ms +- 0.4 ms -> 44.4 ms +- 0.4 ms: 1.02x slower
- fannkuch: 372 ms +- 3 ms -> 378 ms +- 3 ms: 1.02x slower
- richards_super: 49.7 ms +- 0.4 ms -> 50.6 ms +- 0.5 ms: 1.02x slower
- json_dumps: 10.1 ms +- 0.2 ms -> 10.2 ms +- 0.3 ms: 1.02x slower
- regex_dna: 147 ms +- 3 ms -> 149 ms +- 3 ms: 1.02x slower
- genshi_text: 19.6 ms +- 0.2 ms -> 19.9 ms +- 0.2 ms: 1.02x slower
- asyncio_tcp: 429 ms +- 4 ms -> 436 ms +- 5 ms: 1.01x slower
- meteor_contest: 89.6 ms +- 0.4 ms -> 90.7 ms +- 0.5 ms: 1.01x slower
- float: 66.4 ms +- 1.1 ms -> 67.2 ms +- 1.1 ms: 1.01x slower
- typing_runtime_protocols: 153 us +- 3 us -> 154 us +- 3 us: 1.01x slower
- pprint_safe_repr: 667 ms +- 6 ms -> 674 ms +- 7 ms: 1.01x slower
- tomli_loads: 1.93 sec +- 0.02 sec -> 1.95 sec +- 0.02 sec: 1.01x slower
- hexiom: 5.63 ms +- 0.06 ms -> 5.68 ms +- 0.04 ms: 1.01x slower
- xml_etree_process: 54.6 ms +- 0.4 ms -> 55.0 ms +- 0.4 ms: 1.01x slower
- 2to3: 230 ms +- 1 ms -> 232 ms +- 1 ms: 1.01x slower
- regex_compile: 118 ms +- 1 ms -> 119 ms +- 1 ms: 1.01x slower
- xml_etree_iterparse: 78.7 ms +- 0.6 ms -> 79.2 ms +- 0.7 ms: 1.01x slower
- pprint_pformat: 1.38 sec +- 0.01 sec -> 1.39 sec +- 0.01 sec: 1.00x slower
- mdp: 2.33 sec +- 0.02 sec -> 2.34 sec +- 0.01 sec: 1.00x slower
- raytrace: 256 ms +- 3 ms -> 257 ms +- 3 ms: 1.00x slower
- asyncio_tcp_ssl: 1.48 sec +- 0.01 sec -> 1.48 sec +- 0.01 sec: 1.00x slower
- docutils: 2.22 sec +- 0.01 sec -> 2.23 sec +- 0.01 sec: 1.00x slower
- scimark_monte_carlo: 59.3 ms +- 0.4 ms -> 59.5 ms +- 0.5 ms: 1.00x slower

Faster (22):
- regex_effbot: 2.47 ms +- 0.03 ms -> 2.31 ms +- 0.05 ms: 1.07x faster
- scimark_lu: 112 ms +- 1 ms -> 105 ms +- 1 ms: 1.06x faster
- regex_v8: 22.1 ms +- 0.2 ms -> 20.8 ms +- 0.2 ms: 1.06x faster
- json_loads: 22.9 us +- 0.2 us -> 21.7 us +- 0.2 us: 1.06x faster
- logging_silent: 107 ns +- 1 ns -> 102 ns +- 1 ns: 1.05x faster
- unpack_sequence: 37.4 ns +- 1.9 ns -> 36.4 ns +- 0.8 ns: 1.03x faster
- deepcopy_reduce: 2.60 us +- 0.03 us -> 2.55 us +- 0.03 us: 1.02x faster
- unpickle: 12.1 us +- 0.2 us -> 11.9 us +- 0.1 us: 1.02x faster
- deepcopy_memo: 29.6 us +- 0.2 us -> 29.1 us +- 0.3 us: 1.02x faster
- pidigits: 160 ms +- 1 ms -> 158 ms +- 1 ms: 1.01x faster
- gc_traversal: 2.78 ms +- 0.06 ms -> 2.74 ms +- 0.02 ms: 1.01x faster
- create_gc_cycles: 954 us +- 5 us -> 944 us +- 6 us: 1.01x faster
- pathlib: 19.2 ms +- 0.1 ms -> 19.0 ms +- 0.1 ms: 1.01x faster
- bench_thread_pool: 936 us +- 39 us -> 926 us +- 35 us: 1.01x faster
- deepcopy: 250 us +- 2 us -> 248 us +- 2 us: 1.01x faster
- pickle_pure_python: 298 us +- 3 us -> 295 us +- 2 us: 1.01x faster
- unpickle_pure_python: 210 us +- 1 us -> 209 us +- 2 us: 1.01x faster
- unpickle_list: 4.36 us +- 0.07 us -> 4.33 us +- 0.09 us: 1.01x faster
- nbody: 88.3 ms +- 0.6 ms -> 87.8 ms +- 0.6 ms: 1.01x faster
- python_startup: 9.30 ms +- 0.03 ms -> 9.27 ms +- 0.03 ms: 1.00x faster
- python_startup_no_site: 6.52 ms +- 0.03 ms -> 6.49 ms +- 0.03 ms: 1.00x faster
- nqueens: 74.4 ms +- 0.7 ms -> 74.2 ms +- 0.6 ms: 1.00x faster

Benchmark hidden because not significant (10): asyncio_websockets, bench_mp_pool, crypto_pyaes, dulwich_log, pickle, pickle_dict, pickle_list, telco, xml_etree_parse, xml_etree_generate

Geometric mean: 1.02x slower

    split-all-cold

    All benchmarks:
===============

Slower (30):
- sqlglot_normalize: 102 ms +- 1 ms -> 283 ms +- 4 ms: 2.78x slower
- bench_mp_pool: 26.1 ms +- 0.4 ms -> 31.0 ms +- 8.7 ms: 1.19x slower
- mako: 10.2 ms +- 0.1 ms -> 11.0 ms +- 0.2 ms: 1.08x slower
- pyflate: 380 ms +- 3 ms -> 407 ms +- 3 ms: 1.07x slower
- create_gc_cycles: 954 us +- 5 us -> 1.01 ms +- 0.01 ms: 1.06x slower
- unpack_sequence: 37.4 ns +- 1.9 ns -> 39.5 ns +- 2.6 ns: 1.06x slower
- regex_dna: 147 ms +- 3 ms -> 152 ms +- 3 ms: 1.04x slower
- scimark_sparse_mat_mult: 4.21 ms +- 0.11 ms -> 4.36 ms +- 0.08 ms: 1.04x slower
- unpickle_list: 4.36 us +- 0.07 us -> 4.50 us +- 0.10 us: 1.03x slower
- telco: 6.60 ms +- 0.10 ms -> 6.81 ms +- 0.07 ms: 1.03x slower
- scimark_sor: 114 ms +- 1 ms -> 117 ms +- 1 ms: 1.03x slower
- generators: 25.3 ms +- 0.2 ms -> 26.1 ms +- 0.3 ms: 1.03x slower
- gc_traversal: 2.78 ms +- 0.06 ms -> 2.85 ms +- 0.06 ms: 1.03x slower
- raytrace: 256 ms +- 3 ms -> 262 ms +- 2 ms: 1.02x slower
- deltablue: 2.91 ms +- 0.03 ms -> 2.98 ms +- 0.02 ms: 1.02x slower
- xml_etree_iterparse: 78.7 ms +- 0.6 ms -> 80.3 ms +- 0.5 ms: 1.02x slower
- scimark_lu: 112 ms +- 1 ms -> 114 ms +- 2 ms: 1.02x slower
- genshi_text: 19.6 ms +- 0.2 ms -> 20.0 ms +- 0.2 ms: 1.02x slower
- async_generators: 376 ms +- 4 ms -> 383 ms +- 4 ms: 1.02x slower
- comprehensions: 15.3 us +- 0.1 us -> 15.6 us +- 0.1 us: 1.02x slower
- richards_super: 49.7 ms +- 0.4 ms -> 50.4 ms +- 0.5 ms: 1.01x slower
- xml_etree_parse: 114 ms +- 1 ms -> 116 ms +- 1 ms: 1.01x slower
- scimark_fft: 313 ms +- 3 ms -> 317 ms +- 3 ms: 1.01x slower
- xml_etree_generate: 78.3 ms +- 0.5 ms -> 79.2 ms +- 0.6 ms: 1.01x slower
- tomli_loads: 1.93 sec +- 0.02 sec -> 1.94 sec +- 0.02 sec: 1.01x slower
- go: 110 ms +- 1 ms -> 111 ms +- 1 ms: 1.01x slower
- xml_etree_process: 54.6 ms +- 0.4 ms -> 55.0 ms +- 0.5 ms: 1.01x slower
- html5lib: 60.5 ms +- 0.6 ms -> 60.8 ms +- 0.8 ms: 1.01x slower
- richards: 43.6 ms +- 0.4 ms -> 43.8 ms +- 0.4 ms: 1.00x slower
- 2to3: 230 ms +- 1 ms -> 230 ms +- 1 ms: 1.00x slower

Faster (29):
- regex_v8: 22.1 ms +- 0.2 ms -> 20.7 ms +- 0.2 ms: 1.06x faster
- logging_silent: 107 ns +- 1 ns -> 101 ns +- 1 ns: 1.06x faster
- spectral_norm: 93.0 ms +- 0.8 ms -> 88.6 ms +- 1.7 ms: 1.05x faster
- json_loads: 22.9 us +- 0.2 us -> 21.8 us +- 0.2 us: 1.05x faster
- coroutines: 20.4 ms +- 0.3 ms -> 19.5 ms +- 0.4 ms: 1.05x faster
- regex_effbot: 2.47 ms +- 0.03 ms -> 2.36 ms +- 0.05 ms: 1.05x faster
- hexiom: 5.63 ms +- 0.06 ms -> 5.46 ms +- 0.04 ms: 1.03x faster
- deepcopy_memo: 29.6 us +- 0.2 us -> 28.8 us +- 0.3 us: 1.03x faster
- deepcopy_reduce: 2.60 us +- 0.03 us -> 2.54 us +- 0.03 us: 1.02x faster
- pickle: 11.1 us +- 0.4 us -> 10.8 us +- 0.1 us: 1.02x faster
- unpickle: 12.1 us +- 0.2 us -> 11.8 us +- 0.2 us: 1.02x faster
- logging_format: 5.82 us +- 0.13 us -> 5.71 us +- 0.08 us: 1.02x faster
- nbody: 88.3 ms +- 0.6 ms -> 86.6 ms +- 0.6 ms: 1.02x faster
- pathlib: 19.2 ms +- 0.1 ms -> 18.8 ms +- 0.1 ms: 1.02x faster
- bench_thread_pool: 936 us +- 39 us -> 921 us +- 34 us: 1.02x faster
- deepcopy: 250 us +- 2 us -> 247 us +- 3 us: 1.01x faster
- nqueens: 74.4 ms +- 0.7 ms -> 73.4 ms +- 0.5 ms: 1.01x faster
- meteor_contest: 89.6 ms +- 0.4 ms -> 88.5 ms +- 0.5 ms: 1.01x faster
- dulwich_log: 60.0 ms +- 0.3 ms -> 59.2 ms +- 0.3 ms: 1.01x faster
- pprint_pformat: 1.38 sec +- 0.01 sec -> 1.37 sec +- 0.01 sec: 1.01x faster
- fannkuch: 372 ms +- 3 ms -> 368 ms +- 4 ms: 1.01x faster
- chaos: 58.0 ms +- 0.8 ms -> 57.5 ms +- 0.5 ms: 1.01x faster
- pprint_safe_repr: 667 ms +- 6 ms -> 661 ms +- 5 ms: 1.01x faster
- regex_compile: 118 ms +- 1 ms -> 117 ms +- 1 ms: 1.01x faster
- pickle_pure_python: 298 us +- 3 us -> 296 us +- 3 us: 1.00x faster
- crypto_pyaes: 63.6 ms +- 0.6 ms -> 63.3 ms +- 0.4 ms: 1.00x faster
- mdp: 2.33 sec +- 0.02 sec -> 2.32 sec +- 0.01 sec: 1.00x faster
- pidigits: 160 ms +- 1 ms -> 160 ms +- 1 ms: 1.00x faster
- python_startup: 9.30 ms +- 0.03 ms -> 9.29 ms +- 0.03 ms: 1.00x faster

Benchmark hidden because not significant (14): asyncio_tcp, asyncio_tcp_ssl, asyncio_websockets, docutils, float, genshi_xml, json_dumps, logging_simple, pickle_dict, pickle_list, python_startup_no_site, scimark_monte_carlo, typing_runtime_protocols, unpickle_pure_python

Geometric mean: 1.02x slower

  19. zanieb commented on Feb 9, 2025

    @zanieb
    zanieb
    on Feb 9, 2025
    ContributorAuthor

    I'm not sure what's up with sqlglot_normalize

  20. zanieb commented on Feb 10, 2025

    @zanieb
    zanieb
    on Feb 10, 2025
    ContributorAuthor

    I did another benchmark with all three of the aforementioned flags added. It looks like together, there's a more consistent speedup.

    https://gist.github.com/zanieb/2e043b5f1f415a0b1e98fcf72465c0fa

    All benchmarks:
===============

Slower (10):
- sqlglot_normalize: 102 ms +- 1 ms -> 275 ms +- 3 ms: 2.70x slower
- python_startup_no_site: 6.52 ms +- 0.03 ms -> 8.53 ms +- 0.04 ms: 1.31x slower
- python_startup: 9.30 ms +- 0.03 ms -> 11.2 ms +- 0.0 ms: 1.21x slower
- bench_mp_pool: 26.1 ms +- 0.4 ms -> 28.3 ms +- 0.6 ms: 1.08x slower
- create_gc_cycles: 954 us +- 5 us -> 1.00 ms +- 0.01 ms: 1.05x slower
- regex_dna: 147 ms +- 3 ms -> 153 ms +- 1 ms: 1.04x slower
- gc_traversal: 2.78 ms +- 0.06 ms -> 2.84 ms +- 0.03 ms: 1.02x slower
- json_loads: 22.9 us +- 0.2 us -> 23.1 us +- 0.2 us: 1.01x slower
- xml_etree_iterparse: 78.7 ms +- 0.6 ms -> 79.0 ms +- 0.9 ms: 1.00x slower
- xml_etree_parse: 114 ms +- 1 ms -> 114 ms +- 1 ms: 1.00x slower

Faster (58):
- unpack_sequence: 37.4 ns +- 1.9 ns -> 33.1 ns +- 0.5 ns: 1.13x faster
- deltablue: 2.91 ms +- 0.03 ms -> 2.63 ms +- 0.02 ms: 1.11x faster
- nbody: 88.3 ms +- 0.6 ms -> 80.1 ms +- 0.8 ms: 1.10x faster
- logging_silent: 107 ns +- 1 ns -> 97.3 ns +- 1.8 ns: 1.10x faster
- hexiom: 5.63 ms +- 0.06 ms -> 5.13 ms +- 0.05 ms: 1.10x faster
- async_generators: 376 ms +- 4 ms -> 345 ms +- 3 ms: 1.09x faster
- go: 110 ms +- 1 ms -> 101 ms +- 1 ms: 1.09x faster
- chaos: 58.0 ms +- 0.8 ms -> 53.5 ms +- 0.6 ms: 1.08x faster
- regex_compile: 118 ms +- 1 ms -> 110 ms +- 1 ms: 1.08x faster
- unpickle_pure_python: 210 us +- 1 us -> 195 us +- 1 us: 1.08x faster
- pyflate: 380 ms +- 3 ms -> 354 ms +- 4 ms: 1.07x faster
- genshi_text: 19.6 ms +- 0.2 ms -> 18.5 ms +- 0.2 ms: 1.06x faster
- float: 66.4 ms +- 1.1 ms -> 62.5 ms +- 0.6 ms: 1.06x faster
- comprehensions: 15.3 us +- 0.1 us -> 14.4 us +- 0.3 us: 1.06x faster
- deepcopy: 250 us +- 2 us -> 236 us +- 2 us: 1.06x faster
- regex_effbot: 2.47 ms +- 0.03 ms -> 2.33 ms +- 0.04 ms: 1.06x faster
- scimark_fft: 313 ms +- 3 ms -> 297 ms +- 9 ms: 1.06x faster
- scimark_monte_carlo: 59.3 ms +- 0.4 ms -> 56.3 ms +- 0.9 ms: 1.05x faster
- unpickle_list: 4.36 us +- 0.07 us -> 4.14 us +- 0.10 us: 1.05x faster
- pickle_pure_python: 298 us +- 3 us -> 283 us +- 3 us: 1.05x faster
- richards: 43.6 ms +- 0.4 ms -> 41.5 ms +- 0.6 ms: 1.05x faster
- deepcopy_reduce: 2.60 us +- 0.03 us -> 2.48 us +- 0.03 us: 1.05x faster
- telco: 6.60 ms +- 0.10 ms -> 6.30 ms +- 0.11 ms: 1.05x faster
- json_dumps: 10.1 ms +- 0.2 ms -> 9.60 ms +- 0.10 ms: 1.05x faster
- logging_simple: 5.25 us +- 0.07 us -> 5.02 us +- 0.06 us: 1.05x faster
- nqueens: 74.4 ms +- 0.7 ms -> 71.1 ms +- 0.8 ms: 1.05x faster
- genshi_xml: 46.1 ms +- 0.6 ms -> 44.1 ms +- 0.4 ms: 1.05x faster
- meteor_contest: 89.6 ms +- 0.4 ms -> 85.8 ms +- 0.4 ms: 1.04x faster
- spectral_norm: 93.0 ms +- 0.8 ms -> 89.1 ms +- 0.7 ms: 1.04x faster
- xml_etree_process: 54.6 ms +- 0.4 ms -> 52.3 ms +- 0.4 ms: 1.04x faster
- mdp: 2.33 sec +- 0.02 sec -> 2.23 sec +- 0.03 sec: 1.04x faster
- dulwich_log: 60.0 ms +- 0.3 ms -> 57.6 ms +- 0.4 ms: 1.04x faster
- pprint_pformat: 1.38 sec +- 0.01 sec -> 1.33 sec +- 0.01 sec: 1.04x faster
- scimark_lu: 112 ms +- 1 ms -> 107 ms +- 1 ms: 1.04x faster
- coroutines: 20.4 ms +- 0.3 ms -> 19.6 ms +- 0.4 ms: 1.04x faster
- scimark_sparse_mat_mult: 4.21 ms +- 0.11 ms -> 4.05 ms +- 0.12 ms: 1.04x faster
- tomli_loads: 1.93 sec +- 0.02 sec -> 1.85 sec +- 0.02 sec: 1.04x faster
- scimark_sor: 114 ms +- 1 ms -> 110 ms +- 1 ms: 1.04x faster
- xml_etree_generate: 78.3 ms +- 0.5 ms -> 75.7 ms +- 0.6 ms: 1.04x faster
- pprint_safe_repr: 667 ms +- 6 ms -> 646 ms +- 7 ms: 1.03x faster
- 2to3: 230 ms +- 1 ms -> 223 ms +- 1 ms: 1.03x faster
- regex_v8: 22.1 ms +- 0.2 ms -> 21.4 ms +- 0.1 ms: 1.03x faster
- richards_super: 49.7 ms +- 0.4 ms -> 48.3 ms +- 0.5 ms: 1.03x faster
- mako: 10.2 ms +- 0.1 ms -> 9.92 ms +- 0.07 ms: 1.03x faster
- deepcopy_memo: 29.6 us +- 0.2 us -> 28.8 us +- 0.3 us: 1.03x faster
- bench_thread_pool: 936 us +- 39 us -> 913 us +- 38 us: 1.02x faster
- generators: 25.3 ms +- 0.2 ms -> 24.7 ms +- 0.2 ms: 1.02x faster
- raytrace: 256 ms +- 3 ms -> 250 ms +- 2 ms: 1.02x faster
- asyncio_websockets: 508 ms +- 24 ms -> 496 ms +- 2 ms: 1.02x faster
- logging_format: 5.82 us +- 0.13 us -> 5.69 us +- 0.11 us: 1.02x faster
- pidigits: 160 ms +- 1 ms -> 158 ms +- 1 ms: 1.01x faster
- docutils: 2.22 sec +- 0.01 sec -> 2.19 sec +- 0.02 sec: 1.01x faster
- pickle: 11.1 us +- 0.4 us -> 10.9 us +- 0.1 us: 1.01x faster
- typing_runtime_protocols: 153 us +- 3 us -> 151 us +- 6 us: 1.01x faster
- pathlib: 19.2 ms +- 0.1 ms -> 19.0 ms +- 0.1 ms: 1.01x faster
- pickle_list: 3.84 us +- 0.07 us -> 3.82 us +- 0.06 us: 1.00x faster
- asyncio_tcp: 429 ms +- 4 ms -> 428 ms +- 3 ms: 1.00x faster
- asyncio_tcp_ssl: 1.48 sec +- 0.01 sec -> 1.47 sec +- 0.01 sec: 1.00x faster

Benchmark hidden because not significant (5): crypto_pyaes, fannkuch, html5lib, pickle_dict, unpickle

Geometric mean: 1.01x faster
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

buildThe build process and cross-buildperformancePerformance or resource usagetype-featureA feature request or enhancement

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions