Repository navigation
Use mimalloc with purging disabled in each e2e benchmark binary - #10490
Conversation
The SQL, random-access, compress, string and lance benchmark binaries picked up mimalloc only implicitly, through a #[global_allocator] declared in the vortex-bench library. Declare it explicitly in each benchmark binary's main (and in data-gen) instead, and drop it from the library so a library never chooses the allocator for its users. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB
Merging this PR will degrade performance by 18.16%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | threshold_neon[8] |
1.4 µs | 2.6 µs | -47.25% |
| ❌ | WallTime | threshold_neon[80] |
3.4 µs | 4.5 µs | -23.5% |
| ❌ | WallTime | threshold_neon[64] |
2.5 µs | 3.1 µs | -19.69% |
| ❌ | WallTime | filter_neon[80] |
2.5 µs | 2.9 µs | -14.24% |
| ❌ | WallTime | bitpack_blocked_compress_avx2 |
6.7 µs | 7.6 µs | -11.87% |
| ❌ | WallTime | decode_chunked_neon[i64, 524288] |
13.3 µs | 14.8 µs | -10.65% |
| ⚡ | WallTime | starts_with_avx2[8, VarBin] |
46.6 µs | 41.5 µs | +12.38% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/e2e-bench-mimalloc (090d8e6) with develop (5fc5cb8)
Footnotes
-
12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Polar Signals Profiling ResultsLatest Run
Previous Runs (15)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.980x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.029x ➖, 3↑ 3↓)
duckdb / vortex-compact / ns (1.054x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.991x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 1↓)
duckdb / vortex-compact / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 1↑ 0↓)
datafusion / vortex-compact / ns (0.997x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.987x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.036x ➖, 0↑ 2↓)
duckdb / vortex-compact / ns (1.023x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 1↑ 1↓)
File Size Changes (200 files changed, +0.0% overall, 100↑ 100↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.943x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.076x ➖, 0↑ 2↓)
datafusion / parquet / ns (0.960x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.022x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (1.003x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.021x ➖, 0↑ 0↓)
|
Benchmarks: Appian on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (0.979x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.973x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.009x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (1.008x ➖, 1↑ 2↓)
duckdb / parquet / ns (0.998x ➖, 1↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.999x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 1↑ 4↓)
duckdb / vortex-compact / ns (0.999x ➖, 3↑ 2↓)
duckdb / parquet / ns (0.994x ➖, 3↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.989x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.011x ➖, 0↑ 1↓)
datafusion / vortex-compact / ns (0.981x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.981x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.988x ➖, 0↑ 0↓)
duckdb / vortex-compact / ns (0.987x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ms (0.992x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / % (1.000x ➖, 0↑ 0↓)
|
mimalloc v3 returns freed memory to the OS 1s after it is freed, and that purge work lands in the middle of measured queries. Add vortex_bench::configure_allocator(), which sets mimalloc's purge_delay to -1 (never purge), and call it first thing in every benchmark binary. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB
Benchmarks: Compression 📖Commits: PR vortex / vortex-file-compressed / ns (0.976x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / ratio (0.989x ➖, 0↑ 0↓)
vortex / parquet / ns (0.992x ➖, 0↑ 0↓)
vortex / parquet / bytes (1.000x ➖, 0↑ 0↓)
vortex / arrow-ipc / ns (1.000x ➖, 0↑ 2↓)
vortex / arrow-ipc / bytes (1.000x ➖, 0↑ 0↓)
|
Benchmarks: Random Access 📖Commits: PR How to read Verdict and Engines
vortex / arrow-ipc / ns (1.003x ➖, 3↑ 3↓)
random-access / vortex-file-compressed / ns (0.981x ➖, 0↑ 0↓)
random-access / parquet / ns (0.999x ➖, 0↑ 0↓)
random-access / lance / ns (0.988x ➖, 0↑ 0↓)
|
Benchmarks: Random Access (S3) 📖Commits: PR How to read Verdict and Engines
random-access / vortex-file-compressed / ns (1.020x ➖, 1↑ 1↓)
random-access / parquet / ns (0.975x ➖, 0↑ 0↓)
random-access / lance / ns (0.819x ➖, 5↑ 0↓)
|
I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 917b7f9 I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 85e620b I, Claude <noreply@anthropic.com>, hereby add my Signed-off-by to this commit: 2089d24 Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB
commented
Oct 14, 2026
|
Rust tests (windows-x64) failed on infrastructure: AWS interrupted the Spot instance while it was still compiling, before any test ran. The head commit a209097 is an empty DCO sign-off commit on top of 2089d24, where this job passed. I'll re-run the failed job once this CI run finishes. Generated by Claude Code |
commented
Oct 14, 2026
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.043x ➖, 0↑ 3↓)
datafusion / vortex-compact / ns (1.040x ➖, 1↑ 1↓)
datafusion / parquet / ns (0.937x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (1.148x ➖, 0↑ 5↓)
duckdb / vortex-compact / ns (1.037x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.003x ➖, 1↑ 1↓)
|
commented
Oct 14, 2026
Benchmarks: TPC-H SF=10 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-compact / ns (1.015x ➖, 2↑ 3↓)
datafusion / parquet / ns (0.852x ➖, 2↑ 0↓)
duckdb / vortex-compact / ns (1.083x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.006x ➖, 1↑ 1↓)
|
Resolve the compress-bench main.rs conflict by keeping develop's synchronous main (runtime built explicitly) with the allocator declaration and configure_allocator() call from this branch. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB
commented
Oct 18, 2026
|
The assertion came in with #10458. When - REQUIRE(float(scalar::of(float16_t {0x3C00}).get<float16_t>()) == 1.0f);
+ REQUIRE(float(scalar::of(std::bit_cast<float16_t>(uint16_t {0x3C00})).get<float16_t>()) == 1.0f);Generated by Claude Code |
Summary
The end-to-end benchmark binaries (SQL via DataFusion and DuckDB, random access, compression, string, Lance) only ran with mimalloc implicitly, through a
#[global_allocator]declared in thevortex-benchlibrary. This makes the choice explicit in each benchmark binary, and stops mimalloc from purging freed memory back to the OS during a run: mimalloc v3 purges 1s after a free, and that work lands in the middle of measured queries.Changes
#[global_allocator] static GLOBAL: MiMalloc = MiMalloc;in themain.rsofcompress-bench,datafusion-bench,duckdb-bench,lance-bench,random-access-bench,string-bench, and invortex-bench'sdata-genbinary.vortex-bench/src/lib.rs, so the library no longer picks the allocator for whatever links it.vortex_bench::configure_allocator(), which sets mimalloc'spurge_delayoption to-1(never purge), and call it first thing in each of those binaries. This useslibmimalloc-syswith itsextendedfeature (addsctyto the lockfile).mimallocto each benchmark crate's dependencies and updateCargo.lock.Checks:
cargo clippy --locked --bins -D warningsforvortex-bench,compress-bench,random-access-bench,string-benchanddatafusion-bench; formatted with the pinned nightlyrustfmt.duckdb-benchandlance-benchcould not be built locally (DuckDB source download and thelance-encodingbuild script fail in this environment), so CI covers them.🤖 Generated with Claude Code
https://claude.ai/code/session_0122rGn9fTSV47KLv191MTHB