Skip to content

Add Delta decode, probe, filter and take benchmarks - #11429

Merged
robert3005 merged 3 commits into
developfrom
rk/delta-decode-bench
Nov 13, 2026
Merged

robert3005 merged 3 commits into
developfrom
rk/delta-decode-bench

Conversation

@robert3005

@robert3005 robert3005 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

This PR adds benchmarks for the Delta encoding. They are split out of #11428 so that its changes can be measured against develop, and #11428 stacks on this PR.

delta_decode measures a u32/u16/u64 Delta array over bit-packed deltas:
decoding it with the deltas packed and unpacked, reading rows through
one-off and repeated probes, and filtering and taking from it compared
with decoding first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
Signed-off-by: Robert Kruszewski <github@robertk.io>
@codspeed

codspeed Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 15.64%

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 1 benchmark with a significant change ran in a different software environment

Its base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the change might come from this difference rather than from the code. Open the benchmark to see what changed.

See affected benchmarks

⚡ 5 improved benchmarks
✅ 1916 untouched benchmarks
🆕 15 new benchmarks
⏩ 50 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation chunked_bool_canonical_into[(1000, 10)] 37.7 µs 29.6 µs +27.59%
⚡ Walltime scalar_subtract_neon 13.8 µs 11.8 µs +16.58%
⚡ Walltime decode_chunked_neon[i64, 524288] 15.4 µs 13.6 µs +13.64%
⚡ Walltime decode_blocked_bitpacked_chunked_neon[u32, 524288] 30 µs 27.1 µs +10.69%
⚡ Walltime decode_blocked_bitpacked_chunked_neon[i64, 524288] 30.5 µs 27.6 µs +10.54%
🆕 CPU Simulation filter[4096] N/A 226.4 µs N/A
🆕 CPU Simulation filter[64] N/A 243.6 µs N/A
🆕 CPU Simulation probe_once N/A 374.5 µs N/A
🆕 CPU Simulation probe_repeated N/A 377.4 µs N/A
🆕 CPU Simulation decode_primitive_deltas N/A 184.8 µs N/A
🆕 CPU Simulation take[128] N/A 235.8 µs N/A
🆕 Walltime decode_bitpacked_deltas_neon[u16, 524288] N/A 343.9 µs N/A
🆕 Walltime decode_bitpacked_deltas_neon[u32, 524288] N/A 189.2 µs N/A
🆕 Walltime decode_bitpacked_deltas_neon[u64, 524288] N/A 131.3 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx2[u16, 524288] N/A 328.2 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx2[u32, 524288] N/A 180.4 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx2[u64, 524288] N/A 113.4 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx512[u16, 524288] N/A 329 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx512[u32, 524288] N/A 178.5 µs N/A
🆕 Walltime decode_bitpacked_deltas_avx512[u64, 524288] N/A 108.7 µs N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/delta-decode-bench (b4f260b) with develop (d81bd4c)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs joseph-isaacs added the changelog/skip Do not list PR in the changelog label Nov 13, 2026

@joseph-isaacs joseph-isaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

48 benchmarks is too much

robert3005 and others added 2 commits November 13, 2026 10:46
Only decode_bitpacked_deltas runs on each walltime CPU-feature leg, since
the build picks its transpose and decode kernels. The probe, filter and
take benchmarks run once, in simulation. decode_unpacked_deltas becomes
decode_primitive_deltas, which decodes a Delta array over primitive
deltas without unpacking them inside the measurement, and the
decode-then-filter and decode-then-take baselines are dropped. That
leaves 15 benchmarks instead of 48.

The six simulation benchmarks go into scripts/codspeed_bench_counts.json,
which the shard planner requires to list every benchmark target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
Signed-off-by: Robert Kruszewski <github@robertk.io>
Simulation runs roughly 13x slower than native, so the probe benchmarks
over a 512 KiB array took ~150 ms. Use 32 KiB arrays for the
simulation-only benchmarks, read 8 rows from a 2048-row array in the
probe benchmarks, and keep 512 KiB only for the walltime decode.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018SWbErHBxmZXBVmmLcAbLD
@robert3005
robert3005 merged commit 25f4349 into develop Nov 13, 2026
77 checks passed
@robert3005
robert3005 deleted the rk/delta-decode-bench branch November 13, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/skip Do not list PR in the changelog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants