Skip to content

Count each run's selected rows with one pass over the RunEnd filter mask - #11221

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/eloquent-planck-5el9v5
Nov 4, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/eloquent-planck-5el9v5

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

filter_run_end_primitive called mask.count_range(run_start, run_end) for every run. On default x86_64 builds, which have no static popcnt, each of those calls goes through fearless_simd runtime dispatch, even when the range is only a few bits long. On columns with short runs, such as TPC-DS ss_ticket_number, that per-run overhead was most of the extra CPU in DataFusion scans that apply dynamic filters: 2.5 s of the 2.8 s extra over 150 iterations of q73 at SF1.

Split out of #11215.

Changes

  • Adds MaskRank, which counts the set bits before a position for positions that never decrease, reading each 64-bit mask word once. It builds on Arrow's BitChunks, which already handles masks that start at a bit offset and zeroes the bits past the end in the final partial word.
  • Run ends are strictly increasing, and clamping them to length keeps them non-decreasing. Each run's selected count is the difference between successive ranks, so the whole mask is walked once instead of each run being counted from scratch.
  • Adds an rstest that compares the result against per-run count_range for runs of 1–3 rows. The cases cover mask offsets of 0, 3, 5, 61 and 7, a mask ending inside a word, and a mask ending exactly on a word boundary.

run_end_filter bench, median before → after:

case before after
4K rows, run 16 1.28 µs 0.52 µs
16K rows, run 16 4.75 µs 1.81 µs
64K rows, run 4 65.0 µs 24.0 µs
64K rows, run 256 1.79 µs 0.86 µs

Note: without static popcnt, u64::count_ones compiles to a software popcount, so for very long runs (thousands of rows per run) the old SIMD range count could be slightly faster per word. The absolute cost is negligible in that case.


Generated by Claude Code

`filter_run_end_primitive` counted every run's range of the mask from scratch. On columns with
short runs, such as TPC-DS `ss_ticket_number`, that is one popcount call per run, and it took most
of the extra CPU of DataFusion scans that apply dynamic filters (q73 SF1: 2.5 s of 2.8 s extra over
150 iterations). Run ends increase, so walk the mask's words once with a running rank instead.

`run_end_filter` bench, median before -> after: 4K rows run 16: 1.28 us -> 0.52 us; 16K rows run
16: 4.75 us -> 1.81 us; 64K rows run 4: 65.0 us -> 24.0 us; 64K rows run 256: 1.79 us -> 0.86 us.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Nov 4, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 79.21%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 10 improved benchmarks
✅ 1755 untouched benchmarks
⏩ 130 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation filter_run_end[len=65536_run=4_density=0.5] 2,636.2 µs 900.9 µs ×2.9
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.9] 191 µs 87.2 µs ×2.2
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.5] 191 µs 87.2 µs ×2.2
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.1] 193.2 µs 89.2 µs ×2.2
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.1] 56.5 µs 30.4 µs +85.82%
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.5] 56.7 µs 30.7 µs +84.62%
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.9] 57.4 µs 31.4 µs +82.67%
⚡ CPU Simulation filter_run_end[len=65536_run=256_density=0.5] 65.6 µs 45.8 µs +43.25%
⚡ CPU Simulation random_i8[0.01] 119.3 µs 105.4 µs +13.22%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowConstant)] 16.4 µs 14.9 µs +10.59%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/eloquent-planck-5el9v5 (0b80300) with develop (8b01ab5)

Open in CodSpeed

Footnotes

  1. 130 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs joseph-isaacs added the changelog/performance A performance improvement label Nov 4, 2026
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) November 4, 2026 16:31
@joseph-isaacs
joseph-isaacs merged commit 8dcf8d3 into develop Nov 4, 2026
98 of 99 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/eloquent-planck-5el9v5 branch November 4, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants