Skip to content

Speed up RunEnd filter with a rank directory - #10979

Closed
robert3005 wants to merge 4 commits into
rk/runend-filter-benchfrom
rk/runend-filter
Closed

robert3005 wants to merge 4 commits into
rk/runend-filter-benchfrom
rk/runend-filter

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Summary

  • Tracking Issue: #000

Changes

@robert3005
robert3005 added this pull request to stack #10980 October 29, 2026 16:28
@robert3005 robert3005 changed the title rk/runend filter Speed up RunEnd filter with a rank directory Oct 29, 2026
Replace the per-run `BitBuffer::count_range` in `filter_run_end_primitive`
with a word-level rank directory: each filtered run end is the mask rank at
the source run end, which costs one load and one popcount per run. Ranks are
written for every run and the retained ends are compacted by the primitive
filter with the same mask that selects the run values.

The x86-64 baseline lacks POPCNT, so the loop is compiled under
`#[target_feature(enable = "popcnt")]` behind runtime detection. The retain
bitmap is packed by hand because `collect_bool` does not inline into that
caller.

Add a regression test against a per-row reference with offsets and an
unaligned mask.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Give the POPCNT and portable rank loops distinct instantiations through a
const parameter. With one caller per closure type LLVM inlines
BitBuffer::collect_bool, so the hand-packed retain bitmap is no longer
needed, and the inlined fill and SSE2 pack are about 10% faster on the
run_end_filter benchmark.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Replace the hand-rolled POPCNT detection and #[target_feature] wrapper with
fearless_simd's dispatch! and a #[simd] body, matching count_ones and select
in vortex-buffer. Levels from SSE4.2 up enable POPCNT, and each level gets
its own instantiation, so BitBuffer::collect_bool still inlines without the
const-parameter workaround. The AVX2 level also enables BMI, which brings
another 10-15% on the run_end_filter benchmark.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Write each rank at the current retained count and advance the count only
for a retained run. This removes the second pass that filtered the ranks
through PrimitiveArray, and its extra allocation. Also remove the three
bounds checks from the hot loop, write into spare capacity instead of a
zeroed buffer, and allocate the ends through the session allocator.

run_end_filter on an Apple M4 Max is 26-41% faster than before. Of this,
the unchecked indexing gives 10-16%.

Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
robert3005 force-pushed the rk/runend-filter-bench branch from 7315c6c to 1351c90 Compare October 29, 2026 16:32
@robert3005
robert3005 removed this pull request from stack #10980 October 29, 2026 16:32
@robert3005

Copy link
Copy Markdown
Contributor Author

Superseded by #10575, now stacked on #10978.

@robert3005 robert3005 closed this Oct 29, 2026
@robert3005
robert3005 deleted the rk/runend-filter branch October 29, 2026 16:32
@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 8 benchmarks

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 5 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 14 improved benchmarks
❌ 8 regressed benchmarks
✅ 2284 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation filter_powerlaw_by_random[10000] 142.2 ns 1,132.3 ns -87.44%
❌ Walltime mul_u64_nonnull_neon 20.4 µs 39.2 µs -48.06%
❌ CPU Simulation bench_compare_varbin[(50000, 2)] 133.2 µs 162.8 µs -18.15%
❌ CPU Simulation bench_compare_varbin[(50000, 64)] 135 µs 164.6 µs -17.98%
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(7500, 10000)] 200.5 µs 228.4 µs -12.18%
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(10000, 10000)] 206.8 µs 234.5 µs -11.84%
❌ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 78.1 µs 88.6 µs -11.83%
❌ CPU Simulation fsl_i32[16, 0.01] 13.1 µs 14.6 µs -10.23%
⚡ CPU Simulation filter_run_end[len=65536_run=4_density=0.5] 2,635.6 µs 408.2 µs ×6.5
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.9] 190.7 µs 47.9 µs ×4
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.5] 190.7 µs 47.9 µs ×4
⚡ CPU Simulation filter_run_end[len=16384_run=16_density=0.1] 193.1 µs 63.2 µs ×3.1
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.9] 57.3 µs 24.9 µs ×2.3
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.5] 56.3 µs 24.7 µs ×2.3
⚡ CPU Simulation filter_run_end[len=4096_run=16_density=0.1] 56 µs 24.7 µs ×2.3
⚡ Walltime threshold_neon[64] 3.7 µs 2.5 µs +46.65%
⚡ Walltime filter_neon[64] 2.5 µs 1.9 µs +32.44%
⚡ Walltime filter_neon[80] 3.6 µs 2.7 µs +32.27%
⚡ Walltime threshold_neon[80] 4.2 µs 3.3 µs +29.42%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowConstant)] 15.9 µs 14.2 µs +11.89%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/runend-filter (65b4908) with rk/runend-filter-bench (1351c90)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant