Repository navigation
Speed up RunEnd filter with a rank directory - #10979
robert3005 wants to merge 4 commits into
Conversation
Replace the per-run `BitBuffer::count_range` in `filter_run_end_primitive` with a word-level rank directory: each filtered run end is the mask rank at the source run end, which costs one load and one popcount per run. Ranks are written for every run and the retained ends are compacted by the primitive filter with the same mask that selects the run values. The x86-64 baseline lacks POPCNT, so the loop is compiled under `#[target_feature(enable = "popcnt")]` behind runtime detection. The retain bitmap is packed by hand because `collect_bool` does not inline into that caller. Add a regression test against a per-row reference with offsets and an unaligned mask. Signed-off-by: Robert Kruszewski <github@robertk.io>
Give the POPCNT and portable rank loops distinct instantiations through a const parameter. With one caller per closure type LLVM inlines BitBuffer::collect_bool, so the hand-packed retain bitmap is no longer needed, and the inlined fill and SSE2 pack are about 10% faster on the run_end_filter benchmark. Signed-off-by: Robert Kruszewski <github@robertk.io>
Replace the hand-rolled POPCNT detection and #[target_feature] wrapper with fearless_simd's dispatch! and a #[simd] body, matching count_ones and select in vortex-buffer. Levels from SSE4.2 up enable POPCNT, and each level gets its own instantiation, so BitBuffer::collect_bool still inlines without the const-parameter workaround. The AVX2 level also enables BMI, which brings another 10-15% on the run_end_filter benchmark. Signed-off-by: Robert Kruszewski <github@robertk.io>
Write each rank at the current retained count and advance the count only for a retained run. This removes the second pass that filtered the ranks through PrimitiveArray, and its extra allocation. Also remove the three bounds checks from the hot loop, write into spare capacity instead of a zeroed buffer, and allocate the ends through the session allocator. run_end_filter on an Apple M4 Max is 26-41% faster than before. Of this, the unchecked indexing gives 10-16%. Signed-off-by: Robert Kruszewski <github@robertk.io>
7315c6c to
1351c90
Compare
2c08176 to
65b4908
Compare
Merging this PR will regress 8 benchmarks2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing |
Summary
Changes