Repository navigation
Speed up RunEnd filter with a rank directory and POPCNT dispatch - #10575
Conversation
Merging this PR will improve performance by 30.02%1 warning
Performance ChangesTip Curious why performance improved? Comment Comparing Footnotes
|
2bf84fe to
052c8d9
Compare
|
want to stack this on the benchmarks? |
052c8d9 to
65b4908
Compare
|
here #10978 |
010d588 to
100957c
Compare
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.016x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.983x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.989x ➖, 0↑ 0↓)
|
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.984x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.990x ➖, 0↑ 0↓)
|
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.970x ➖, 1↑ 1↓)
|
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.159x ➖, 0↑ 5↓)
datafusion / parquet / ns (1.146x ➖, 0↑ 3↓)
duckdb / vortex-file-compressed / ns (1.135x ➖, 1↑ 7↓)
duckdb / parquet / ns (1.127x ➖, 1↑ 5↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.982x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.947x ➖, 5↑ 0↓)
|
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.954x ➖, 2↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.991x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.968x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.997x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.978x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.978x ➖, 2↑ 2↓)
|
100957c to
11dd5ab
Compare
11dd5ab to
24638a2
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
|
fwiw this matters tpc-ds the most |
connortsui20
left a comment
There was a problem hiding this comment.
this is pretty cool. Though it took a while for me to grok it (and I needed the help of an LLM to understand)
24638a2 to
885dd0a
Compare
Pull Request is not mergeable
Replace the per-run `BitBuffer::count_range` in `filter_run_end_primitive` with a word-level rank directory: each filtered run end is the mask rank at the source run end, which costs one load and one popcount per run. Ranks are written for every run and the retained ends are compacted by the primitive filter with the same mask that selects the run values. The x86-64 baseline lacks POPCNT, so the loop is compiled under `#[target_feature(enable = "popcnt")]` behind runtime detection. The retain bitmap is packed by hand because `collect_bool` does not inline into that caller. Add a regression test against a per-row reference with offsets and an unaligned mask. Signed-off-by: Robert Kruszewski <github@robertk.io>
Give the POPCNT and portable rank loops distinct instantiations through a const parameter. With one caller per closure type LLVM inlines BitBuffer::collect_bool, so the hand-packed retain bitmap is no longer needed, and the inlined fill and SSE2 pack are about 10% faster on the run_end_filter benchmark. Signed-off-by: Robert Kruszewski <github@robertk.io>
Replace the hand-rolled POPCNT detection and #[target_feature] wrapper with fearless_simd's dispatch! and a #[simd] body, matching count_ones and select in vortex-buffer. Levels from SSE4.2 up enable POPCNT, and each level gets its own instantiation, so BitBuffer::collect_bool still inlines without the const-parameter workaround. The AVX2 level also enables BMI, which brings another 10-15% on the run_end_filter benchmark. Signed-off-by: Robert Kruszewski <github@robertk.io>
Write each rank at the current retained count and advance the count only for a retained run. This removes the second pass that filtered the ranks through PrimitiveArray, and its extra allocation. Also remove the three bounds checks from the hot loop, write into spare capacity instead of a zeroed buffer, and allocate the ends through the session allocator. run_end_filter on an Apple M4 Max is 26-41% faster than before. Of this, the unchecked indexing gives 10-16%. Signed-off-by: Robert Kruszewski <github@robertk.io>
Add BitRankDirectory and BitBuffer::rank_directory to vortex-buffer so other code can make constant-time rank queries over a bitmap. The RunEnd filter now uses this type. Expand the module documentation of the RunEnd filter with a worked example, as the review suggested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
The `u64: AsPrimitive<R>` bound hides the `usize` implementation during method lookup, so call the `usize` conversion by its full path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
e55315b to
2596b29
Compare
Run ends never decrease, so one walk over the mask words answers every rank query. The directory allocated and filled a 16-byte entry for every mask word before the first query, which made long runs slower than develop. The walk builds no index and still inlines into the POPCNT dispatch body. This removes BitRankDirectory from vortex-buffer, which has no other users. Signed-off-by: Robert Kruszewski <github@robertk.io>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Both tests compared the filtered ends with a per-run count. One test now covers masks that start inside a byte, run ends before and after the slice, and runs that span several mask words. Signed-off-by: Robert Kruszewski <github@robertk.io>
Summary
Speeds up the run-preserving path of the RunEnd filter kernel, most for short runs. The old loop counted the selected rows in each run with a separate
BitBuffer::count_rangecall, and for short runs that call's setup cost more than the counting.Changes
fearless_simd. The x86-64 baseline has nopopcnt, so the loop is dispatched withdispatch!(Level::new(), …). Each level gets its own copy, which also letsBitBuffer::collect_boolinline.