Repository navigation
Read integer validity a word per chunk, and fill nulls for runs - #10342
Conversation
The compressor benchmarks are excluded from CodSpeed, and none covers nullable arrays. Add one benchmark of integer statistics over a non-null array, and one over the same array with one value in ten null, and run the compressor benchmarks in a CodSpeed shard. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018J6qahqrVFb4UCPkwUzY11
Integer statistics sliced the validity per chunk of 64 values to count its set bits, then tested every value's bit. With nulls spread out, almost every chunk took that path. Read one validity word per chunk instead. Without distinct values, fill the nulls of a chunk with the valid value before them, which adds no runs, and count the runs with the branch-free loop of non-null chunks. With distinct values, push only the set bits. Statistics of 64Ki u32 with one value in ten null take 207 us instead of 517 us. Without nulls they are unchanged. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018J6qahqrVFb4UCPkwUzY11
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 26.72%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | integer_stats_nullable |
2.6 ms | 1.8 ms | +43.25% |
| ⚡ | WallTime | bitpack_blocked_compress_avx2 |
7.6 µs | 6.8 µs | +12.09% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing ji/int-stats-null-words (f6c7aa9) with develop (d9ad4cf)
Footnotes
-
518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
| } | ||
| } | ||
| fn forward_fill<T: Copy>(values: &[T; 64], valid: u64, prev: T) -> [T; 64] { | ||
| let mut filled = *values; |
There was a problem hiding this comment.
Do we need to make this copy if we're just going to immediately take a reference to it in inner_loop_nonnull_impl?
There was a problem hiding this comment.
This fills in the invalid spots with the value and then we immediately reduce it, we need to make a copy here to fill
There was a problem hiding this comment.
i guess I was asking if we could fuse and avoid the copy, but I doubt it changes much
connortsui20
left a comment
There was a problem hiding this comment.
Definitely an improvement, though left a small comment
Keep this branch's integer.rs, which already contains #10342's change. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Stacked on #10341, which adds the benchmarks.
Integer statistics with nulls sliced the validity
BitBufferfor every chunk of 64 values to count its set bits, then tested each value's bit with a branch. With nulls spread out, almost no chunk is fully valid (0.9⁶⁴ ≈ 0.1% at 10% nulls), so nearly every chunk took that slow path.Changes
In
vortex-compressor/src/stats/integer.rs:u64word per chunk instead of slicing a bit buffer. A word of zeros is skipped, and a full word takes the non-null loop as before.Min/max is unchanged: it is still computed by the
min_maxaggregate before this loop.test_matches_naive_referencenow also covers statistics without distinct values and chunks that are entirely null. Filling the nulls with their own values instead makes 38 of its cases fail.