Skip to content

Read integer validity a word per chunk, and fill nulls for runs - #10342

Merged
connortsui20 merged 3 commits into
developfrom
ji/int-stats-null-words
Oct 7, 2026
Merged

connortsui20 merged 3 commits into
developfrom
ji/int-stats-null-words

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Stacked on #10341, which adds the benchmarks.

Integer statistics with nulls sliced the validity BitBuffer for every chunk of 64 values to count its set bits, then tested each value's bit with a branch. With nulls spread out, almost no chunk is fully valid (0.9⁶⁴ ≈ 0.1% at 10% nulls), so nearly every chunk took that slow path.

Changes

In vortex-compressor/src/stats/integer.rs:

  • Read the validity one u64 word per chunk instead of slicing a bit buffer. A word of zeros is skipped, and a full word takes the non-null loop as before.
  • Without distinct values, fill each null with the valid value before it. A filled null adds no run, so the chunk takes the same branch-free run count as a non-null chunk.
  • With distinct values, push only the set bits, since a filled null would count as an occurrence.
  • The trailing values are padded into a last chunk whose padding is null. This replaces the separate per-bit remainder loop.

Min/max is unchanged: it is still computed by the min_max aggregate before this loop.

test_matches_naive_reference now also covers statistics without distinct values and chunks that are entirely null. Filling the nulls with their own values instead makes 38 of its cases fail.

joseph-isaacs and others added 2 commits October 6, 2026 15:45
The compressor benchmarks are excluded from CodSpeed, and none covers
nullable arrays. Add one benchmark of integer statistics over a
non-null array, and one over the same array with one value in ten null,
and run the compressor benchmarks in a CodSpeed shard.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018J6qahqrVFb4UCPkwUzY11
Integer statistics sliced the validity per chunk of 64 values to count
its set bits, then tested every value's bit. With nulls spread out,
almost every chunk took that path. Read one validity word per chunk
instead. Without distinct values, fill the nulls of a chunk with the
valid value before them, which adds no runs, and count the runs with
the branch-free loop of non-null chunks. With distinct values, push only
the set bits.

Statistics of 64Ki u32 with one value in ten null take 207 us instead of
517 us. Without nulls they are unchanged.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018J6qahqrVFb4UCPkwUzY11
@joseph-isaacs joseph-isaacs added the changelog/performance A performance improvement label Oct 6, 2026
Base automatically changed from ji/int-stats-bench to develop October 6, 2026 16:03
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed

codspeed Bot commented Oct 6, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 26.72%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
✅ 2107 untouched benchmarks
⏩ 518 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation integer_stats_nullable 2.6 ms 1.8 ms +43.25%
⚡ WallTime bitpack_blocked_compress_avx2 7.6 µs 6.8 µs +12.09%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/int-stats-null-words (f6c7aa9) with develop (d9ad4cf)

Open in CodSpeed

Footnotes

  1. 518 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

}
}
fn forward_fill<T: Copy>(values: &[T; 64], valid: u64, prev: T) -> [T; 64] {
let mut filled = *values;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to make this copy if we're just going to immediately take a reference to it in inner_loop_nonnull_impl?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fills in the invalid spots with the value and then we immediately reduce it, we need to make a copy here to fill

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i guess I was asking if we could fuse and avoid the copy, but I doubt it changes much

@connortsui20 connortsui20 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Definitely an improvement, though left a small comment

@connortsui20
connortsui20 merged commit 438600d into develop Oct 7, 2026
114 checks passed
@connortsui20
connortsui20 deleted the ji/int-stats-null-words branch October 7, 2026 13:40
joseph-isaacs added a commit that referenced this pull request Oct 7, 2026
Keep this branch's integer.rs, which already contains #10342's change.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants