Skip to content

Add a mergeable HyperLogLog sketch for distinct counts - #11554

Open
vishalkotak wants to merge 1 commit into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-aggregate
Open

vishalkotak wants to merge 1 commit into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-aggregate

Conversation

@vishalkotak

@vishalkotak vishalkotak commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Tracking issue: #9868

This is the first of three PRs for NDV statistics. #11555 adds the aggregate and #11556 wires it into the footer. I put the sketch on its own because its bytes end up in file footers, so the format is the part that's hardest to change later. Golden files pin it down.

I wrote the HyperLogLog here instead of pulling in a crate, since we need to own the format, hash and seed for as long as those files exist, and none of the crates I looked at promise a stable serialized form. Precision tops out at 16, so a column can add at most 64 KiB to a footer. At the default of 12 it's at most 4 KiB, and low-cardinality columns only take a few bytes.

Could a maintainer add the changelog/feature label? I can't set labels from a fork.

I used Claude Code for analysis, debugging, and ideas.

@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-aggregate branch from ab8c575 to f277e23 Compare November 17, 2026 17:15
@vishalkotak vishalkotak changed the title Add an approximate distinct count (NDV) aggregate for footer statistics Add a mergeable HyperLogLog sketch for distinct counts Nov 17, 2026
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-aggregate branch 3 times, most recently from 6ac3c0d to 63873b0 Compare November 17, 2026 18:02
@vishalkotak
vishalkotak marked this pull request as ready for review November 17, 2026 18:07
@joseph-isaacs

joseph-isaacs commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

64 KiB per columns is a lot. Do we really need a full data structure or just the count?
I gave it a go this way #11214, but didn't have time to finish
Do you have any thoughts?

@vishalkotak

Copy link
Copy Markdown
Contributor Author

I hadn't seen #11214, thanks for the pointer.

On size, 64 KiB is only the cap at the highest precision. The default is at most 4 KiB per column, and low-cardinality columns are a few bytes thanks to the sparse encoding. On TPC-H lineitem at SF1, the footer grew by 29 KB on a 15 MB file, about 0.19%.

I went with a sketch because counts don't merge. When I split lineitem into 8 files, summing the per-file counts overcounted by up to 8x, while the merged sketches stayed within 3%. That said, neither DataFusion nor DuckDB merges sketches today, so for planning right now a count does the same job.

One middle ground could be keeping just the count in the footer and storing the sketch separately, read only when something needs to merge. Or I'm happy to help finish #11214 if you'd rather go with counts. Whichever you and @robert3005 prefer works for me.

First part of NDV statistics (vortex-data#9868): the sketch on its own, so its
persisted format can be reviewed before the aggregate that uses it.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants