Repository navigation
Add a mergeable HyperLogLog sketch for distinct counts - #11554
vishalkotak wants to merge 1 commit into
Conversation
ab8c575 to
f277e23
Compare
6ac3c0d to
63873b0
Compare
|
64 KiB per columns is a lot. Do we really need a full data structure or just the count? |
|
I hadn't seen #11214, thanks for the pointer. On size, 64 KiB is only the cap at the highest precision. The default is at most 4 KiB per column, and low-cardinality columns are a few bytes thanks to the sparse encoding. On TPC-H I went with a sketch because counts don't merge. When I split One middle ground could be keeping just the count in the footer and storing the sketch separately, read only when something needs to merge. Or I'm happy to help finish #11214 if you'd rather go with counts. Whichever you and @robert3005 prefer works for me. |
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its persisted format can be reviewed before the aggregate that uses it. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
63873b0 to
0680d0a
Compare
Tracking issue: #9868
This is the first of three PRs for NDV statistics. #11555 adds the aggregate and #11556 wires it into the footer. I put the sketch on its own because its bytes end up in file footers, so the format is the part that's hardest to change later. Golden files pin it down.
I wrote the HyperLogLog here instead of pulling in a crate, since we need to own the format, hash and seed for as long as those files exist, and none of the crates I looked at promise a stable serialized form. Precision tops out at 16, so a column can add at most 64 KiB to a footer. At the default of 12 it's at most 4 KiB, and low-cardinality columns only take a few bytes.
Could a maintainer add the
changelog/featurelabel? I can't set labels from a fork.I used Claude Code for analysis, debugging, and ideas.