Repository navigation
Add the ApproxDistinct aggregate - #11555
Open
vishalkotak wants to merge 2 commits into
Open
vishalkotak wants to merge 2 commits into
vishalkotak wants to merge 2 commits into
Conversation
This was referenced Nov 17, 2026
vishalkotak
force-pushed
the
vishalkotak/ndv-aggregate-fn
branch
4 times, most recently
from
November 17, 2026 18:02
05e36b9 to
e47d1fa
Compare
vishalkotak
marked this pull request as ready for review
November 17, 2026 18:07
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its persisted format can be reviewed before the aggregate that uses it. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each valid value into the HyperLogLog sketch and keeps the sketch as its partial state, registered in the default aggregate session. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
vishalkotak
force-pushed
the
vishalkotak/ndv-aggregate-fn
branch
from
November 19, 2026 16:44
e47d1fa to
37094b8
Compare
Contributor
Author
|
Almost all of the time goes into the hash. Dispatching primitives by byte width rather than by type didn't change these numbers; it only cuts the generated code. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue: #9868
Stacked on #11554, so only the last commit is new here.
This adds the
ApproxDistinctaggregate. It hashes every valid value into the sketch and keeps the sketch as its partial state, so results from different chunks and files merge without counting a shared value twice. I left decimals out for now, since the same value can be stored at different widths in different chunks. Grouped aggregation returns an error for the moment, becausefinalizedoesn't get an execution context to decode the sketches.The new benchmark comes out at about 1.3 ns per primitive value, and the full table is in a comment below.
Could a maintainer add the
changelog/featurelabel? I can't set labels from a fork.I used Claude Code for analysis, debugging, and ideas.