Skip to content

Add the ApproxDistinct aggregate - #11555

Open
vishalkotak wants to merge 2 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-aggregate-fn
Open

vishalkotak wants to merge 2 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-aggregate-fn

Conversation

@vishalkotak

@vishalkotak vishalkotak commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Tracking issue: #9868

Stacked on #11554, so only the last commit is new here.

This adds the ApproxDistinct aggregate. It hashes every valid value into the sketch and keeps the sketch as its partial state, so results from different chunks and files merge without counting a shared value twice. I left decimals out for now, since the same value can be stored at different widths in different chunks. Grouped aggregation returns an error for the moment, because finalize doesn't get an execution context to decode the sketches.

The new benchmark comes out at about 1.3 ns per primitive value, and the full table is in a comment below.

Could a maintainer add the changelog/feature label? I can't set labels from a fork.

I used Claude Code for analysis, debugging, and ideas.

@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-aggregate-fn branch 4 times, most recently from 05e36b9 to e47d1fa Compare November 17, 2026 18:02
@vishalkotak
vishalkotak marked this pull request as ready for review November 17, 2026 18:07
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its
persisted format can be reviewed before the aggregate that uses it.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each
valid value into the HyperLogLog sketch and keeps the sketch as its
partial state, registered in the default aggregate session.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-aggregate-fn branch from e47d1fa to 37094b8 Compare November 19, 2026 16:44
@vishalkotak

Copy link
Copy Markdown
Contributor Author

aggregate_approx_distinct results, median of 15,000 values per run on an M-series Mac:

Input Median
i64 19.4 µs
f32 18.8 µs
i64, half null 24.9 µs
utf8, 8 bytes 40.3 µs
utf8, 64 bytes 58.4 µs

Almost all of the time goes into the hash. Dispatching primitives by byte width rather than by type didn't change these numbers; it only cuts the generated code.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant