Skip to content

[Testing only] Report the stored distinct-count estimate to DataFusion - #11560

Draft
vishalkotak wants to merge 4 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-datafusion
Draft

vishalkotak wants to merge 4 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-datafusion

Conversation

@vishalkotak

@vishalkotak vishalkotak commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Tracking issue: #9868

This one's only for trying things out end to end. I'm not asking to merge it. It's stacked on #11556, so only the last commit is new.

It makes infer_stats report the stored estimate to DataFusion as an inexact distinct_count, while a column that's known to be constant still reports an exact 1. To try it, write files with include_nested_statistics() and disable_editions(). Only the ListingTable path is wired up so far, and the V2 source would need the same lookup.

I used Claude Code for analysis, debugging, and ideas.

@robert3005

Copy link
Copy Markdown
Contributor

So the thing that is most interesting here is whether this should be an approx distinct count or an exact distinct count. How much does it cost to compute the exact variant? I think the reason we haven't just merged a pr like this yet was because we haven't had time to do the in depth analysis (and anyone can plug their own aggregate in the meantime). There's some fun results with duckdb if you start reporting ndv and I wonder if you have seen this in your experiments?

@vishalkotak

Copy link
Copy Markdown
Contributor Author

Thanks, this was a good nudge. I hadn't looked at DuckDB at all, so I spent some time on it today with TPC-H SF1. It's a rough first pass, but here's what I found.

Exact turns out to be pretty expensive. On lineitem it took about a second and 365 MiB at SF1, and 862 MiB at SF3, while the sketch stays at 81 KiB no matter what. Exact counts also don't add up across files. When I split the table into 8 files, summing them overcounted by up to 8x and taking the max undercounted by up to 7.6x, while the merged sketches stayed within 3%. One catch is that neither DataFusion nor DuckDB merges sketches today, so we'd have to do that on our side to get a real table-level number.

And yes, I ran into the fun results too. In DataFusion, q21 got about 25% slower once NDV was there, though nudging the values by 5% didn't change any plans. DuckDB was more dramatic: q21 got about 2x slower with the default sketch, but a more precise one (p=16) avoided it. So accuracy seems to matter a lot more for DuckDB.

What did you see on your side? I'm curious whether it lines up.

First part of NDV statistics (vortex-data#9868): the sketch on its own, so its
persisted format can be reviewed before the aggregate that uses it.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each
valid value into the HyperLogLog sketch and keeps the sketch as its
partial state, registered in the default aggregate session.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-datafusion branch from 91e093f to 251ce8a Compare November 19, 2026 16:44
Third part of NDV statistics (vortex-data#9868): list ApproxDistinct in the default
file-level aggregates. It belongs to no edition yet, so writers that
enforce editions keep leaving it out until a follow-up edition change.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Draft for trying NDV statistics (vortex-data#9868) end to end: when a file's footer
holds an ApproxDistinct sketch, infer_stats reports its estimate as an
inexact distinct_count.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-datafusion branch from 251ce8a to b21acc3 Compare November 19, 2026 17:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants