Repository navigation
[Testing only] Report the stored distinct-count estimate to DataFusion - #11560
vishalkotak wants to merge 4 commits into
Conversation
a421833 to
91e093f
Compare
|
So the thing that is most interesting here is whether this should be an approx distinct count or an exact distinct count. How much does it cost to compute the exact variant? I think the reason we haven't just merged a pr like this yet was because we haven't had time to do the in depth analysis (and anyone can plug their own aggregate in the meantime). There's some fun results with duckdb if you start reporting ndv and I wonder if you have seen this in your experiments? |
|
Thanks, this was a good nudge. I hadn't looked at DuckDB at all, so I spent some time on it today with TPC-H SF1. It's a rough first pass, but here's what I found. Exact turns out to be pretty expensive. On And yes, I ran into the fun results too. In DataFusion, q21 got about 25% slower once NDV was there, though nudging the values by 5% didn't change any plans. DuckDB was more dramatic: q21 got about 2x slower with the default sketch, but a more precise one (p=16) avoided it. So accuracy seems to matter a lot more for DuckDB. What did you see on your side? I'm curious whether it lines up. |
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its persisted format can be reviewed before the aggregate that uses it. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each valid value into the HyperLogLog sketch and keeps the sketch as its partial state, registered in the default aggregate session. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
91e093f to
251ce8a
Compare
Third part of NDV statistics (vortex-data#9868): list ApproxDistinct in the default file-level aggregates. It belongs to no edition yet, so writers that enforce editions keep leaving it out until a follow-up edition change. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Draft for trying NDV statistics (vortex-data#9868) end to end: when a file's footer holds an ApproxDistinct sketch, infer_stats reports its estimate as an inexact distinct_count. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
251ce8a to
b21acc3
Compare
Tracking issue: #9868
This one's only for trying things out end to end. I'm not asking to merge it. It's stacked on #11556, so only the last commit is new.
It makes
infer_statsreport the stored estimate to DataFusion as an inexactdistinct_count, while a column that's known to be constant still reports an exact 1. To try it, write files withinclude_nested_statistics()anddisable_editions(). Only theListingTablepath is wired up so far, and the V2 source would need the same lookup.I used Claude Code for analysis, debugging, and ideas.