Skip to content

Record ApproxDistinct in the default footer statistics - #11556

Open
vishalkotak wants to merge 3 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-footer-default
Open

vishalkotak wants to merge 3 commits into
vortex-data:developfrom
vishalkotak:vishalkotak/ndv-footer-default

Conversation

@vishalkotak

@vishalkotak vishalkotak commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Tracking issue: #9868

Stacked on #11555, so only the last commit is new here.

This is the part suggested in #9868: adding ApproxDistinct to the default footer aggregates. Nothing changes for default writes yet. It isn't in an edition, and since it has no legacy Stat slot it's only written alongside nested statistics. I also made legacy-only footers skip aggregates that have no slot. Until now every default aggregate had one, so this is the first that could be computed and then thrown away. On TPC-H lineitem at SF1, the footer grows by 29 KB on a 15 MB file, about 0.19%.

Two things I'm unsure about. Footer stats store every value as Precision::exact, so the estimate reads back as exact, and I'm not sure whether approximate aggregates should say Inexact. And whether about 4 KiB per high-cardinality column is OK, which we've been discussing on #11554.

Could a maintainer add the changelog/feature label? I can't set labels from a fork.

I used Claude Code for analysis, debugging, and ideas.

@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-footer-default branch 3 times, most recently from b0e2b36 to c0f25b7 Compare November 17, 2026 17:49
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-footer-default branch from c0f25b7 to 840c844 Compare November 17, 2026 18:02
@vishalkotak
vishalkotak marked this pull request as ready for review November 17, 2026 18:07
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its
persisted format can be reviewed before the aggregate that uses it.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each
valid value into the HyperLogLog sketch and keeps the sketch as its
partial state, registered in the default aggregate session.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-footer-default branch from 840c844 to b73dc22 Compare November 19, 2026 16:44
Third part of NDV statistics (vortex-data#9868): list ApproxDistinct in the default
file-level aggregates. It belongs to no edition yet, so writers that
enforce editions keep leaving it out until a follow-up edition change.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/ndv-footer-default branch from b73dc22 to e18bc9b Compare November 19, 2026 17:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant