Repository navigation
Record ApproxDistinct in the default footer statistics - #11556
Open
vishalkotak wants to merge 3 commits into
Open
vishalkotak wants to merge 3 commits into
vishalkotak wants to merge 3 commits into
Conversation
vishalkotak
force-pushed
the
vishalkotak/ndv-footer-default
branch
3 times, most recently
from
November 17, 2026 17:49
b0e2b36 to
c0f25b7
Compare
vishalkotak
force-pushed
the
vishalkotak/ndv-footer-default
branch
from
November 17, 2026 18:02
c0f25b7 to
840c844
Compare
vishalkotak
marked this pull request as ready for review
November 17, 2026 18:07
First part of NDV statistics (vortex-data#9868): the sketch on its own, so its persisted format can be reviewed before the aggregate that uses it. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Second part of NDV statistics (vortex-data#9868): an aggregate that hashes each valid value into the HyperLogLog sketch and keeps the sketch as its partial state, registered in the default aggregate session. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
vishalkotak
force-pushed
the
vishalkotak/ndv-footer-default
branch
from
November 19, 2026 16:44
840c844 to
b73dc22
Compare
Third part of NDV statistics (vortex-data#9868): list ApproxDistinct in the default file-level aggregates. It belongs to no edition yet, so writers that enforce editions keep leaving it out until a follow-up edition change. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
vishalkotak
force-pushed
the
vishalkotak/ndv-footer-default
branch
from
November 19, 2026 17:04
b73dc22 to
e18bc9b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue: #9868
Stacked on #11555, so only the last commit is new here.
This is the part suggested in #9868: adding
ApproxDistinctto the default footer aggregates. Nothing changes for default writes yet. It isn't in an edition, and since it has no legacyStatslot it's only written alongside nested statistics. I also made legacy-only footers skip aggregates that have no slot. Until now every default aggregate had one, so this is the first that could be computed and then thrown away. On TPC-Hlineitemat SF1, the footer grows by 29 KB on a 15 MB file, about 0.19%.Two things I'm unsure about. Footer stats store every value as
Precision::exact, so the estimate reads back as exact, and I'm not sure whether approximate aggregates should sayInexact. And whether about 4 KiB per high-cardinality column is OK, which we've been discussing on #11554.Could a maintainer add the
changelog/featurelabel? I can't set labels from a fork.I used Claude Code for analysis, debugging, and ideas.