Skip to content

Validate nullability against physical validity - #10225

Closed
connortsui20 wants to merge 1 commit into
developfrom
ct/stats-validity-checks
Closed

connortsui20 wants to merge 1 commit into
developfrom
ct/stats-validity-checks

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Superseded by the narrower VarBinView validity check in #10269. The broader validation policy is deferred.

Original PR description

Tracking Issue: #10177

Summary

With accurate cached statistics, the existing validity shortcuts are correct. This PR changes whether validation may rely on those statistics when safe callers can populate the cache independently of the data.

For example, a validity mask of [false, true] with an incorrect cached Min = true is treated as all-valid. The regression deliberately supplies that incorrect hint. It does not demonstrate a bug in normal statistic computation or propagation.

This proposed validation policy is split out of #10178 for separate review. Its need and cost should be assessed independently of replacing the statistics API. Open question: should these boundaries inspect the mask, or should the statistics API restrict how trusted facts are established?

Changes

Inspect the actual mask when converting nullable validity to non-nullable validity, converting a boolean array to a mask, and checking that a MaskedArray child has no nulls. Both safe MaskedArray construction paths use the same policy.

The regression covers incorrect cached minima in both directions. Establishing an unchecked constructor invariant is distinct from demonstrating that a wrong hint can cause memory unsafety, which this PR does not establish.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20

Copy link
Copy Markdown
Member Author

So this PR was generated by codex. On the one hand this is probably not necessary. But on the other hand, it does seem weird that we just allow safe callers to supply whatever statistic they want even if it might be wrong? Do we want to build in some better guarantees here?

@connortsui20 connortsui20 added the changelog/chore A trivial change label Oct 2, 2026
@connortsui20
connortsui20 deleted the ct/stats-validity-checks branch October 5, 2026 10:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant