Skip to content

Prune zones beneath nullable structs - #11641

Closed
connortsui20 wants to merge 1 commit into
developfrom
ct/nullable-struct-zone-maps
Closed

connortsui20 wants to merge 1 commit into
developfrom
ct/nullable-struct-zone-maps

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

On develop, the writer omits zone maps for fields beneath nullable structs. A filter such as s.x < 10 therefore reads candidate zones even when a lower bound could rule them out.

This PR writes stored bounds and float NaN counts for those fields in vortex.zoned layouts. Both the classic and V2 scans use these statistics to skip zones when they prove that the filter has no matches.

s present Stored s.x Value seen by the filter
Yes 40 40
Yes 50 50
No 1000 NULL

The stored minimum of 40 rules out s.x < 10. If the hidden value were 0, the scan would keep the zone and check the rows. The writer omits null counts because a stored count of zero would incorrectly skip this zone for s.x IS NULL. A stored NaN count of zero proves there are no visible NaNs because hidden values can only add NaNs.

Writers that use the legacy vortex.stats format still omit zone maps beneath nullable structs. Related to #11160, which normalizes hidden child values before writing.

Stacked on #11638

This change was implemented and reviewed with OpenAI Codex.

@connortsui20 connortsui20 added the changelog/performance A performance improvement label Nov 19, 2026 — with ChatGPT Codex Connector
@connortsui20 connortsui20 self-assigned this Nov 19, 2026
@connortsui20
connortsui20 force-pushed the ct/nullable-struct-zone-maps branch from 23a26b5 to fe8e020 Compare November 19, 2026 19:20
Base automatically changed from ct/nullable-struct-projection to develop November 19, 2026 22:41
Fields beneath nullable structs had no zone maps, so filters read zones that bounds could discard. Bounds over stored values still cover visible rows, and a zero physical NaN count proves there are no visible NaNs. Use those facts in pruning plans while keeping exact filtering masked.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20

Copy link
Copy Markdown
Member Author

Closing in favor of #11710, which keeps the reader half of this PR. The writer half is left to #10961, which rewrites the same writer code to aggregate only visible values. The branch stays on the remote in case the writer change is needed after all.

@connortsui20
connortsui20 deleted the ct/nullable-struct-zone-maps branch November 20, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant