You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Children of nullable Struct may have null values propagated to zone maps #9205
All Vortex files which have somewhere a nullable Struct with children face the following bug: children's values which are NULL may have their (unspecified) values propagated to zone maps. This also influences null count statistic.
What's worse, we can reproduce this simpler with FlatLayoutStrategy. As we use null count for falsifying zones, we may reject a valid zone.
See the following example: is_null filter returns 0 and not 1.
The reason for this bug is we write fields unmasked (i.e. not applying validity) and storing struct validity separately, and our filters evaluate column's validity only, not taking parent validity into account.
I’m taking a look at this. I plan to add a regression test for nested nullable structs and fix the statistics path so child validity reflects parent validity during zone map and null count evaluation. I’ll keep the physical child buffers unchanged if possible and make the inherited validity explicit where statistics are computed.
child validity reflects parent validity during zone map
Even after this change you can't assume child validity is possible to evaluate alone (from parent validity) because we have files written with older versions, and we can't make a breaking change to discard all old files.
Activity
myrrc commentedon Aug 5, 2026
The reason for this bug is we write fields unmasked (i.e. not applying validity) and storing struct validity separately, and our filters evaluate column's validity only, not taking parent validity into account.
Bardakor commentedon Aug 20, 2026
I’m taking a look at this. I plan to add a regression test for nested nullable structs and fix the statistics path so child validity reflects parent validity during zone map and null count evaluation. I’ll keep the physical child buffers unchanged if possible and make the inherited validity explicit where statistics are computed.
myrrc commentedon Aug 20, 2026
Even after this change you can't assume child validity is possible to evaluate alone (from parent validity) because we have files written with older versions, and we can't make a breaking change to discard all old files.
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Added new Unrecorded node to the StatsNode. This is utilized for
Record nested file statistics beneath nullable structs
Aggregate nested columns under nullable parents with a demand mask