Repository navigation
Prune zones using zone maps beneath nullable structs - #11710
connortsui20 wants to merge 1 commit into
Conversation
A field of a nullable struct reaches the reader wrapped in the struct's mask, so neither scan could prune with its zone map. Both scans now look through the mask and keep only proofs that stay true when the zone map also covers values the struct hides, because releases up to 0.88.0 wrote zone maps over the stored values and no file records whether a zone map excludes them. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
2c3606c to
add4115
Compare
|
More detail on the open question about The rewrite only ever makes a clause less true. It swaps a bound for a stored bound that is harder to satisfy, or it drops the clause to The pack rule weakens a nullable struct's combiner whenever the parent The examples below use a root The first example would skip every zone. Take a plugin predicate over The The root is now The second example flips a bound that the rewrite keeps. The proof Child The same logic written with A This is latent today. The built-in rules combine falsifiers across fields only with Two possible fixes:
The first option is the real fix, and the second only narrows the problem. |
A field beneath a nullable struct holds values in rows where the struct is null, so develop writes no zone map for it, and filters on fields inside nullable structs never prune. Bounds stay correct for the field as read through the struct, because the hidden values can only loosen them, and a NaN count of zero still proves that a zone holds no NaN. Such a field now records its bounds and its NaN count, and no counts that the hidden rows would make wrong. The scans from #11710 read these zone maps only in that conservative way. This is the writer half of the prototype's pruning beneath nullable structs (91dbd9b and 7ac0dbb). #10961 rewrites the same writer to aggregate only the visible values. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The branch carried the prototype versions of the changes that the open PRs make, and those PRs have moved on in review. For example, #11640 now names the layout `vortex.extension`. This merge takes the tree of a stack on develop at 967bcb6, which holds the PR heads and ports of the two changes that have no PR yet: - #11640 at 24fd9da. - #11710 at add4115. - #11639 at 4f25e2f. Its newer head, 115318b, fails three of its own tests. - #11642 at d16038c. - Zone maps of bounds beneath nullable structs. - Single fields of list elements on both scans. A merge instead of a force-push keeps every commit that the branch published reachable. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A field of a nullable struct is stored with a value even in rows where the struct is null, and filters reach the field wrapped in the struct's mask. Neither scan could see through that mask, so a field's zone map never ruled out a zone, even in files that have one. Releases up to 0.88.0 wrote these zone maps and develop writes none, so for now only scans of those older files skip more zones.
Both scans keep only proofs that stay true when a zone map also covers hidden values. A hidden value can only widen the stored bounds, so a minimum or maximum still rules a zone out, but a stored null count can miss the struct's nulls and is never used. Treating the statistics as exact is not possible yet, because no file records whether its zone map excludes hidden values, and the 0.88.0 files would then return wrong results for
s.x IS NULL. Pruning with null counts needs that marker and is left for after #10961.The writer change from #11641 is left out, because #10961 rewrites the same writer code to aggregate only the visible values. The classic scan also looks through the mask inside
BetweenandDynamicComparison, since the scan builder turns range filters intoBetweenbefore the struct reader sees them. The tests write zone maps the way 0.88.0 did, in both thevortex.zonedand the legacyvortex.statslayout.Open questions:
weaken_pruning_proofis only sound when the weakened proof reaches the root of the pruning plan throughANDandOR. The built-in stats rules satisfy this, but nothing enforces it.s.k != 'a'can skip zones in the classic scan but not in V2.Splits the reader half out of #11641.
This change was implemented with OpenAI Codex and Claude Code.