Skip to content

Prune zones using zone maps beneath nullable structs - #11710

Draft
connortsui20 wants to merge 1 commit into
developfrom
ct/prune-nullable-struct-fields
Draft

connortsui20 wants to merge 1 commit into
developfrom
ct/prune-nullable-struct-fields

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

A field of a nullable struct is stored with a value even in rows where the struct is null, and filters reach the field wrapped in the struct's mask. Neither scan could see through that mask, so a field's zone map never ruled out a zone, even in files that have one. Releases up to 0.88.0 wrote these zone maps and develop writes none, so for now only scans of those older files skip more zones.

Both scans keep only proofs that stay true when a zone map also covers hidden values. A hidden value can only widen the stored bounds, so a minimum or maximum still rules a zone out, but a stored null count can miss the struct's nulls and is never used. Treating the statistics as exact is not possible yet, because no file records whether its zone map excludes hidden values, and the 0.88.0 files would then return wrong results for s.x IS NULL. Pruning with null counts needs that marker and is left for after #10961.

The writer change from #11641 is left out, because #10961 rewrites the same writer code to aggregate only the visible values. The classic scan also looks through the mask inside Between and DynamicComparison, since the scan builder turns range filters into Between before the struct reader sees them. The tests write zone maps the way 0.88.0 did, in both the vortex.zoned and the legacy vortex.stats layout.

Open questions:

  • weaken_pruning_proof is only sound when the weakened proof reaches the root of the pruning plan through AND and OR. The built-in stats rules satisfy this, but nothing enforces it.
  • The V2 scan keeps fewer proof shapes than the classic scan, so a filter such as s.k != 'a' can skip zones in the classic scan but not in V2.

Splits the reader half out of #11641.

This change was implemented with OpenAI Codex and Claude Code.

A field of a nullable struct reaches the reader wrapped in the struct's
mask, so neither scan could prune with its zone map. Both scans now look
through the mask and keep only proofs that stay true when the zone map
also covers values the struct hides, because releases up to 0.88.0 wrote
zone maps over the stored values and no file records whether a zone map
excludes them.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20

Copy link
Copy Markdown
Member Author

More detail on the open question about weaken_pruning_proof: the rewrite is only sound when the clause it changes reaches the root of the pruning proof through AND and OR, and nothing enforces that today.

The rewrite only ever makes a clause less true. It swaps a bound for a stored bound that is harder to satisfy, or it drops the clause to null (or to false when the clause is non-nullable). Under AND and OR, a less true clause can only skip fewer zones, but under NOT it can skip more.

The pack rule weakens a nullable struct's combiner whenever the parent Eval passes is_zone_pruning (vortex-scan/src/plan/builtins/pack/mod.rs:404), and is_zone_pruning only checks the shape: a boolean expression with a stat and no root. When the root pack splits a proof by field, the part for a nullable struct has that shape, so the struct's pack weakens it as if it were the whole proof, even when the root's combiner puts a NOT above it.

The examples below use a root {s: {x: i64}?, y: i64} with zone maps on s.x and y. The plans come from a throwaway test on this branch that passes each proof straight to optimize_pruning, trimmed to the relevant lines. The test printed plans only and did not run a scan.

The first example would skip every zone. Take a plugin predicate over s.x and y that matches rows where s.x = 1 and y = 2. Its rule could falsify it as "skip unless both fields may contain the value":

not(may_contain(s.x, 1) and may_contain(y, 2))
where may_contain(c, v) = fill_null(min(c) <= v and max(c) >= v, true)

The fill_null makes the s.x clause non-nullable, and the rewrite has no case for fill_null, so it drops the clause to false:

root: vortex.plan.eval(bool, rows=2000) expr=vortex.not(($.0 and $.1))
  child: vortex.plan.pack({0=bool, 1=bool}, rows=2000)
    0: vortex.plan.eval(bool, rows=2000) expr=false
    1: vortex.plan.zoned(bool, rows=2000) prune=vortex.fill_null(((stat($, vortex.min()) <= 2i64) and (stat($, vortex.max()) >= 2i64)), true)

The root is now not(false and $.1), which is true for every zone, so a scan with this proof would skip every zone, even one with a row where s.x = 1 and y = 2.

The second example flips a bound that the rewrite keeps. The proof not(max(s.x) <= 5 and max(y) <= 5) becomes:

root: vortex.plan.eval(bool?, rows=2000) expr=vortex.not(($.0 and $.1))
  child: vortex.plan.pack({0=bool?, 1=bool?}, rows=2000)
    0: vortex.plan.zoned(bool?, rows=2000) prune=(stat($, vortex.max()) <= 5i64)
    1: vortex.plan.zoned(bool?, rows=2000) prune=(stat($, vortex.max()) <= 5i64)

Child 0 now reads the stored max of s.x, which includes values hidden by s. Take a zone where the visible s.x values are 1 to 5, a row with a null s stores x = 100, and every y is at most 5. Over the visible values the proof is not(true and true), which is false, so the zone is kept. With the stored max it is not(false and true), which is true, so the zone is skipped. A true result no longer implies the original proof.

The same logic written with OR is safe. For max(s.x) > 5 or max(y) > 5, the s.x clause drops to null, and the root null or $.1 keeps the zone unless y alone proves it:

root: vortex.plan.eval(bool?, rows=2000) expr=($.0 or $.1)
  child: vortex.plan.pack({0=bool?, 1=bool?}, rows=2000)
    0: vortex.plan.eval(bool?, rows=2000) expr=null
    1: vortex.plan.zoned(bool?, rows=2000) prune=(stat($, vortex.max()) > 5i64)

A NOT over a single field is also safe. not(max(s.x) <= 5) drops to null and not(is_null(max(s.x))) drops to false, because the rewrite treats the whole not(...) as one clause. The bad shape needs the NOT in an outer combiner, above a split between a nullable struct's field and another field.

This is latent today. The built-in rules combine falsifiers across fields only with AND and OR (the And and Or rules in vortex-expr/src/stats/rewrite/builtins.rs), and the only built-in rule that emits a NOT, the bloom filter's not(bloom_contains(...)), covers a single column. A rule registered with StatsSession::register_rewrite can produce the bad shape, though, for example one for a plugin predicate over two columns. The classic scan does not have this problem, because unmask_pruning_filter stops at the first function that is neither strict nor a top-level AND or OR.

Two possible fixes:

  • Only weaken a part of the proof that reaches the root through AND and OR. The struct's pack cannot see the root's combiner, so the root pack would need to pass that position down when it splits the proof, or the weakening would need to run where the whole proof is visible.
  • Drop a non-nullable clause to null instead of false. This fixes the first example, because not(null and $.1) keeps a zone where y may match, but it does not fix the second example, and it changes the clause's dtype from bool to bool?.

The first option is the real fix, and the second only narrows the problem.

connortsui20 added a commit that referenced this pull request Nov 20, 2026
A field beneath a nullable struct holds values in rows where the struct is
null, so develop writes no zone map for it, and filters on fields inside
nullable structs never prune. Bounds stay correct for the field as read through
the struct, because the hidden values can only loosen them, and a NaN count of
zero still proves that a zone holds no NaN. Such a field now records its bounds
and its NaN count, and no counts that the hidden rows would make wrong. The
scans from #11710 read these zone maps only in that conservative way.

This is the writer half of the prototype's pruning beneath nullable structs
(91dbd9b and 7ac0dbb). #10961 rewrites the same writer to aggregate only
the visible values.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20 added a commit that referenced this pull request Nov 20, 2026
The branch carried the prototype versions of the changes that the open
PRs make, and those PRs have moved on in review. For example, #11640
now names the layout `vortex.extension`. This merge takes the tree of a
stack on develop at 967bcb6, which holds the PR heads and ports of
the two changes that have no PR yet:

- #11640 at 24fd9da.
- #11710 at add4115.
- #11639 at 4f25e2f. Its newer head, 115318b, fails three of its
  own tests.
- #11642 at d16038c.
- Zone maps of bounds beneath nullable structs.
- Single fields of list elements on both scans.

A merge instead of a force-push keeps every commit that the branch
published reachable.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant