Skip to content

Lift masks above strict, infallible scalar functions - #11389

Closed
asubiotto wants to merge 1 commit into
vortex-data:developfrom
dash0hq:alfonso/mask-lift-strict-fns
Closed

asubiotto wants to merge 1 commit into
vortex-data:developfrom
dash0hq:alfonso/mask-lift-strict-fns

Conversation

@asubiotto

Copy link
Copy Markdown
Contributor

Summary

A field of a nullable struct reaches a scalar function as mask(field, validity). When some struct rows are null, the validity is a bool array, the mask is not folded away, and the function stops at the mask. For example, lower(mask(runend(dict), bool)) runs lower once per row, while the same expression without the mask runs it once per dictionary value.

This is independent of #11388. Each change helps on its own, and a sliced run-end field of a nullable struct needs both.

Changes

  • Add a parent rule for any scalar function with a mask child: f(.., mask(x, m), ..) becomes mask(f(.., x, ..), m). Then f reaches the encoding of x, and the run-end and dictionary push-down rules apply.
  • The rule applies only when f is strict and infallible:
    • Strict: where m is false, f(.., null, ..) is null, the same as the masked result. This holds for any number of arguments, so the other arguments do not have to be constant.
    • Infallible: after the rewrite, f also runs on the values that m hides, so it must not be able to fail on them.
    • The rule does not apply when f is itself mask, so two masks cannot swap forever.
  • The rule does not add work for canonical inputs: a mask over those is already folded into their validity before this rule sees it, and a mask never stopped f from computing the hidden rows.
  • Tests in scalar_fn/rules.rs:
    • A snapshot of lower over mask(runend(dict), bool): it reaches the dictionary values and returns the expected values.
    • A binary function with a non-constant sibling: < lifts; checked subtraction (fallible) and Kleene AND (not strict) do not. Each result equals the unoptimized array.
    • Each guard was checked by removing it: the matching case fails.

This PR was written with the help of an AI coding agent. I reviewed the change and the tests.

A nullable struct field reaches a scalar function as mask(field,
validity). When the validity is not constant the mask cannot be folded
away, and functions such as lower then stop at the mask instead of
reaching the run-end or dictionary values below it.

Rewrite f(.., mask(x, m), ..) to mask(f(.., x, ..), m) when f is strict
and infallible. Strictness keeps the masked rows null for any number of
arguments, and infallibility makes it safe to evaluate f on the values
the mask hides.

Co-authored-by: Amp <amp@ampcode.com>
Signed-off-by: Alfonso Subiotto <alfonso.subiotto@dash0.com>
@asubiotto asubiotto closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant