Repository navigation
Conversation
A nullable struct field reaches a scalar function as mask(field, validity). When the validity is not constant the mask cannot be folded away, and functions such as lower then stop at the mask instead of reaching the run-end or dictionary values below it. Rewrite f(.., mask(x, m), ..) to mask(f(.., x, ..), m) when f is strict and infallible. Strictness keeps the masked rows null for any number of arguments, and infallibility makes it safe to evaluate f on the values the mask hides. Co-authored-by: Amp <amp@ampcode.com> Signed-off-by: Alfonso Subiotto <alfonso.subiotto@dash0.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A field of a nullable struct reaches a scalar function as
mask(field, validity). When some struct rows are null, the validity is a bool array, the mask is not folded away, and the function stops at the mask. For example,lower(mask(runend(dict), bool))runsloweronce per row, while the same expression without the mask runs it once per dictionary value.This is independent of #11388. Each change helps on its own, and a sliced run-end field of a nullable struct needs both.
Changes
maskchild:f(.., mask(x, m), ..)becomesmask(f(.., x, ..), m). Thenfreaches the encoding ofx, and the run-end and dictionary push-down rules apply.fis strict and infallible:mis false,f(.., null, ..)is null, the same as the masked result. This holds for any number of arguments, so the other arguments do not have to be constant.falso runs on the values thatmhides, so it must not be able to fail on them.fis itselfmask, so two masks cannot swap forever.ffrom computing the hidden rows.scalar_fn/rules.rs:lowerovermask(runend(dict), bool): it reaches the dictionary values and returns the expected values.<lifts; checked subtraction (fallible) and KleeneAND(not strict) do not. Each result equals the unoptimized array.This PR was written with the help of an AI coding agent. I reviewed the change and the tests.