Repository navigation
Push filters on run-end encoded columns into Vortex scans - #11682
Merged
robert3005 merged 1 commit intoNov 20, 2026
Merged
Conversation
For `col = 'x'` on a `RunEndEncoded<_, Utf8>` column, DataFusion keeps the column type and wraps the literal in REE, so there is no cast for vortex-data#10637 to drop. The converter's supported-type check unwrapped dictionaries but not REE, so the filter stayed in DataFusion above the scan. The check now looks at the REE values type, as it does for dictionaries. Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap run-end encoding only as the outermost layer, so REE nested in REE or in a dictionary is declined. DataFusion reads the nulls of a run-end encoded IN list from its physical validity, which is always empty, so it ignores a NULL element there; such lists are declined so both engines keep the same rows. Co-authored-by: Amp <amp@ampcode.com> Signed-off-by: Ivan Kiselev <ivan.kiselev@dash0.com> Amp-Thread-ID: https://ampcode.com/threads/T-8f2eff4f-acf9-4c62-9d71-bfef4f73cc66
robert3005
approved these changes
Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For
col = 'x'on aRunEndEncoded<_, Utf8>column, DataFusion keeps the column type and wraps the literal in REE, so there is no cast for #10637 to drop, and the converter's supported-type check rejected REE. The check now looks at the REE values type, as it already does for dictionaries, so these filters run inside the scan instead of in aFilterExecabove it.Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap run-end encoding only as the outermost layer, so REE nested in REE or in a dictionary is declined. DataFusion reads the nulls of a run-end encoded
INlist from theRunArray's physical validity, which is always empty, so for a value not in(a, b, c, NULL)it returns false where SQL returns NULL; such lists are declined so both engines keep the same rows.The new test compares Vortex with DataFusion for 26 value types on plain, dictionary and REE columns, using four-value
INlists because DataFusion rewrites shorter columnINlists into comparisons.Related: #10637
Written with AI assistance (Amp).