Skip to content

Push filters on run-end encoded columns into Vortex scans - #11682

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
dash0hq:dat-366-ree-pushdown-upstream
Nov 20, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
dash0hq:dat-366-ree-pushdown-upstream

Conversation

@ivan-kiselev

Copy link
Copy Markdown
Contributor

For col = 'x' on a RunEndEncoded<_, Utf8> column, DataFusion keeps the column type and wraps the literal in REE, so there is no cast for #10637 to drop, and the converter's supported-type check rejected REE. The check now looks at the REE values type, as it already does for dictionaries, so these filters run inside the scan instead of in a FilterExec above it.

Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap run-end encoding only as the outermost layer, so REE nested in REE or in a dictionary is declined. DataFusion reads the nulls of a run-end encoded IN list from the RunArray's physical validity, which is always empty, so for a value not in (a, b, c, NULL) it returns false where SQL returns NULL; such lists are declined so both engines keep the same rows.

The new test compares Vortex with DataFusion for 26 value types on plain, dictionary and REE columns, using four-value IN lists because DataFusion rewrites shorter column IN lists into comparisons.

Related: #10637

Written with AI assistance (Amp).

For `col = 'x'` on a `RunEndEncoded<_, Utf8>` column, DataFusion keeps the
column type and wraps the literal in REE, so there is no cast for vortex-data#10637 to
drop. The converter's supported-type check unwrapped dictionaries but not
REE, so the filter stayed in DataFusion above the scan. The check now looks
at the REE values type, as it does for dictionaries.

Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap run-end
encoding only as the outermost layer, so REE nested in REE or in a dictionary
is declined. DataFusion reads the nulls of a run-end encoded IN list from its
physical validity, which is always empty, so it ignores a NULL element there;
such lists are declined so both engines keep the same rows.

Co-authored-by: Amp <amp@ampcode.com>
Signed-off-by: Ivan Kiselev <ivan.kiselev@dash0.com>
Amp-Thread-ID: https://ampcode.com/threads/T-8f2eff4f-acf9-4c62-9d71-bfef4f73cc66
@robert3005 robert3005 added the changelog/feature A new feature label Nov 20, 2026
@robert3005
robert3005 merged commit cb0ebee into vortex-data:develop Nov 20, 2026
104 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants