Repository navigation
Fix DataFusion expression conversion and pushdown semantics - #9785
Conversation
01bc390 to
47d4415
Compare
Merging this PR will regress 1 benchmark
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | WallTime | bitpack_blocked_compress_avx512 |
4.9 µs | 5.6 µs | -13.27% |
| ⚡ | WallTime | threshold_neon[8] |
4 µs | 2.9 µs | +38.49% |
| ⚡ | WallTime | threshold_neon[80] |
5.8 µs | 4.7 µs | +21.98% |
| ⚡ | WallTime | threshold_neon[64] |
5.3 µs | 4.4 µs | +21.54% |
| ⚡ | WallTime | filter_neon[64] |
3.5 µs | 3 µs | +15.82% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing adamg/df-fix-bugs-conversion (7bb0bd5) with develop (9d5a844)
47d4415 to
4d7f747
Compare
Polar Signals Profiling ResultsLatest Run
Previous Runs (40)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.981x ➖, 2↑ 1↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.027x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.992x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.071x ➖, 2↑ 5↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench Sorted on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.018x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.019x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.980x ➖, 0↑ 0↓)
File Size Changes (100 files changed, -0.0% overall, 48↑ 52↓)
Totals:
|
Benchmarks: FineWeb S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.812x ➖, 3↑ 0↓)
datafusion / parquet / ns (0.482x ✅, 8↑ 0↓)
duckdb / vortex-file-compressed / ns (0.923x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.742x ➖, 4↑ 0↓)
|
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 3↑ 2↓)
duckdb / parquet / ns (0.997x ➖, 1↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 4↑ 4↓)
duckdb / parquet / ns (1.006x ➖, 4↑ 4↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.983x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: Statistical and Population Genetics 📖Commits: PR How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.762x ➖, 9↑ 0↓)
datafusion / parquet / ns (0.940x ➖, 1↑ 0↓)
duckdb / vortex-file-compressed / ns (0.886x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.923x ➖, 0↑ 0↓)
|
5a76dc2 to
2dd2e1a
Compare
55e1679 to
d37c4e1
Compare
## Problem `can_scalar_fn_be_pushed_down` accepts every `GetFieldFunc` without checking its source. `get_field(named_struct(...), 'value')` is valid in DataFusion, but `named_struct` has no Vortex expression conversion. Marking the whole expression pushable lets planning proceed into a conversion error instead of leaving it above the scan. ## Change Only mark `GetFieldFunc` pushable when its complete expression converts. The regression builds the actual DataFusion `named_struct` and `get_field` physical expressions and checks the pushdown decision. It fails on current `develop` and passes with this change. ## Verification - `cargo test -p vortex-datafusion test_get_field_of_unsupported_struct_function_stays_in_datafusion --lib` (with the repository-pinned `flatc` 25.12.19) - `cargo +nightly fmt --package vortex-datafusion -- --check` #9785 has a broader schema-aware expression-conversion redesign that also addresses this class of error. This PR offers a focused fix and regression for current `develop`; it can be superseded if #9785 lands first. Signed-off-by: Ivan Bobchenkov <me@bobchenkov.ru>
Replace separate eligibility and conversion checks with a schema-aware `try_convert`, preserve SQL null semantics for IN/NOT IN, and reject pushdown where casts, short-circuit evaluation, or nullable struct access could differ. Re-split filters after per-file schema adaptation so unsupported conjuncts run as DataFusion residual filters before projection and limit. Fix v2 filter pushdown after projection swapping, and rename `ExpressionConvertor` to `ExpressionConverter`. Squashed onto develop, porting the bind-before-optimize change from #10002. Signed-off-by: Adam Gutglick <adam@spiraldb.com> Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wUbDvL9BL68K3YceQiuZw Signed-off-by: Robert Kruszewski <github@robertk.io>
Build the v2 `DefaultExpressionConverter` from the data source's session so registered Arrow extension types resolve the same way as in the file source, and fold the pushed filters into the existing filter with `and_collect`. Signed-off-by: "Claude" <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wUbDvL9BL68K3YceQiuZw Signed-off-by: Robert Kruszewski <github@robertk.io>
The `CASE WHEN hd_vehicle_count > 0 THEN hd_dep_count / hd_vehicle_count END` conjunct is no longer pushed into the Vortex scan: its THEN branch can fail, and Vortex may evaluate it on rows the WHEN condition excludes. DataFusion now evaluates that conjunct in a FilterExec while the remaining conjuncts stay pushed down. Signed-off-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013wUbDvL9BL68K3YceQiuZw Signed-off-by: Robert Kruszewski <github@robertk.io>
Signed-off-by: Robert Kruszewski <github@robertk.io>
c31dafb to
7162b8d
Compare
One .profraw file per nextest test process fills the coverage runner disk. Use a pool of 8 merged files per binary. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Summary - Tracking Issue: #10634 For `ree_dict = 'x'` on a `RunEndEncoded<_, Dictionary<_, Utf8>>` column, DataFusion's comparison coercion produces `CAST(ree_dict AS Dictionary(UInt32, Utf8)) = 'x'`. Since #9785 the converter rejects that cast, so the filter runs in DataFusion instead of the scan. Results stay correct, but pushdown is lost. Both Arrow types map to Vortex `Utf8`, so Vortex has nothing to cast. ## Changes `supported_cast` becomes `classify_cast`, which either drops a cast, keeps it as a Vortex `cast`, or leaves it in DataFusion: - **Drop** (the converted child is returned unchanged) when the Arrow cast cannot fail and the Vortex dtype is unchanged: - identity casts; - `REE<Int16 | Int32, Dict<K, V>>` → `Dict<K, V>`. Arrow expands runs with `take` on `Int32` indices, and `take` on a dictionary only gathers keys. A dropped cast no longer blocks pushdown under `AND`/`OR` or in `IN` lists. - **Keep** numeric widening, as before. - **Leave in DataFusion** everything else, including: - `Int64` run ends (Arrow's indices may wrap); - `REE<_, V>` → `V` when expanding values can fail (for example, `Utf8`/`Binary` offset overflow); - `Dict<K, V>` → `V` (out of scope; see below); - dictionary key changes; - `V` → `Dict<K, V>`; - `LargeUtf8` → `Utf8`. Tests: - The issue's repro (coerce, simplify, plan) for `=`, `OR` and `IN`, asserting no `cast` node remains. - Accepted and rejected casts. - Vortex vs. DataFusion evaluation on an REE batch with nulls and unreferenced dictionary values. - A `.vortex` scan vs. `MemTable` test that also asserts the filter is accepted by the scan when pushdown is enabled. ## Follow-up scope `Dict<K, V>` → `V` is deliberately out of scope for this PR. Arrow decodes a dictionary by gathering one value per key. For offset-based values such as `Utf8` or `Binary`, the decoded output can exceed the offset limit and the cast can fail. A follow-up can allow specific value types only after checking that the Arrow cast cannot fail, preserves values and nulls, and leaves the expected Vortex dtype. For now, all dictionary-to-value casts stay in DataFusion. --------- Signed-off-by: Yiquan Zhou <yiquan.zhou@dash0.com>
…taFusion (#11161) ## Summary The DataFusion converter now pushes down integer and boolean casts to `Float64`, plus list casts whose element cast is supported. These were left out of the cast allowlist from #9785, but Arrow and Vortex convert them the same way. `array_sum` needs this: DataFusion always plans it as `array_sum(CAST(xs AS List<Float64>))`. ## Changes - `Boolean` and every integer type → `Float64`: both engines use `as f64` (round to nearest, never fails; booleans become 1.0/0.0). - Lossless unsigned → wider signed casts, such as `UInt32 → Int64`. - `List`/`LargeList` casts are classified by their element cast. `FixedSizeList → List` and narrowing element nullability stay in DataFusion. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Summary
Fix native pushdown in
vortex-datafusionto preserve DataFusion semantics and fall back for unsupported expressions. Replace separate eligibility and conversion checks with schema-awaretry_convert, preserving SQL null semantics forIN/NOT INand rejecting pushdown where casts, short-circuit evaluation, or nullable struct access could differ. Partially supported projections now run entirely in DataFusion over deduplicated raw columns.Re-split filters after each file’s schema adaptation: Vortex evaluates supported conjuncts, and DataFusion evaluates residuals before projection and limit. Disable scan limits whenever filters exist, include file and predicate context in errors, and fix
v2pushdown after projection swapping.Add conversion and residual-filter regression tests, plus 16 SQL cases compared against an Arrow
MemTablewith pushdown enabled and disabled.Rename
ConvertortoConverterLLMS were used as part of this work