Skip to content

fix(duckdb): Push down IS [NOT] DISTINCT FROM filters - #11398

Merged
robert3005 merged 7 commits into
vortex-data:developfrom
AYUSHNSUT:ayush/duckdb-distinct-from-pushdown
Nov 18, 2026
Merged

robert3005 merged 7 commits into
vortex-data:developfrom
AYUSHNSUT:ayush/duckdb-distinct-from-pushdown

Conversation

@AYUSHNSUT

@AYUSHNSUT AYUSHNSUT commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

IS [NOT] DISTINCT FROM filters fail against a Vortex scan in DuckDB with Binder Error: cannot convert DUCKDB_VX_EXPR_TYPE_COMPARE_DISTINCT_FROM. So do IS [NOT] TRUE and IS [NOT] FALSE, which DuckDB rewrites to them. can_push_expression accepts every BoundComparison, but the converter only handles the six operators in Operator.

This lowers the two operators in the converter so the filter is evaluated by the scan.

Changes

  • vortex-duckdb/src/convert/expr.rs: lower IS NOT DISTINCT FROM as (l IS NULL AND r IS NULL) OR (l IS NOT NULL AND r IS NOT NULL AND l = r), and IS DISTINCT FROM as its negation. Unlike =, the result is never NULL, so it is correct under NOT and OR.
  • vortex-sqllogictest/slt/is_distinct_from.slt: covers a constant, NULL, column against column, NOT (...) and IS [NOT] TRUE. On DuckDB it also asserts through EXPLAIN that no FILTER node remains.

Tests

  • The slt fails on develop with the error above and passes with this change. It also fails if the lowering uses plain = or drops the not-null guards.
  • cargo test -p vortex-duckdb --lib and cargo test -p vortex-sqllogictest --test sqllogictests pass; clippy -D warnings and fmt on the pinned nightly are clean.

Notes

IS DISTINCT FROM lowers to a NOT(...), which has no zone-map falsification rule, so it is not pruned by zone maps. This is from reading the stats rewrite rules, not from a measurement.

AI assistance

Developed with Claude Code (Claude Sonnet 5.5); I reviewed the diff and the test results and stand behind the change.

🤖 Generated with Claude Code

DuckDB filters `x IS [NOT] DISTINCT FROM <value>`, and the `IS [NOT] TRUE/FALSE`
forms it rewrites to them, failed to bind against a Vortex scan with "cannot
convert DUCKDB_VX_EXPR_TYPE_COMPARE_DISTINCT_FROM". `can_push_expression`
accepted every comparison, but the converter only knew the operators in
`Operator`.

Lower them in the converter as
`(l IS NULL AND r IS NULL) OR (l IS NOT NULL AND r IS NOT NULL AND l = r)` and
its negation. Unlike `=`, this is never NULL, so it stays correct under NOT
and OR.

Add a sqllogictest that runs on DuckDB and DataFusion and, on DuckDB, asserts
that the filter is pushed into the scan.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Ayush Mishra <ayushm1906@gmail.com>
@codspeed

codspeed Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1975 untouched benchmarks
⏩ 53 skipped benchmarks1


Comparing AYUSHNSUT:ayush/duckdb-distinct-from-pushdown (374990e) with develop (ac96584)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Comment on lines +430 to +436
fn not_distinct_from(left: Expression, right: Expression) -> Expression {
let both_null = and(is_null(left.clone()), is_null(right.clone()));
let both_equal = and(
and(is_not_null(left.clone()), is_not_null(right.clone())),
eq(left, right),
);
or(both_null, both_equal)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add to todo here to use NULL-safe equality.

We will add a mode to eq that has this semantics going forwards

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joseph-isaacs Added the comment. Makes sense, then the function should just simplify to the new eq call

@robert3005 robert3005 added the changelog/fix A bug fix label Nov 18, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 18, 2026 23:38
@robert3005
robert3005 merged commit 22b241c into vortex-data:develop Nov 18, 2026
94 of 95 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants