Skip to content

fix: keep filters with correlated subqueries above extension nodes - #25294

Merged
kosiew merged 2 commits into
apache:mainfrom
kakiuwang-ui:fix/push-down-filter-subquery-outer-refs
Oct 6, 2026
Merged

kosiew merged 2 commits into
apache:mainfrom
kakiuwang-ui:fix/push-down-filter-subquery-outer-refs

Conversation

@kakiuwang-ui

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

An extension node can name columns whose predicates must not be pushed past it, via UserDefinedLogicalNode::prevent_predicate_push_down_columns. A predicate whose only dependency on those columns runs through a correlated subquery is pushed down anyway, and the resulting plan does not just perform worse — it fails to execute.

Using the NoopPlan node already present in the optimizer's own tests, which refuses push-down of column c:

Filter: EXISTS (SELECT sq.a FROM sq WHERE outer_ref(test.c) = sq.a)
  NoopPlan
    TableScan: test

optimizing that plan today gives

Optimizer rule 'push_down_filter' failed
  Invalid (non-executable) plan after Optimizer rule: push_down_filter
    In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan,
    Window functions, Aggregate and Join plan nodes, but was used in [NoopPlan]

The predicate is moved below the node that asked to keep it, lands somewhere subqueries are not permitted, and the invariant check turns that into a query failure.

The cause is the one #15046 describes. The check is

!expr.column_refs().iter().any(|c| prevent_cols.contains(&c.name))

and column_refs collects only Expr::Column. A subquery records the outer columns it correlates on in Subquery::outer_ref_columns, which Expr's traversal does not descend into, so EXISTS (... WHERE outer.c = ...) reports no column references at all and looks unconditionally safe to push.

What changes are included in this PR?

A references_any_column helper in push_down_filter.rs that asks column_refs first and then, for Expr::Exists, Expr::InSubquery and Expr::ScalarSubquery, also looks at the outer_ref_columns those carry. The extension-node branch uses it instead of column_refs alone. Nothing else changes.

I deliberately did not change column_refs itself, which is what the issue title asks for. It has many callers whose behaviour would shift — optimize_projections, CSE, window and join handling all use it, and several of them want "columns produced by this plan's own schema", not outer references belonging to a parent. Widening it looks like a separate change with its own risk, so this PR fixes the consumer that has a demonstrable failure. Happy to look at the broader change instead if you would rather that were the fix.

What is the testing strategy for this PR?

New unit test push_down_filter::tests::user_defined_plan_outer_referenced_column, next to the existing user_defined_plan test and reusing its NoopPlan. It asserts the EXISTS predicate stays above the node.

The test discriminates: with the outer-reference collection short-circuited out, it fails with the invalid-plan error quoted above; with the fix it passes. datafusion-optimizer (808) and datafusion-expr (262) pass, as does sqllogictest.

Are there any user-facing changes?

No public API changes. Plans change only where an extension node declares prevent_predicate_push_down_columns and a predicate correlates a subquery on one of those columns — previously an invalid plan, now a correct one. No in-tree node declares those columns outside tests, so no bundled plan snapshots move.

Two adjacent things found while working on this, neither touched here:

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kakiuwang-ui,

Thanks for working on this. The correlated subquery handling is close, but SetComparison is still missing from the protected-column dependency check and can hit the same invalid-plan case. I left an inline comment with the details.

let mut found = false;
expr.apply(|e| {
let outer_refs = match e {
Expr::Exists(exists) => &exists.subquery.outer_ref_columns,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

references_any_column handles Exists, InSubquery, and ScalarSubquery, but it misses Expr::SetComparison, which also stores correlated columns in Subquery::outer_ref_columns. Please include SetComparison here and add an ANY or ALL regression test where the comparison expression itself does not reference the protected column.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in 479bcf5. SetComparison is now handled alongside the other three, and I added user_defined_plan_outer_referenced_column_set_comparison: test.a > ANY (SELECT sq.a FROM sq WHERE test.c = sq.a), so the comparison expression names only test.a and the dependency on the protected test.c exists purely in outer_ref_columns.

Without the SetComparison arm that test fails exactly the way the original bug did:

Invalid (non-executable) plan after Optimizer rule: push_down_filter
In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan,
Window functions, Aggregate and Join plan nodes, but was used in [NoopPlan]

/// predicate, and `Expr`'s own traversal does not descend into that field, so
/// looking only at `column_refs` would report such a predicate as depending on
/// nothing and let it be pushed past a node that asked to keep those columns.
fn references_any_column(expr: &Expr, names: &HashSet<String>) -> bool {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we check ordinary Expr::Column references in the existing expr.apply walk instead of calling column_refs() first? That would preserve the behavior while avoiding the temporary HashSet and second traversal.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 479bcf5 — Expr::Column is now matched in the same apply walk, so there is no column_refs() HashSet and no second traversal. Behavior is unchanged because add_column_refs collects columns with that same Expr::apply walk; the only difference is that the column case can now stop the traversal early instead of always walking the whole expression first.

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Sep 29, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.55102% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (7b00b63) to head (1735fc6).
⚠️ Report is 233 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/optimizer/src/push_down_filter.rs 77.55% 4 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25294      +/-   ##
==========================================
- Coverage   81.91%   81.91%   -0.01%     
==========================================
  Files        1134     1134              
  Lines      425637   425674      +37     
  Branches   425637   425674      +37     
==========================================
+ Hits       348654   348678      +24     
- Misses      56302    56308       +6     
- Partials    20681    20688       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kakiuwang-ui,

Thanks for working on this. The updated traversal correctly accounts for ordinary column references and correlated subquery outer references, including SetComparison. The added ANY regression test covers the new case well. Looks good to me.

@kosiew

kosiew commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

@kakiuwang-ui
Can you resolve the CI errors?

`UserDefinedLogicalNode::prevent_predicate_push_down_columns` lets an
extension node name columns whose predicates must not be pushed past it.
`PushDownFilter` checked that list against `Expr::column_refs`, which only
collects `Expr::Column`. A subquery records the outer columns it correlates
on in `Subquery::outer_ref_columns`, and `Expr`'s traversal does not descend
into that field, so a predicate like `EXISTS (... WHERE outer.c = ...)` looked
like it referenced no columns at all and was pushed below the node.

For the node in the optimizer's own tests the result is not merely a worse
plan: the pushed-down `EXISTS` lands somewhere subqueries are not allowed, and
the invariant check fails the query with "In/Exist/SetComparison subquery can
only be used in Projection, Filter, TableScan, Window functions, Aggregate and
Join plan nodes".

Collect the outer references from `Exists`, `InSubquery` and `ScalarSubquery`
alongside `column_refs` when deciding what may be pushed.

Closes apache#15046
`Expr::SetComparison` (`= ANY`, `> ALL`) also carries its correlated
columns in `Subquery::outer_ref_columns`, so a predicate whose comparison
expression names no protected column was still pushed past the extension
node and failed the invariant check. Handle it alongside the other three
subquery expressions.

Ordinary `Expr::Column` references are now matched inside the same
`Expr::apply` walk instead of building a temporary `HashSet` via
`column_refs` first; `column_refs` collects columns with that same
traversal, so the behavior is unchanged.
@kakiuwang-ui
kakiuwang-ui force-pushed the fix/push-down-filter-subquery-outer-refs branch from 479bcf5 to e69e4fb Compare October 2, 2026 16:15
@kakiuwang-ui

Copy link
Copy Markdown
Contributor Author

Thanks for the ping — fixed and rebased onto main (e69e4fb).

The CI failure was a silent conflict rather than anything feature-related: references_any_column uses TreeNodeRecursion, which my branch's base still imported in push_down_filter.rs. On main that name was later dropped from the datafusion_common::tree_node import list, and since my diff never touched that line the merge took main's version — so the merged tree lost the import and failed with five E0433: cannot find type TreeNodeRecursion in this scope. That is also why the four failures were MSRV / wasm / macOS test / linux build (every job that actually compiles the crate) while fmt, semver and the doc jobs passed.

Rebasing reproduced it locally; the import is restored in the commit that introduces the function, so each commit still builds on its own.

Green locally on top of c922f88:

  • cargo test -p datafusion-optimizer — 923 + 26 + 5 passed, 0 failed
  • cargo fmt -p datafusion-optimizer -- --check — clean
  • cargo clippy -p datafusion-optimizer --all-targets -- -D warnings — clean

@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Oct 5, 2026
@kosiew

kosiew commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

🚀
@kakiuwang-ui
Thank you for your contribution.

@kosiew
kosiew added this pull request to the merge queue Oct 6, 2026
Merged via the queue into apache:main with commit 8f4c51f Oct 6, 2026
42 checks passed
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Oct 11, 2026
…pache#25294)

## Which issue does this PR close?

- Closes apache#15046.

## Rationale for this change

An extension node can name columns whose predicates must not be pushed
past it, via
`UserDefinedLogicalNode::prevent_predicate_push_down_columns`. A
predicate whose only dependency on those columns runs through a
correlated subquery is pushed down anyway, and the resulting plan does
not just perform worse — it fails to execute.

Using the `NoopPlan` node already present in the optimizer's own tests,
which refuses push-down of column `c`:

```
Filter: EXISTS (SELECT sq.a FROM sq WHERE outer_ref(test.c) = sq.a)
  NoopPlan
    TableScan: test
```

optimizing that plan today gives

```
Optimizer rule 'push_down_filter' failed
  Invalid (non-executable) plan after Optimizer rule: push_down_filter
    In/Exist/SetComparison subquery can only be used in Projection, Filter, TableScan,
    Window functions, Aggregate and Join plan nodes, but was used in [NoopPlan]
```

The predicate is moved below the node that asked to keep it, lands
somewhere subqueries are not permitted, and the invariant check turns
that into a query failure.

The cause is the one apache#15046 describes. The check is

```rust
!expr.column_refs().iter().any(|c| prevent_cols.contains(&c.name))
```

and `column_refs` collects only `Expr::Column`. A subquery records the
outer columns it correlates on in `Subquery::outer_ref_columns`, which
`Expr`'s traversal does not descend into, so `EXISTS (... WHERE outer.c
= ...)` reports no column references at all and looks unconditionally
safe to push.

## What changes are included in this PR?

A `references_any_column` helper in `push_down_filter.rs` that asks
`column_refs` first and then, for `Expr::Exists`, `Expr::InSubquery` and
`Expr::ScalarSubquery`, also looks at the `outer_ref_columns` those
carry. The extension-node branch uses it instead of `column_refs` alone.
Nothing else changes.

**I deliberately did not change `column_refs` itself**, which is what
the issue title asks for. It has many callers whose behaviour would
shift — `optimize_projections`, CSE, window and join handling all use
it, and several of them want "columns produced by this plan's own
schema", not outer references belonging to a parent. Widening it looks
like a separate change with its own risk, so this PR fixes the consumer
that has a demonstrable failure. Happy to look at the broader change
instead if you would rather that were the fix.

## What is the testing strategy for this PR?

New unit test
`push_down_filter::tests::user_defined_plan_outer_referenced_column`,
next to the existing `user_defined_plan` test and reusing its
`NoopPlan`. It asserts the `EXISTS` predicate stays above the node.

The test discriminates: with the outer-reference collection
short-circuited out, it fails with the invalid-plan error quoted above;
with the fix it passes. `datafusion-optimizer` (808) and
`datafusion-expr` (262) pass, as does `sqllogictest`.

## Are there any user-facing changes?

No public API changes. Plans change only where an extension node
declares `prevent_predicate_push_down_columns` and a predicate
correlates a subquery on one of those columns — previously an invalid
plan, now a correct one. No in-tree node declares those columns outside
tests, so no bundled plan snapshots move.

Two adjacent things found while working on this, neither touched here:

- `find_out_reference_exprs` does not see these outer references either,
for the same reason, so it cannot be used as the collection mechanism.
That is the same traversal gap as apache#16147, which @AdamGS has taken.
- The equivalent guard for correlated subqueries in other push-down
branches was not audited; this PR only covers the extension-node branch
that apache#15046 reports.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expression column_refs does not return outer referenced columns in subqueries

3 participants