Skip to content

Fix volatile projection filter pushdown - #26288

Closed
Tiancheng-Xu wants to merge 1 commit into
apache:mainfrom
Tiancheng-Xu:tc/fix-datafusion-26267
Closed

Tiancheng-Xu wants to merge 1 commit into
apache:mainfrom
Tiancheng-Xu:tc/fix-datafusion-26267

Conversation

@Tiancheng-Xu

Copy link
Copy Markdown

Which issue does this PR close?

Closes #26267.

Rationale for this change

A filter on a projected volatile value can evaluate the expression again after physical filter pushdown. For example, a filter on x in SELECT random() AS x can use a different sample from the returned x, so rows can fail their own predicate or disappear from a tautology.

What changes are included in this PR?

Check the rewritten predicate for volatility before pushing it through ProjectionExec. Keep unsupported predicates in their original parent coordinates, above the projection. Independently immutable predicates and computed immutable projections still support pushdown. This is independent of the PhysicalFilter API prototype in #25760 and the logical CTE work in #26275.

Add three scoped Clippy allowances for existing borrowed Expr partition-key sets. Rust 1.99 flags Arrow's interior-mutable reservation accounting, which does not change the expression values used for hashing or equality. Clean main produces the same three diagnostics; no optimizer algorithm changes are included in these allowances.

What is the testing strategy for this PR?

Extend parquet_filter_pushdown.slt with deterministic plans and returned-value invariants, covering pushdown enabled/disabled, nested volatile expressions, mixed predicates, and immutable controls. Before the fix, six cases fail, including 759 instead of 1,000 rows for a tautology and 82 returned values violating their filter. The focused file and 51 projection unit tests pass with the fix. Full-feature Clippy also passes.

The extended workspace tests pass (12,598 tests; all 534 SQL test files), along with 110 CLI tests, 189 core doctests, all-feature Clippy, and the complete lint suite. Forward and reverse CI-profile planning microbenchmarks detect no statistically significant change. They run on a shared macOS machine with an unoptimized build and do not establish query-runtime performance.

Are there any user-facing changes?

Filters on volatile projections retain the projected value's semantics. There are no public API changes. This addresses static projection predicates; a separate scan-embedded Post dynamic-filter path has not been runtime-tested here.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 09:56
@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Oct 14, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

0 open findings

What changed in this PR

Fixes incorrect semantics when pushing filters through ProjectionExec if the rewritten predicate becomes volatile (e.g., via random()), ensuring filters evaluate against the projected value rather than re-evaluating volatile expressions downstream.

Changes:

  • Prevent physical filter pushdown through ProjectionExec when the rewritten predicate is volatile.
  • Add sqllogictest regressions to validate both plan shape and runtime invariants for volatile projections under pushdown on/off.
  • Add scoped Clippy allowances for HashSet<&Expr> partition-key usage impacted by Arrow’s interior mutability bookkeeping.
File Description
datafusion/​sqllogictest/​test_files/​parquet_filter_pushdown.slt Adds regression coverage verifying volatile projection predicates aren’t pushed into scans and results uphold invariants.
datafusion/​physical-plan/​src/​projection.rs Blocks predicate pushdown when unproject_expr yields a volatile expression, preserving volatility semantics.
datafusion/​optimizer/​src/​push_down_filter.rs Adds narrow Clippy allowances for borrowed Expr keys in partition-key sets.

🧠 Review effort: Lite


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@choksi2212 choksi2212 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The projection guard is the right fix for the reported path: unprojecting x < 0.5 through random() AS x is exactly where a non-volatile predicate becomes volatile, and the slt cases pin both the plan shape and the counts.

One more spot worth considering while in this area: the boundary in DataSourceExec::handle_child_pushdown_result (datafusion/datasource/src/source.rs) still hands every parent_filters entry straight to try_pushdown_filters, without a volatility check of its own. With this change the invariant holds indirectly — the projection now refuses to pass the unprojected volatile predicate down — but the guarantee at the source is only as strong as every intermediate node being equally careful. If some other pass-through node later rewrites a predicate into a volatile expression the way unproject does, the scan would silently start evaluating it per row again, with the same wrong-results shape as this issue.

Filtering at the boundary would make the invariant local: hold volatile predicates out of try_pushdown_filters and report them as PushedDown::No, so the parent keeps its FilterExec and the per-index alignment of the result vector stays intact. is_volatile from datafusion_physical_expr_common::physical_expr is already the check both sites would share.

I have a local prototype of that boundary guard (plus a physical_optimizer unit test with a TestScan that would otherwise accept the volatile predicate) — happy to push it to this branch if you think it is worth adding; otherwise the projection fix alone does close the reported case in #26267.

@adriangb

Copy link
Copy Markdown
Contributor

@Tiancheng-Xu could you rebase this PR please?

@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-optimizer v55.1.0 (current)
       Built [  20.811s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.025s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [  20.655s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.026s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.123s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  43.124s] datafusion-optimizer
    Building datafusion-physical-plan v55.1.0 (current)
       Built [  27.983s] (current)
     Parsing datafusion-physical-plan v55.1.0 (current)
      Parsed [   0.160s] (current)
    Building datafusion-physical-plan v55.1.0 (baseline)
       Built [  28.140s] (baseline)
     Parsing datafusion-physical-plan v55.1.0 (baseline)
      Parsed [   0.164s] (baseline)
    Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.485s] 223 checks: 216 pass, 7 fail, 0 warn, 31 skip

--- failure enum_missing: pub enum removed or renamed ---

Description:
A publicly-visible enum cannot be imported by its prior path. A `pub use` may have been removed, or the enum itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_missing.ron

Failed in:
  enum datafusion_physical_plan::aggregates::order::GroupClustering, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/order/mod.rs:53
  enum datafusion_physical_plan::aggregates::order::GroupClusteringMode, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/order/mod.rs:35
  enum datafusion_physical_plan::aggregates::GroupClusteringMode, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/order/mod.rs:35
  enum datafusion_physical_plan::joins::utils::ProbeOffset, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/joins/join_hash_map.rs:30

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/function_missing.ron

Failed in:
  function datafusion_physical_plan::proto::registry::execution_plan_names, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/registry.rs:175
  function datafusion_physical_plan::proto::execution_plan_names, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/registry.rs:175

--- failure inherent_method_missing: pub method removed or renamed ---

Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/inherent_method_missing.ron

Failed in:
  PartitionedTopKExec::with_ranking_field, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/sorts/partitioned_topk.rs:271
  PartitionedTopKExec::ranking_field, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/sorts/partitioned_topk.rs:313
  ExecutionPlanDecodeCtx::decode_children, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/mod.rs:314
  ExecutionPlanEncodeCtx::extension_node, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/mod.rs:224
  AggregateExec::group_clustering_mode, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/mod.rs:1534

--- failure module_missing: pub module removed or renamed ---

Description:
A publicly-visible module cannot be imported by its prior path. A `pub use` may have been removed, or the module may have been renamed, removed, or made non-public.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/module_missing.ron

Failed in:
  mod datafusion_physical_plan::proto::registry, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/registry.rs:5
  mod datafusion_physical_plan::materialized_cte, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/materialized_cte.rs:5

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_physical_plan::materialized_cte::MaterializedCteExec, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/materialized_cte.rs:481
  struct datafusion_physical_plan::aggregates::order::GroupClusteringFull, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/order/full.rs:44
  struct datafusion_physical_plan::materialized_cte::MaterializedCteBuffer, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/materialized_cte.rs:255
  struct datafusion_physical_plan::materialized_cte::MaterializedCteScanExec, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/materialized_cte.rs:658
  struct datafusion_physical_plan::aggregates::order::GroupClusteringPartial, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/aggregates/order/partial.rs:51

--- failure trait_method_missing: pub trait method removed or renamed ---

Description:
A trait method is no longer callable, and may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_method_missing.ron

Failed in:
  method try_to_proto of trait LazyBatchGenerator, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/memory.rs:161

--- failure trait_missing: pub trait removed or renamed ---

Description:
A publicly-visible trait cannot be imported by its prior path. A `pub use` may have been removed, or the trait itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/trait_missing.ron

Failed in:
  trait datafusion_physical_plan::proto::registry::ExtensionPlanFromProto, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/registry.rs:66
  trait datafusion_physical_plan::proto::ExtensionPlanFromProto, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/e087c92e62e65382a5fb41b42035f99f8f8e20c5/datafusion/physical-plan/src/proto/registry.rs:66

     Summary semver requires new major version: 7 major and 0 minor checks failed
    Finished [  57.938s] datafusion-physical-plan
    Building datafusion-sqllogictest v55.1.0 (current)
       Built [  65.343s] (current)
     Parsing datafusion-sqllogictest v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-sqllogictest v55.1.0 (baseline)
       Built [  65.380s] (baseline)
     Parsing datafusion-sqllogictest v55.1.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.069s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 132.905s] datafusion-sqllogictest

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Oct 28, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.79%. Comparing base (0d79252) to head (94922ba).
⚠️ Report is 255 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26288      +/-   ##
==========================================
- Coverage   82.80%   82.79%   -0.01%     
==========================================
  Files        1149     1149              
  Lines      454146   454150       +4     
  Branches   454146   454150       +4     
==========================================
- Hits       376035   376027       -8     
- Misses      55260    55270      +10     
- Partials    22851    22853       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@alamb

alamb commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.

https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful

This has the hallmarks of AI code that was not reviewed by the author: unrelated clippy allows and over a hundred lines of slt around a five-line fix, with a description built around total test counts.

If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.

@alamb alamb closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change optimizer Optimizer rules physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wrong results: with pushdown_filters = true, a filter on a volatile subquery column is evaluated again in the Parquet scan

6 participants