[adriangb]

Which issue does this PR close?

• Closes Make the materialized CTE nodes first-class LogicalPlan variants #26238.

Rationale for this change

#25580 plans WITH x AS MATERIALIZED (...) with two built-in UserDefinedLogicalNodes inside LogicalPlan::Extension. Rules that must know about the nodes find them with a downcast, and code outside DataFusion sees an unknown extension node.

For users of SQL nothing changes. Plans and results are the same. For library users, the nodes are now visible in a match:

// before
LogicalPlan::Extension(Extension { node }) if node.as_any().is::<MaterializedCte>() => { /* downcast */ }
// after
LogicalPlan::MaterializedCte(MaterializedCtePlan::Cte(cte)) => { /* cte.body(), cte.continuation() */ }
LogicalPlan::MaterializedCte(MaterializedCtePlan::Scan(scan)) => { /* scan.id() */ }

It is easier to do this before more rules are written against the extension nodes.

What changes are included in this PR?

• One new variant, LogicalPlan::MaterializedCte(MaterializedCtePlan). MaterializedCtePlan has two cases: Cte, which computes the body once and evaluates the continuation, and Scan, a leaf that reads the buffered output. This keeps the number of top-level variants low, in the same way as LogicalPlan::Statement and LogicalPlan::Ddl. A match arm that treats both cases the same way is LogicalPlan::MaterializedCte(_).
• The reference is a Scan case and not a TableScan over a TableSource. A TableScan would need a downcast of the source to read the CTE id, and OptimizeProjections would write a projection into it, which changes the plans.
• MaterializedCte::try_new checks that each scan of the CTE in the continuation has the column names and types of the body. with_new_exprs and the TreeNode rewrites do not run this check again.
• MaterializedCteScan::find_in returns the scans of one CTE in a plan, also in subquery expressions.
• The downcasts are removed from PushDownFilter, PullUpCorrelatedExpr, LogicalPlan::head_output_expr and the physical planner. Each match now states the behaviour, and the behaviour is the same as before.
• The unparser, the Substrait producer and the protobuf serializer return a "not implemented" error for the new variant. Protobuf support is Protobuf serialization for materialized CTE plan nodes #26231.

Comparison with PostgreSQL and DuckDB. PostgreSQL plans a CTE as a separate subplan that CteScan nodes read. DuckDB uses a LOGICAL_MATERIALIZED_CTE node and LOGICAL_CTE_REF leaves. This PR uses the same two-part shape.

Are these changes tested?

Yes. cte_materialized.slt passes without changes, so no plan changed. There are new unit tests for the constructor check and for find_in. The "not implemented" arms have no tests.

Are there any user-facing changes?

Yes, this is a public API change. An exhaustive match on LogicalPlan in downstream code needs one new arm. MaterializedCte::cte and MaterializedCte::continuation are now Arc<LogicalPlan>. There is an entry in the 56.0.0 upgrade guide.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 77.60736% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.88%. Comparing base (484183d) to head (14c4e73).

Files with missing lines	Patch %	Lines
...tafusion/expr/src/logical_plan/materialized_cte.rs	79.19%	22 Missing and 9 partials ⚠️
datafusion/expr/src/logical_plan/plan.rs	59.25%	11 Missing ⚠️
datafusion/expr/src/logical_plan/display.rs	0.00%	8 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	83.72%	4 Missing and 3 partials ⚠️
datafusion/core/src/physical_planner.rs	85.71%	2 Missing and 3 partials ⚠️
...ion/substrait/src/logical_plan/producer/rel/mod.rs	0.00%	4 Missing ⚠️
datafusion/optimizer/src/decorrelate.rs	76.92%	1 Missing and 2 partials ⚠️
datafusion/optimizer/src/optimizer.rs	60.00%	0 Missing and 2 partials ⚠️
datafusion/expr/src/logical_plan/tree_node.rs	93.75%	0 Missing and 1 partial ⚠️
datafusion/sql/src/unparser/plan.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26269      +/-   ##
==========================================
- Coverage   82.89%   82.88%   -0.02%     
==========================================
  Files        1154     1154              
  Lines      457630   457707      +77     
  Branches   457630   457707      +77     
==========================================
+ Hits       379370   379378       +8     
- Misses      55256    55308      +52     
- Partials    23004    23021      +17     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning 7de176016126f02f217c5b9b1cbbc6280ed02b33
    Building datafusion v55.1.0 (current)
       Built [  62.259s] (current)
     Parsing datafusion v55.1.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion v55.1.0 (baseline)
       Built [  17.710s] (baseline)
     Parsing datafusion v55.1.0 (baseline)
      Parsed [   0.039s] (baseline)
    Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.599s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  82.173s] datafusion
    Building datafusion-expr v55.1.0 (current)
       Built [  22.544s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.108s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [   7.521s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.109s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   1.389s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_added.ron

Failed in:
  variant LogicalPlan:MaterializedCte in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:370
  variant LogicalPlan:MaterializedCte in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:370

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  32.722s] datafusion-expr
    Building datafusion-optimizer v55.1.0 (current)
       Built [  17.084s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [   7.620s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.038s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.167s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  25.667s] datafusion-optimizer
    Building datafusion-proto v55.1.0 (current)
       Built [  28.429s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  22.023s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.014s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.125s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  51.720s] datafusion-proto
    Building datafusion-sql v55.1.0 (current)
       Built [  20.267s] (current)
     Parsing datafusion-sql v55.1.0 (current)
      Parsed [   0.034s] (current)
    Building datafusion-sql v55.1.0 (baseline)
       Built [  12.820s] (baseline)
     Parsing datafusion-sql v55.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-sql v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.251s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  34.319s] datafusion-sql
    Building datafusion-substrait v55.1.0 (current)
       Built [ 307.215s] (current)
     Parsing datafusion-substrait v55.1.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-substrait v55.1.0 (baseline)
       Built [  19.244s] (baseline)
     Parsing datafusion-substrait v55.1.0 (baseline)
      Parsed [   0.013s] (baseline)
    Checking datafusion-substrait v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.212s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [ 328.258s] datafusion-substrait

[jayzhan211]

nit: the field is cte but the accessor is body(), so call sites end up reading cte.cte (e.g. in decorrelate.rs and optimize_projections). Since #25580 hasn't shipped in a release yet, renaming the field to body now would be free:

-    pub cte: Arc<LogicalPlan>,
+    pub body: Arc<LogicalPlan>,

[jayzhan211]

Thanks @adriangb! This is ready to go once the conflicts are resolved.

[alamb]

I merged up to resolve conflicts