Repository navigation
Conversation
github-actions
Bot
added
documentation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #26276 +/- ##
==========================================
- Coverage 82.89% 82.89% -0.01%
==========================================
Files 1154 1155 +1
Lines 457630 458251 +621
Branches 457630 458251 +621
==========================================
+ Hits 379370 379855 +485
- Misses 55256 55288 +32
- Partials 23004 23108 +104 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
adriangb
force-pushed
the
mcte-auto-materialize
branch
2 times, most recently
from
October 16, 2026 03:46
2490f15 to
2de017a
Compare
adriangb
force-pushed
the
mcte-auto-materialize
branch
2 times, most recently
from
October 16, 2026 16:57
26d5165 to
644524b
Compare
adriangb
force-pushed
the
mcte-auto-materialize
branch
3 times, most recently
from
October 17, 2026 21:58
4f0729c to
2b967a0
Compare
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 28, 2026 00:59
2b967a0 to
b6991f1
Compare
github-actions
Bot
removed
substrait
adriangb
marked this pull request as ready for review
October 28, 2026 01:03
Add `MaterializedCte::automatic`, which tells an explicit `AS MATERIALIZED` node apart from one that the planner may create without a directive from the user. The optimizer will be free to inline an automatic node, and never inlines an explicit one. The flag defaults to `false`, is kept by tree node rewrites, and is shown in EXPLAIN as `automatic=true`. Add `MaterializedCte::inline`, which returns the continuation with a copy of the body in place of each scan of the CTE, also in subquery expressions. Each copy keeps the schema of the scan it replaces (with a projection when the qualified names differ), so the parents of the scans stay valid. A nested occurrence of the same CTE keeps its own scans. Nothing creates an automatic node yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `datafusion.optimizer.enable_automatic_cte_materialization` and its companion `datafusion.optimizer.automatic_cte_materialization_keeps_volatile`. Both are off by default, so each query plans as before. When the option is on, the SQL planner plans each non-recursive CTE that has no `MATERIALIZED` or `NOT MATERIALIZED` directive as a `MaterializedCte` with `automatic = true`. `NOT MATERIALIZED`, a body with an outer reference, and the CTEs of a `WITH RECURSIVE` block are still inlined. With `SELECT INTO`, the DDL node must stay the root of the plan, so an automatic CTE is inlined there instead of returning the "not implemented" error that an explicit one returns. No optimizer rule inlines automatic nodes yet, so with the option on every such CTE is materialized; the next commits add that decision. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `cte_auto_materialize.slt`, which runs CTE queries with `datafusion.optimizer.enable_automatic_cte_materialization` on: CTEs with zero, one and more references, volatile bodies, bodies with outer references, simple and complex bodies, LIMITs above the references, directives, nested and recursive CTEs, `SELECT INTO`, and unbounded bodies. The expected output is recorded at this commit, where no rule inlines an automatic CTE yet: every CTE without a directive is materialized, and a query whose CTE body is unbounded fails. The following commits update the expectations, so their diffs show how the plans change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the optimizer rule `InlineAutomaticCtes`, which runs first in the default rule list and decides which automatic materialized CTEs stay materialized. It replaces an automatic node with `MaterializedCte::inline` when: - the CTE has zero or one reference, - its body is volatile (unless `automatic_cte_materialization_keeps_volatile` is on) or has an outer reference, - its body is only projections and filters on one table, so computing it again is cheap and inlining keeps filter and projection pushdown, or - a `LIMIT` with a fetch is above all references in the chain of one-input nodes from the root of the query, so the query may read only a few rows. All other automatic CTEs stay materialized and are computed one time. Explicit `AS MATERIALIZED` CTEs are never inlined. The diff of `cte_auto_materialize.slt` shows which CTEs are now inlined. Update the optimizer rule reference and `explain.slt` for the new rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add the option `datafusion.optimizer.enable_automatic_cte_materialization`, which is off by default. With the default, each query plans as before. When the option is on, the SQL planner plans each non-recursive CTE that has no directive as a `MaterializedCte` with `automatic = true`, in each dialect. The new optimizer rule `InlineAutomaticCtes` replaces an automatic node with `MaterializedCte::inline` when the CTE has zero or one reference, or when its body is volatile, has an outer reference, or is only projections and filters on one table, or when a `LIMIT` with a fetch is above all references in the chain of one-input nodes from the root of the query. All other automatic CTEs stay materialized and are computed one time. `AS MATERIALIZED` and `AS NOT MATERIALIZED` override the rule. This commit completes the feature: the physical planner inlines an automatic CTE that has an unbounded body, because such a body cannot be buffered and the query worked before. Planning such a body returns an internal `UnboundedAutomaticCte` error, and `create_initial_plan` catches it, inlines that CTE and plans again. An explicit `AS MATERIALIZED` CTE with an unbounded body still returns a planning error. The diff of `cte_auto_materialize.slt` shows the unbounded cases that now plan and run. Closes apache#26237. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
mcte-auto-materialize
branch
from
October 28, 2026 05:30
b6991f1 to
063402b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Stacked on #26269. Only the last commit is new.
Rationale for this change
A CTE that has no directive is inlined at each reference, so its body runs one time for each reference:
With the option off, which is the default,
tis aggregated two times. With the option on, the body is computed one time:This works in each SQL dialect. A CTE with one reference, or with a cheap body, plans exactly as before.
What changes are included in this PR?
datafusion.optimizer.enable_automatic_cte_materialization, defaultfalse. With the default, each query plans as before.datafusion.optimizer.automatic_cte_materialization_keeps_volatile, defaultfalse. See rule 2 below.MaterializedCtehas the new fieldautomatic. When the option is on, the SQL planner plans each non-recursive CTE that has no directive as an automaticMaterializedCte.InlineAutomaticCtes, the first rule of the optimizer. It replaces an automatic node with a copy of the body in place of each scan when one of these conditions is true:datafusion.optimizer.automatic_cte_materialization_keeps_volatileisfalse(the default). Each reference then evaluates the expression again, as with the main option off (Avoid inlining non deterministic CTE #10337). With the second optiontrue, the CTE stays materialized and each reference reads the same rows. This is what PostgreSQL and DuckDB do.VALUESlist or empty relation, without a subquery expression.AS MATERIALIZEDandAS NOT MATERIALIZEDoverride the rule.WITH RECURSIVEblock and a CTE with a correlated body are inlined by the SQL planner, as before.SELECT INTOwith an automatic CTE inlines the CTE.create_initial_plancatches; a different mechanism is possible.Known limits:
EXPLAINshows theMaterializedCtenode in the logical plan and the inlined query in the physical plan, and the inlined copies do not get logical filter pushdown.MaterializedCtenodes for each CTE. The unparser and the Substrait producer return a "not implemented" error for these nodes.The rule also inlines an automatic CTE when a
LIMITwith a fetch is above all of its references. The rule finds such a limit by following the nodes with one input from the root of the query, and it stops at a node with a subquery expression. A materialized body is computed completely before any reference reads a row, so a limit above the references cannot stop it early. Inlined copies stop when the limit has its rows. This follows theContainsLimitcheck of DuckDB, with one difference: a sort with a fetch (TopK) does not count, because TopK reads all of its input, and inlining would then compute the body more than one time. A limit above only some references, for example above one side of a join, keeps the CTE materialized. A limit inside the CTE body also keeps it materialized.Comparison with PostgreSQL and DuckDB. PostgreSQL inlines a CTE with no directive only when it has one reference and no volatile functions, and it does not use cost (subselect.c). DuckDB materializes by default since v1.4.0. It keeps a CTE with a volatile body or an aggregate body materialized, and it inlines a cheap body (cte_inlining.cpp). The rules of this PR are closest to DuckDB's. Two things differ: a volatile body is inlined unless
automatic_cte_materialization_keeps_volatileis set, and a "cheap" body also includes filters and projections on one table, because DataFusion pushes those into the scan.Are these changes tested?
Yes. New file
cte_auto_materialize.slt: two references to an aggregate body, one reference, no reference, cheap body, volatile body (with each value ofautomatic_cte_materialization_keeps_volatile), scan in a subquery expression, CTE read by another CTE, correlated body, recursive CTE,SELECT INTO, the two directives, the MySQL dialect, an unbounded body, and aLIMITabove all references, above one side of a join, and none. Unit tests forInlineAutomaticCtesand forMaterializedCte::inline. The full sqllogictest suite also passes with the default of the option set totrue, except for the tests that show the default value.Are there any user-facing changes?
Yes. There are two new configuration options, which are off by default, and a new optimizer rule.
MaterializedCtehas the new public fieldautomaticand the new methodswith_automatic,is_automaticandinline. The upgrade guide for 56.0.0 and the configuration documentation are updated.