Repository navigation
perf: compute using columns once per SELECT, not per item - #26317
Conversation
kosiew
left a comment
There was a problem hiding this comment.
Thanks for addressing the repeated USING-column discovery during SELECT planning. Sharing the result across explicit projection expressions is an appropriate optimization, and the new benchmark targets the wide, deeply nested queries described in the issue.
The architectural boundary is sound: USING-column discovery depends on the input plan, not on individual expressions, so computing it once per projection avoids redundant traversal without introducing persistent caching or changing public APIs. However, moving discovery unconditionally to the projection level also makes wildcard-only queries perform unnecessary full-plan traversals.
I'm requesting changes to preserve the existing wildcard fast paths while retaining the explicit-expression speedup. Please make discovery conditional or lazy and validate the qualified-wildcard case against the parent revision. The additional semantic and scaling coverage suggestions are non-blocking.
SqlToRel::sql_select_to_rex called LogicalPlan::using_columns() for every SELECT item, walking the whole input plan each time. Compute it once in prepare_select_exprs (and once for a wildcard's REPLACE items) and pass it down. Adds a nested wide SELECT case to the sql_planner benchmark.
Wildcard-only projections never normalize an expression, so they no longer walk the input plan for USING columns. REPLACE items share the same lazily collected set. Adds a width/depth matrix and a qualified-wildcard control to the nested SELECT benchmark, and an SLT for REPLACE over a USING join. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e6789df to
3e532c4
Compare
|
thanks so much for the review @kosiew. I addressed your comments (with AI assistance) |
There was a problem hiding this comment.
Thanks for the follow-up changes. The projection-local lazy cache appropriately eliminates redundant USING-column traversals while preserving expression normalization and wildcard REPLACE behavior. The added regression coverage and nested-projection benchmarks address the relevant correctness and performance concerns.
The ownership boundary is appropriate: USING-column metadata belongs to the SELECT input plan and can be shared across its projection expressions. Lazy initialization also avoids unnecessary collection for wildcard-only projections, while passing the same cache through REPLACE planning preserves consistent resolution semantics.
All previous comments have been addressed, including the wildcard-traversal concern. I found no remaining blockers or actionable suggestions.
|
@kosiew thanks! anyone we should ask for a review? |
|
@alamb this is ready to merge if you have a moment 🙏 |
|
We normally leave PRs open for at least 24 hours after approval to give time for others to review if they want to. We'll get this merged shortly Thanks @JoeryH |
What evidence do we haev that planning is faster? |
The sql_planner benches added here (cargo bench -p datafusion --bench sql_planner -- logical_nested; CLICKBENCH_DATA_PATH=datafusion/core/tests/data/clickbench_hits_10.parquet is enough, the nested cases plan against an in-memory 700-column table). Rerun just now, main vs this PR:
|
|
Thanks @JoeryH -- can you please split the tests into their own PR so we can run them with our own bot runner to confirm? |
The cases now live in apache#27572, so the benchmark bot can run them against main before this change lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
benchmark run corroborates comment on expected improvement |
|
thanks @kosiew ! I'm a little bit confused now, should I put the benchmarks back into this PR? |
No, I included the above to verify the performance impact. |
## Which issue does this PR close? - Part of apache#26313. ## Rationale for this change Split out of apache#26317 so the benchmark bot can compare that change against main. ## What changes are included in this PR? New `sql_planner` cases: - `logical_nested_select_{100,700}_columns_{1,8}_levels`: nested SELECTs that each name every column of the one below. - `logical_nested_qualified_wildcard_700_columns_{1,8}_levels`: the same nesting through qualified wildcards, as a control. They plan against the in-memory 700-column `t700` table, so `CLICKBENCH_DATA_PATH=datafusion/core/tests/data/clickbench_hits_10.parquet` is enough to run them: ``` cargo bench -p datafusion --bench sql_planner -- logical_nested ``` ## What is the testing strategy for this PR? Benchmark only. The cases pass in criterion's test mode (`cargo test --bench sql_planner -- logical_nested`). ## Are there any user-facing changes? No. --- Prepared with AI assistance (Claude Code). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Joery van den Hoff <joery.vandenhoff@imc.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
run benchmark sql_planner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing sql-using-columns-once (33554e7) to 904afff (merge-base) diff Run configurationrun benchmark sql_plannerResults will be posted here when complete File an issue against this benchmark runner |
| ) -> Result<Vec<SelectExpr>> { | ||
| let mut prepared_select_exprs = Vec::with_capacity(projection.len()); | ||
| let mut error_builder = DataFusionErrorBuilder::new(); | ||
| let mut using_columns = LazyUsingColumns::default(); |
There was a problem hiding this comment.
I wonder if we could/should put this structure on the PlannerContext as it applies to all calls within that query?
There was a problem hiding this comment.
Thanks @alamb! I don't think it fits on PlannerContext: the set depends on the SELECT's input plan, and the context is shared by every SELECT in the query (derived tables, subqueries, set-operation legs, CTEs), each with a different input. Keeping it correct there would mean saving and restoring it around every SELECT like outer_queries_schemas_stack, or keying it by plan. One projection is exactly the scope where the input is fixed, so that's where the cache lives.
What's left is that each nested SELECT still walks its own input once, inner SELECTs included. Reusing an inner SELECT's set in the outer one could remove that, but it needs a way to tie sets to subtrees. Happy to file a follow-up if profiles show it's still significant after this.
There was a problem hiding this comment.
No need for a follow up -- what I was thinking was that if there are other calls like this that can be cached it would be valuable to consolidate / make a pattern that could be easily followed rather than just a bunch of one offs
Thanks for the response
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing sql-using-columns-once (33554e7) to 904afff (merge-base) diff Run configurationrun benchmark sql_plannerCPU Details (lscpu)Details
Resource Usagesql_planner — base (merge-base)
sql_planner — branch
File an issue against this benchmark runner |
Which issue does this PR close?
LogicalPlan::using_columns()once per SELECT item, making wide nested SELECTs quadratic #26313.Rationale for this change
Planning a SELECT costs O(items × input plan size), because
SqlToRel::sql_select_to_rexcallsLogicalPlan::using_columns()once per SELECT item. Each call walks the whole input plan, subqueries and expressions included, and the result depends only on the input plan. Nested wide SELECTs (views over views) therefore plan in roughly O(width² × depth²). The issue has the numbers. At 4000 columns × 12 levels,statement_to_plangoes from 8.0 s to 4.4 s with this change alone, and to 0.6 s once theDFSchemascan from #26314 is also gone.What changes are included in this PR?
prepare_select_exprscomputesplan.using_columns()once and passes it tosql_select_to_rex.plan_wildcard_optionsdoes the same for a wildcard'sREPLACEitems.logical_nested_select_700_columns_8_levelsgoes from 168 ms to 50 ms (full numbers in the comments).One behaviour difference: if
using_columns()itself fails, the error now surfaces once for the SELECT instead of once per item throughDataFusionErrorBuilder.using_columnsonly errors if the tree walk does, so I don't expect this to be observable, but I can keep the old shape with a lazily filledOptionif you'd prefer.What is the testing strategy for this PR?
The planned output is unchanged, so this relies on the existing
datafusion-sqltests and sqllogictests, which pass locally. Those cover USING / NATURAL joins and wildcard expansion over them, the casesusing_columnsexists for. The benchmark cases in #27572 cover the performance side.Are there any user-facing changes?
No. Planning is faster and plans are identical.
Prepared with AI assistance (Claude Code).