Repository navigation
bench: add nested wide SELECT cases to sql_planner - #27572
Conversation
Nested SELECTs that each name every column of the one below, over a width/depth matrix, against the same nesting through qualified wildcards. They measure the per-item USING-column discovery from apache#26313. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cases now live in apache#27572, so the benchmark bot can run them against main before this change lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| }); | ||
| } | ||
|
|
||
| // Nested SELECTs that each name every column of the one below, against |
There was a problem hiding this comment.
The spell-check job flags SELEC in Nested SELECTs. Please change it to Nested SELECT statements to restore CI.
There was a problem hiding this comment.
Done, it now reads Nested SELECT statements.
| }; | ||
| let nested_query = |depth: usize, projection: &dyn Fn(&str) -> String| { | ||
| let mut query = format!("SELECT {} FROM t700", projection("t700")); | ||
| for level in 0..depth { |
There was a problem hiding this comment.
Could we clarify that _levels counts wrappers? For example, 8_levels generates nine SELECT statements, including the base query.
There was a problem hiding this comment.
Added: the comment now says {depth}_levels counts the wrappers around the base query, so 8_levels plans nine SELECT statements. I kept the names so they line up with your bot run.
| ); | ||
| } | ||
| for depth in [1, 8] { | ||
| let query = nested_query(depth, &|qualifier| format!("{qualifier}.*")); |
There was a problem hiding this comment.
Could we document these wildcard cases as controls for #26317? Compare each benchmark across revisions, since differences between projection styles also include parsing and other planning costs.
Spell out that `_levels` counts the wrappers around the base query and that the qualified wildcard cases are controls, and drop `SELECTs`, which the typos check reads as a misspelling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
## Which issue does this PR close? - Closes apache#26313. ## Rationale for this change Planning a SELECT costs O(items × input plan size), because `SqlToRel::sql_select_to_rex` calls `LogicalPlan::using_columns()` once per SELECT item. Each call walks the whole input plan, subqueries and expressions included, and the result depends only on the input plan. Nested wide SELECTs (views over views) therefore plan in roughly O(width² × depth²). The issue has the numbers. At 4000 columns × 12 levels, `statement_to_plan` goes from 8.0 s to 4.4 s with this change alone, and to 0.6 s once the `DFSchema` scan from apache#26314 is also gone. ## What changes are included in this PR? - `prepare_select_exprs` computes `plan.using_columns()` once and passes it to `sql_select_to_rex`. `plan_wildcard_options` does the same for a wildcard's `REPLACE` items. - The benchmark cases moved to apache#27572, so the bot can compare this against main. With them, `logical_nested_select_700_columns_8_levels` goes from 168 ms to 50 ms (full numbers in the comments). One behaviour difference: if `using_columns()` itself fails, the error now surfaces once for the SELECT instead of once per item through `DataFusionErrorBuilder`. `using_columns` only errors if the tree walk does, so I don't expect this to be observable, but I can keep the old shape with a lazily filled `Option` if you'd prefer. ## What is the testing strategy for this PR? The planned output is unchanged, so this relies on the existing `datafusion-sql` tests and sqllogictests, which pass locally. Those cover USING / NATURAL joins and wildcard expansion over them, the cases `using_columns` exists for. The benchmark cases in apache#27572 cover the performance side. ## Are there any user-facing changes? No. Planning is faster and plans are identical. --- Prepared with AI assistance (Claude Code). --------- Co-authored-by: Joery van den Hoff <8142891+JoeryH@users.noreply.github.com> Co-authored-by: Joery van den Hoff <joery.vandenhoff@imc.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Which issue does this PR close?
LogicalPlan::using_columns()once per SELECT item, making wide nested SELECTs quadratic #26313.Rationale for this change
Split out of #26317 so the benchmark bot can compare that change against main.
What changes are included in this PR?
New
sql_plannercases:logical_nested_select_{100,700}_columns_{1,8}_levels: nested SELECTs that each name every column of the one below.logical_nested_qualified_wildcard_700_columns_{1,8}_levels: the same nesting through qualified wildcards, as a control.They plan against the in-memory 700-column
t700table, soCLICKBENCH_DATA_PATH=datafusion/core/tests/data/clickbench_hits_10.parquetis enough to run them:What is the testing strategy for this PR?
Benchmark only. The cases pass in criterion's test mode (
cargo test --bench sql_planner -- logical_nested).Are there any user-facing changes?
No.
Prepared with AI assistance (Claude Code).
🤖 Generated with Claude Code