Repository navigation
fix: split a leaf projection stranded above a limit by a pushed-down OFFSET - #27395
cool9850311 wants to merge 5 commits into
Conversation
…OFFSET A source that supports skip pushdown wraps its scan in a `GlobalLimitExec` for the skip, and when the scan has more than one file group `EnsureRequirements` adds a `CoalescePartitionsExec` under that. Either way the nodes land between the scan and the leaf projection the logical rules placed on it. Neither lets that projection swap back down, so `LimitPushdown` leaves two projections adjacent and the next `ProjectionPushdown` merges them, hoisting the leaf expression back above the limit: `file_row_index()` then errors at execution and `get_field` reads the whole struct. `try_split_leaf_projection` already handles the same shape above a `RepartitionExec`. Generalise its host from one hardcoded node to a chain: `split_host_child` accepts a round-robin `RepartitionExec`, a `GlobalLimitExec` or a `CoalescePartitionsExec`, and `split_host_chain` walks as many as are stacked. A single `OFFSET` on a multi-file table already stacks two, so stopping at the first would leave that shape broken. Each arm evaluates no expression of its own, has one child, and passes its child's schema through. That last property is what lets the outer expressions keep their column indices across several hosts, and it holds by construction rather than by luck: none of the three `compute_properties` takes a schema, and each clones its input's equivalence properties, so a host's schema is its child's. A fourth arm that projects would have to re-establish that, and nothing here will catch it; the doc comment says so. Splitting rather than swapping the whole projection past the limit is what keeps the cost right: only the leaf half descends, so the rows the limit skips pay for a field access the source was going to make anyway and never for the outer expressions. Part of apache#27131 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…imit Every `OFFSET` in the new sections sets a skip that differs from the fetch, never `OFFSET n LIMIT n`: the split rebuilds the limit, and with the two numbers equal every assertion would hold just as well if they were swapped. `file_row_index.slt` gets five value queries for the marker UDFs, all of which error before the fix, an `EXPLAIN` for the rewritten plan, and two controls that never depended on it: `OFFSET 0`, and `OFFSET` with no `LIMIT`, where the skip is not pushed into the scan at all. It also gets a json table: `ListingTable::supports_skip_pushdown` is unconditional and the `GlobalLimitExec` wrap only looks at `skip`, so the defect is not specific to parquet, and every other query in the file is. `file_row_index()` itself is parquet-only, so that section uses `input_file_name()`. `projection_pushdown.slt` gets two value queries and four `EXPLAIN`s covering a bare leaf, a leaf beside a computed sibling, a leaf nested inside `upper(...)`, and the no-leaf case. The value queries pass before the fix on purpose: they pin the rows, which is the one thing a plan assertion cannot show. A dropped or swapped `skip` turns the plan text red as well, so they are not the only cover for it. The no-leaf `EXPLAIN` is labelled a control rather than a guard, because the scan prunes to `[id]` whether or not the rule bails out early; the unit test is what pins the bail-out. A further value query puts the marker inside a lambda body under an `OFFSET`: the lambda queries above it have no limit and the limit queries have no lambda, so nothing covered both, and it is also what pins the parameter rebase. Both files get a section with two file groups, which is the shape that stacks a `CoalescePartitionsExec` under the limit. Those count the surviving rows instead of listing them, because which rows survive the skip depends on the order the partitions are merged in, and `count(r)` rather than `count(*)`, so `optimize_projections` cannot drop the marker and leave nothing to evaluate. Neither new section contains a `RepartitionExec`, so both exercise this path rather than the one apache#27152 fixed. Ten unit tests hand-build the tree, because plain SQL does not reach these shapes: the split committing with both of the limit's numbers intact, the chain descending past two hosts and past three with a distinct pair of numbers on each, a `CoalescePartitionsExec` host keeping its own `fetch`, a `fetch` of `None`, a projection with no leaf expression left alone, `LocalLimitExec` kept out of the host set, a source that refuses the lower projection, the three nodes that meet the host conditions but are left out of the allowlist appearing instead as the chain bottom, and a `FilterExec` bottom, which takes the lower projection without absorbing it so the scan narrows only on this rule's second scheduled run. Part of apache#27131 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jayzhan211
left a comment
There was a problem hiding this comment.
@cool9850311 thanks for following up on the OFFSET row! Looks good to me. The three inline comments are optional.
Co-authored-by: Jay Zhan <jayzhan211@gmail.com>
alamb
left a comment
There was a problem hiding this comment.
Thanks @cool9850311 and @jayzhan211 -- the code looks ok but I think the comments make this really hard to understand. Can you please cut them down (by reading all comments, and removing anything you don't understand yourself)
I also have some ideas on how to improve the tests
Thank you for the contribution, and sorry for the request, I just think we need to fight to keep DataFusion understandable
| #[tokio::test] | ||
| async fn split_descends_through_global_limit() -> Result<()> { | ||
| let (schema, scan) = limit_split_scan()?; | ||
| // What a source that pushes `OFFSET` down builds: the skip becomes a |
There was a problem hiding this comment.
I don't understand these comments -- they are trying to describe in english the plan shape
I think these tests would be much more readable if instead of trying to describe this in textual comments, you instead add a snapshot of the orignal and the optimized plan
Sort of like
assert_snapshot!(input_and_optimized_plan_text(projection)?,
@r"
--- Input Plan
...
--- OutputPlan
ProjectionExec: expr=[__datafusion_leaf_0@1 as v, id@0 + 1 as w]
GlobalLimitExec: skip=1, fetch=3
DataSourceExec: file_groups={1 group: [[x]]}, projection=[id, get_field(s@1, value) as __datafusion_leaf_0], file_type=parquet
"
More background on the rationale here: https://andrew.nerdnetworks.org/blog/2026/11/13/transcript-testing/
| // a plain column, and is then no narrower than the widened child, or it | ||
| // narrows and holds a computed one. `RepartitionExec` declines both, it | ||
| // is not removable because it renames, and nothing else moved. | ||
| // stuck above a chain of the nodes `split_host_child` accepts. The |
There was a problem hiding this comment.
Can you please update these comments to not refer to implementation details elsewere -- pelase try and rephrase in your own words
| /// changed the width would leave both against the wrong one. | ||
| fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> { | ||
| if let Some(repartition) = node.downcast_ref::<RepartitionExec>() { | ||
| // Hash and Range partitioning reference columns by index, which would |
There was a problem hiding this comment.
likewise, these comments is far too verbise -- please rewrite these in your own words rather than leaving this re-explanation of the code in
| /// expression sharing it never reaches the source. Splitting leaves the | ||
| /// computation above the repartition, still parallelised. | ||
| /// Two nodes strand a `MoveTowardsLeafNodes` expression above the source this | ||
| /// way, and neither lets the whole projection past: |
| @@ -314,6 +314,191 @@ SELECT file_row_index() + 1, column1 FROM parquet_table ORDER BY column1 | |||
| # `RESET` restores the global default rather than the harness value, which the | |||
| # harness rejects as leftover configuration. | |||
|
|
|||
| # A leaf expression must also reach the scan when the source pushes an `OFFSET` | |||
There was a problem hiding this comment.
can you please reword these comments to explain the case they are teseting, not details about the code that implements it?
Sorry for not paying attention on readability of comments and tests, will work on this. |
…ments down Addresses the review on the tests and on the comments. Tests: the ten split tests asserted only the optimized plan, through a local `optimized_plan_text` helper. They now use `OptimizationTest`, which this test directory already has (`pushdown_sort.rs` uses it) and which renders the input plan and the output side by side, so a reader sees what the rule changed without opening the test body. It also compares the plan's schema before and after whenever the rule's `schema_check` is true, which `ProjectionPushdown` sets, so each of the ten now carries a schema assertion as well. Two tests needed restructuring to fit: the pass-through-bottom case was a loop over two node types, which inline snapshots cannot be, and is now two sequential assertions; the filter case takes one snapshot per scheduled run of the rule. The same change removes the inline snapshot formatting that the earlier review round asked about. Comments: the first pass measured the surroundings but did not treat the measurement as a ceiling. The second pass asks of every block whether it can be removed rather than whether it can be shortened, keeping one only if a reader would otherwise misread what the code does, or would make a change that breaks something no test catches. What went from the rule: the narration of `optimize()`'s two passes, which the two lines say already; the `split_host_child` and `split_host_chain` doc comments, since the schema pass-through is enforced by `schema_check` and the multi-host chain is stated by `split_descends_through_limit_and_coalesce`; the allowlist note, which `split_declines_local_limit` now carries alone; the `SplitHostChain` type and `child` field docs; the termination guard's note, which claimed non-termination the suite does not demonstrate; and the paragraph on what strands a leaf expression, which is context for the pull request rather than for the function. What stayed is the five facts not inferable from the code: why only round-robin partitioning qualifies, why `CoalescePartitionsExec` is in the chain, that `hosts` runs outermost first, that a `Some` from `try_swapping_with_projection` does not mean the source absorbed the projection, and why the hosts' properties are recomputed. In the tests and the `.slt` files the helper docs and the sentences restating a neighbouring query went; the ones that keep a test from looking redundant, or that explain a deliberately weak assertion, stayed. The termination guard's comment was also the one the review asked to reword. Measuring it first showed the claim was wrong: removing the guard does not hang the suite, it changes one test's output, so the comment went rather than being reworded. Rule: 78 comment lines down to 12, hunk density 64.5% to 21.8%, whole-file 11.9% to 9.4% against a 10.4% baseline, and the longest block this branch adds down from 15 lines to 4 (the file's own maximum is 7). Tests 17.7% to 6.0%, projection_pushdown.slt 18.1% to 10.4%, file_row_index.slt 22.6% to 14.4%. cargo fmt --check clean, clippy clean on datafusion-physical-optimizer --all-targets and on the core_integration test target, 41 projection pushdown tests and 51 physical-optimizer unit tests passing, both touched .slt files green, and the full sqllogictest suite green before the comment pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| - GlobalLimitExec: skip=1, fetch=3 | ||
| - CooperativeExec | ||
| - DataSourceExec: file_groups={1 group: [[x]]}, projection=[id, get_field(s@1, value) as __datafusion_leaf_0], file_type=parquet | ||
| " |
jayzhan211
left a comment
There was a problem hiding this comment.
@cool9850311 thanks for cutting the comments down, the input/output snapshots read much better! My approval stands. The four inline suggestions are optional comment tweaks: two put back a reason the cuts removed, two replace "corpus".
| format!("{LEAF_COLUMN_PREFIX}_{index}") | ||
| } | ||
|
|
||
| fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> { |
There was a problem hiding this comment.
Optional cleanup: with the old doc gone, nothing says what a node has to satisfy to be allowed here, and the tests now say "host" without a definition. One short reason would keep the next person from adding, say, a SortExec arm, whose sort expressions would point at the wrong columns after the split:
| fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> { | |
| /// Returns the input of `node` if the split may place the leaf projection below | |
| /// it. The split gives `node` an input with different columns, so `node` must | |
| /// not reference its input's columns and must output them unchanged, or the | |
| /// outer projection's column indices would point at the wrong columns. | |
| fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> { |
| /// Splits a `ProjectionExec` that mixes leaf expressions with ones that must | ||
| /// stay in place, moving the leaves into a projection the source absorbs. |
There was a problem hiding this comment.
Optional cleanup: main's doc here explained why the split is needed at all, and that reason is gone now. A short version that also covers the new limit case:
| /// Splits a `ProjectionExec` that mixes leaf expressions with ones that must | |
| /// stay in place, moving the leaves into a projection the source absorbs. | |
| /// Splits a `ProjectionExec` that mixes leaf expressions with ones that must | |
| /// stay in place, moving the leaves into a projection the source absorbs. | |
| /// Without the split the leaves stay above the source: no projection swaps past | |
| /// a `GlobalLimitExec`, and none with a computed expression swaps past a | |
| /// `RepartitionExec`. |
| /// `LocalLimitExec` is kept out of the host set. The corpus never strands a | ||
| /// leaf above one, but that is what the corpus shows, not a proof that nothing | ||
| /// can, so this pins the exclusion. |
There was a problem hiding this comment.
Optional cleanup: "corpus" isn't defined anywhere, so a reader can't tell which queries it means. Suggested wording:
| /// `LocalLimitExec` is kept out of the host set. The corpus never strands a | |
| /// leaf above one, but that is what the corpus shows, not a proof that nothing | |
| /// can, so this pins the exclusion. | |
| /// `LocalLimitExec` is left out of `split_host_child`: no SQL query strands a | |
| /// leaf expression above one today. This pins that choice. |
| /// Six numbers have to come back in the right places, not two. The corpus | ||
| /// reaches a chain this deep but never with a leaf to move. |
There was a problem hiding this comment.
Optional cleanup: same "corpus" wording here. The snapshot already shows the chain, so the comment only needs to say what is checked:
| /// Six numbers have to come back in the right places, not two. The corpus | |
| /// reaches a chain this deep but never with a leaf to move. | |
| /// Each of the three limits keeps its own `skip` and `fetch` after the rebuild. |
…tisfy Round 2 of review feedback, all four comment-only: - `split_host_child` had no doc after the previous cut, so nothing said what a node has to satisfy to be listed. A `SortExec` arm would compile and no test would catch it, because the arms are an allowlist and an unlisted node is never reached. The doc also states that satisfying the condition is necessary rather than sufficient: `LocalLimitExec`, `CooperativeExec` and `BufferExec` all satisfy it and are deliberately left out. - `try_split_leaf_projection` lost the reason the split exists at all, which main documented before this PR. Restored in one sentence that covers both the repartition and the limit case. - the two test doc comments said "corpus", which is not defined anywhere, and one of them claimed a chain depth the test corpus was never measured for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Done. |
alamb
left a comment
There was a problem hiding this comment.
I don't understand the need for this PR.
If you want to proceed, please clean up the ticket to explain the symptom you are solving, not all the internal details of what you think is a problem
The #27131 claims the query fails
COPY (SELECT value AS column1 FROM range(0, 20000))
TO '/tmp/df_leaf_repro/big.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE big
STORED AS PARQUET LOCATION '/tmp/df_leaf_repro/big.parquet';
SELECT file_row_index(), column1 + 1 FROM big; -- Execution errorHowever, when I ran it it works find
SELECT file_row_index(), column1 + 1 FROM big;
+------------------+------------------------+
| file_row_index() | big.column1 + Int64(1) |
+------------------+------------------------+
| 0 | 1 |
| 1 | 2 |
| 2 | 3 |
| 3 | 4 |
| 4 | 5 |
| 5 | 6 |
| 6 | 7 |
| 7 | 8 |
| 8 | 9 |
| 9 | 10 |
| 10 | 11 |
| 11 | 12 |
| 12 | 13 |
| 13 | 14 |
| 14 | 15 |
| 15 | 16 |
| 16 | 17 |
| 17 | 18 |
| 18 | 19 |
| 19 | 20 |
| 20 | 21 |
| 21 | 22 |
| 22 | 23 |
| 23 | 24 |
| 24 | 25 |
| 25 | 26 |
| 26 | 27 |
| 27 | 28 |
| 28 | 29 |
| 29 | 30 |
| 30 | 31 |
| 31 | 32 |
| 32 | 33 |
| 33 | 34 |
| 34 | 35 |
| 35 | 36 |
| 36 | 37 |
| 37 | 38 |
| 38 | 39 |
| 39 | 40 |
| . |
| . |
| . |
+------------------+------------------------+
20000 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 0.004 seconds.
jayzhan211
left a comment
There was a problem hiding this comment.
@cool9850311 thanks for putting those two reasons back. alamb's last review asks for the symptom rather than the internals, and the PR description still opens with the rule-by-rule walk-through, line numbers and mutation tables (about 10k characters). Could you cut it down to the template's sections, along the lines below? I checked both issues' repros: on this PR's base SELECT file_row_index() FROM small OFFSET 1 LIMIT 2 errors and s['a'] under OFFSET 5 LIMIT 3 scans projection=[s]. On this branch the first returns 1, 2 and the scan reads projection=[get_field(s@0, a) ...].
Suggested shorter PR description
## Which issue does this PR close?
Closes #27494.
Closes #27495.
## Rationale for this change
With `OFFSET n LIMIT m` on a listing table, `file_row_index()` and `input_file_name()` fail, and `s['a']` makes the scan read the whole struct:
```sql
SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
-- Execution error: file_row_index() is source dependent and cannot be evaluated directly
```
The skip pushed into the scan puts a `GlobalLimitExec` (and, with several files, a `CoalescePartitionsExec`) between the scan and the projection holding these expressions, and no projection moves below a `GlobalLimitExec`.
## What changes are included in this PR?
The leaf-projection split that #27152 added for a round-robin `RepartitionExec` now also moves past `GlobalLimitExec` and `CoalescePartitionsExec`, including several stacked. Only the leaf expressions move below the limit; computed ones stay above it, so the rows the `OFFSET` skips don't evaluate them.
## Are these changes tested?
Yes: value and `EXPLAIN` queries in `file_row_index.slt` and `projection_pushdown.slt`, and unit tests in `core/tests/physical_optimizer/projection_pushdown.rs` showing each input and output plan.
## Are there any user-facing changes?
These queries now return rows instead of an error, and `s['a']` under `OFFSET` reads only that field. No API change.Dismissing my earlier approval; see my latest review comment.
Done. |
|
Thanks, we are getting better here. Now that we have broken the symptoms into two separate tickets, can you pleaes break this PR into two separate / more focused PRs One for the fix / feature #27494 and the other for the new pushdown feature The goal is to make it as easy for you and us to review this PR as possible |
It's a little bit tricky, the root cause of them are actually the same, that's why this PR close both issues. |
@alamb The leaf projection is planted on the |
Which issue does this PR close?
file_row_index()/input_file_name()fail underOFFSET n LIMIT m#27494.OFFSET n LIMIT mstopsget_field()from being pushed into the file scan #27495.Rationale for this change
With
OFFSET n LIMIT mon a listing table,file_row_index()andinput_file_name()fail, ands['a']makes the scan read the whole struct:The skip pushed into the scan puts a
GlobalLimitExec(and, with several files, aCoalescePartitionsExec) between the scan and the projection holding these expressions, and no projection moves below aGlobalLimitExec.What changes are included in this PR?
The leaf-projection split that #27152 added for a round-robin
RepartitionExecnow also moves pastGlobalLimitExecandCoalescePartitionsExec, including several stacked. Only the leaf expressions move below the limit; computed ones stay above it, so the rows theOFFSETskips don't evaluate them.What is the testing strategy for this PR?
Yes: value and
EXPLAINqueries infile_row_index.sltandprojection_pushdown.slt, and unit tests incore/tests/physical_optimizer/projection_pushdown.rsshowing each input and output plan.Are there any user-facing changes?
These queries now return rows instead of an error, and
s['a']underOFFSETreads only that field. No API change.