Skip to content

fix: split a leaf projection stranded above a limit by a pushed-down OFFSET - #27395

Open
cool9850311 wants to merge 5 commits into
apache:mainfrom
cool9850311:fix-27131-leaf-projection-past-limit
Open

cool9850311 wants to merge 5 commits into
apache:mainfrom
cool9850311:fix-27131-leaf-projection-past-limit

Conversation

@cool9850311

@cool9850311 cool9850311 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

With OFFSET n LIMIT m on a listing table, file_row_index() and input_file_name() fail, and s['a'] makes the scan read the whole struct:

SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
-- Execution error: file_row_index() is source dependent and cannot be evaluated directly

The skip pushed into the scan puts a GlobalLimitExec (and, with several files, a CoalescePartitionsExec) between the scan and the projection holding these expressions, and no projection moves below a GlobalLimitExec.

What changes are included in this PR?

The leaf-projection split that #27152 added for a round-robin RepartitionExec now also moves past GlobalLimitExec and CoalescePartitionsExec, including several stacked. Only the leaf expressions move below the limit; computed ones stay above it, so the rows the OFFSET skips don't evaluate them.

What is the testing strategy for this PR?

Yes: value and EXPLAIN queries in file_row_index.slt and projection_pushdown.slt, and unit tests in core/tests/physical_optimizer/projection_pushdown.rs showing each input and output plan.

Are there any user-facing changes?

These queries now return rows instead of an error, and s['a'] under OFFSET reads only that field. No API change.

cool9850311 and others added 2 commits November 12, 2026 23:20
…OFFSET

A source that supports skip pushdown wraps its scan in a `GlobalLimitExec`
for the skip, and when the scan has more than one file group
`EnsureRequirements` adds a `CoalescePartitionsExec` under that. Either way
the nodes land between the scan and the leaf projection the logical rules
placed on it. Neither lets that projection swap back down, so `LimitPushdown`
leaves two projections adjacent and the next `ProjectionPushdown` merges
them, hoisting the leaf expression back above the limit: `file_row_index()`
then errors at execution and `get_field` reads the whole struct.

`try_split_leaf_projection` already handles the same shape above a
`RepartitionExec`. Generalise its host from one hardcoded node to a chain:
`split_host_child` accepts a round-robin `RepartitionExec`, a
`GlobalLimitExec` or a `CoalescePartitionsExec`, and `split_host_chain` walks
as many as are stacked. A single `OFFSET` on a multi-file table already
stacks two, so stopping at the first would leave that shape broken.

Each arm evaluates no expression of its own, has one child, and passes its
child's schema through. That last property is what lets the outer expressions
keep their column indices across several hosts, and it holds by construction
rather than by luck: none of the three `compute_properties` takes a schema,
and each clones its input's equivalence properties, so a host's schema is its
child's. A fourth arm that projects would have to re-establish that, and
nothing here will catch it; the doc comment says so.

Splitting rather than swapping the whole projection past the limit is what
keeps the cost right: only the leaf half descends, so the rows the limit
skips pay for a field access the source was going to make anyway and never
for the outer expressions.

Part of apache#27131

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…imit

Every `OFFSET` in the new sections sets a skip that differs from the fetch,
never `OFFSET n LIMIT n`: the split rebuilds the limit, and with the two
numbers equal every assertion would hold just as well if they were swapped.

`file_row_index.slt` gets five value queries for the marker UDFs, all of
which error before the fix, an `EXPLAIN` for the rewritten plan, and two
controls that never depended on it: `OFFSET 0`, and `OFFSET` with no `LIMIT`,
where the skip is not pushed into the scan at all. It also gets a json table:
`ListingTable::supports_skip_pushdown` is unconditional and the
`GlobalLimitExec` wrap only looks at `skip`, so the defect is not specific to
parquet, and every other query in the file is. `file_row_index()` itself is
parquet-only, so that section uses `input_file_name()`.

`projection_pushdown.slt` gets two value queries and four `EXPLAIN`s covering
a bare leaf, a leaf beside a computed sibling, a leaf nested inside
`upper(...)`, and the no-leaf case. The value queries pass before the fix on
purpose: they pin the rows, which is the one thing a plan assertion cannot
show. A dropped or swapped `skip` turns the plan text red as well, so they are
not the only cover for it. The no-leaf `EXPLAIN` is labelled a control rather
than a guard, because the scan prunes to `[id]` whether or not the rule bails
out early; the unit test is what pins the bail-out. A further value query puts
the marker inside a lambda body under an `OFFSET`: the lambda queries above it
have no limit and the limit queries have no lambda, so nothing covered both,
and it is also what pins the parameter rebase.

Both files get a section with two file groups, which is the shape that stacks
a `CoalescePartitionsExec` under the limit. Those count the surviving rows
instead of listing them, because which rows survive the skip depends on the
order the partitions are merged in, and `count(r)` rather than `count(*)`, so
`optimize_projections` cannot drop the marker and leave nothing to evaluate.

Neither new section contains a `RepartitionExec`, so both exercise this path
rather than the one apache#27152 fixed.

Ten unit tests hand-build the tree, because plain SQL does not reach these
shapes: the split committing with both of the limit's numbers intact, the
chain descending past two hosts and past three with a distinct pair of
numbers on each, a `CoalescePartitionsExec` host keeping its own `fetch`, a
`fetch` of `None`, a projection with no leaf expression left alone,
`LocalLimitExec` kept out of the host set, a source that refuses the lower
projection, the three nodes that meet the host conditions but are left out of
the allowlist appearing instead as the chain bottom, and a `FilterExec`
bottom, which takes the lower projection without absorbing it so the scan
narrows only on this rule's second scheduled run.

Part of apache#27131

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Nov 13, 2026
jayzhan211
jayzhan211 previously approved these changes Nov 14, 2026

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cool9850311 thanks for following up on the OFFSET row! Looks good to me. The three inline comments are optional.

Co-authored-by: Jay Zhan <jayzhan211@gmail.com>

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @cool9850311 and @jayzhan211 -- the code looks ok but I think the comments make this really hard to understand. Can you please cut them down (by reading all comments, and removing anything you don't understand yourself)

I also have some ideas on how to improve the tests

Thank you for the contribution, and sorry for the request, I just think we need to fight to keep DataFusion understandable

#[tokio::test]
async fn split_descends_through_global_limit() -> Result<()> {
let (schema, scan) = limit_split_scan()?;
// What a source that pushes `OFFSET` down builds: the skip becomes a

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand these comments -- they are trying to describe in english the plan shape

I think these tests would be much more readable if instead of trying to describe this in textual comments, you instead add a snapshot of the orignal and the optimized plan

Sort of like

   assert_snapshot!(input_and_optimized_plan_text(projection)?,
        @r"
    ---  Input Plan
...
 
    ---  OutputPlan
    ProjectionExec: expr=[__datafusion_leaf_0@1 as v, id@0 + 1 as w]
      GlobalLimitExec: skip=1, fetch=3
        DataSourceExec: file_groups={1 group: [[x]]}, projection=[id, get_field(s@1, value) as __datafusion_leaf_0], file_type=parquet
    "

More background on the rationale here: https://andrew.nerdnetworks.org/blog/2026/11/13/transcript-testing/

// a plain column, and is then no narrower than the widened child, or it
// narrows and holds a computed one. `RepartitionExec` declines both, it
// is not removable because it renames, and nothing else moved.
// stuck above a chain of the nodes `split_host_child` accepts. The

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please update these comments to not refer to implementation details elsewere -- pelase try and rephrase in your own words

/// changed the width would leave both against the wrong one.
fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> {
if let Some(repartition) = node.downcast_ref::<RepartitionExec>() {
// Hash and Range partitioning reference columns by index, which would

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

likewise, these comments is far too verbise -- please rewrite these in your own words rather than leaving this re-explanation of the code in

/// expression sharing it never reaches the source. Splitting leaves the
/// computation above the repartition, still parallelised.
/// Two nodes strand a `MoveTowardsLeafNodes` expression above the source this
/// way, and neither lets the whole projection past:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please reword

@@ -314,6 +314,191 @@ SELECT file_row_index() + 1, column1 FROM parquet_table ORDER BY column1
# `RESET` restores the global default rather than the harness value, which the
# harness rejects as leftover configuration.

# A leaf expression must also reach the scan when the source pushes an `OFFSET`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please reword these comments to explain the case they are teseting, not details about the code that implements it?

@cool9850311

Copy link
Copy Markdown
Contributor Author

Thanks @cool9850311 and @jayzhan211 -- the code looks ok but I think the comments make this really hard to understand. Can you please cut them down (by reading all comments, and removing anything you don't understand yourself)

I also have some ideas on how to improve the tests

Thank you for the contribution, and sorry for the request, I just think we need to fight to keep DataFusion understandable

Sorry for not paying attention on readability of comments and tests, will work on this.

…ments down

Addresses the review on the tests and on the comments.

Tests: the ten split tests asserted only the optimized plan, through a local
`optimized_plan_text` helper. They now use `OptimizationTest`, which this test
directory already has (`pushdown_sort.rs` uses it) and which renders the input
plan and the output side by side, so a reader sees what the rule changed
without opening the test body. It also compares the plan's schema before and
after whenever the rule's `schema_check` is true, which `ProjectionPushdown`
sets, so each of the ten now carries a schema assertion as well. Two tests
needed restructuring to fit: the pass-through-bottom case was a loop over two
node types, which inline snapshots cannot be, and is now two sequential
assertions; the filter case takes one snapshot per scheduled run of the rule.
The same change removes the inline snapshot formatting that the earlier review
round asked about.

Comments: the first pass measured the surroundings but did not treat the
measurement as a ceiling. The second pass asks of every block whether it can
be removed rather than whether it can be shortened, keeping one only if a
reader would otherwise misread what the code does, or would make a change that
breaks something no test catches.

What went from the rule: the narration of `optimize()`'s two passes, which the
two lines say already; the `split_host_child` and `split_host_chain` doc
comments, since the schema pass-through is enforced by `schema_check` and the
multi-host chain is stated by `split_descends_through_limit_and_coalesce`; the
allowlist note, which `split_declines_local_limit` now carries alone; the
`SplitHostChain` type and `child` field docs; the termination guard's note,
which claimed non-termination the suite does not demonstrate; and the
paragraph on what strands a leaf expression, which is context for the pull
request rather than for the function.

What stayed is the five facts not inferable from the code: why only
round-robin partitioning qualifies, why `CoalescePartitionsExec` is in the
chain, that `hosts` runs outermost first, that a `Some` from
`try_swapping_with_projection` does not mean the source absorbed the
projection, and why the hosts' properties are recomputed. In the tests and the
`.slt` files the helper docs and the sentences restating a neighbouring query
went; the ones that keep a test from looking redundant, or that explain a
deliberately weak assertion, stayed.

The termination guard's comment was also the one the review asked to reword.
Measuring it first showed the claim was wrong: removing the guard does not
hang the suite, it changes one test's output, so the comment went rather than
being reworded.

Rule: 78 comment lines down to 12, hunk density 64.5% to 21.8%, whole-file
11.9% to 9.4% against a 10.4% baseline, and the longest block this branch adds
down from 15 lines to 4 (the file's own maximum is 7). Tests 17.7% to 6.0%,
projection_pushdown.slt 18.1% to 10.4%, file_row_index.slt 22.6% to 14.4%.

cargo fmt --check clean, clippy clean on datafusion-physical-optimizer
--all-targets and on the core_integration test target, 41 projection pushdown
tests and 51 physical-optimizer unit tests passing, both touched .slt files
green, and the full sqllogictest suite green before the comment pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- GlobalLimitExec: skip=1, fetch=3
- CooperativeExec
- DataSourceExec: file_groups={1 group: [[x]]}, projection=[id, get_field(s@1, value) as __datafusion_leaf_0], file_type=parquet
"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb
Is it what you want?

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cool9850311 thanks for cutting the comments down, the input/output snapshots read much better! My approval stands. The four inline suggestions are optional comment tweaks: two put back a reason the cuts removed, two replace "corpus".

format!("{LEAF_COLUMN_PREFIX}_{index}")
}

fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: with the old doc gone, nothing says what a node has to satisfy to be allowed here, and the tests now say "host" without a definition. One short reason would keep the next person from adding, say, a SortExec arm, whose sort expressions would point at the wrong columns after the split:

Suggested change
fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> {
/// Returns the input of `node` if the split may place the leaf projection below
/// it. The split gives `node` an input with different columns, so `node` must
/// not reference its input's columns and must output them unchanged, or the
/// outer projection's column indices would point at the wrong columns.
fn split_host_child(node: &Arc<dyn ExecutionPlan>) -> Option<&Arc<dyn ExecutionPlan>> {

Comment on lines 527 to 528
/// Splits a `ProjectionExec` that mixes leaf expressions with ones that must
/// stay in place, moving the leaves into a projection the source absorbs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: main's doc here explained why the split is needed at all, and that reason is gone now. A short version that also covers the new limit case:

Suggested change
/// Splits a `ProjectionExec` that mixes leaf expressions with ones that must
/// stay in place, moving the leaves into a projection the source absorbs.
/// Splits a `ProjectionExec` that mixes leaf expressions with ones that must
/// stay in place, moving the leaves into a projection the source absorbs.
/// Without the split the leaves stay above the source: no projection swaps past
/// a `GlobalLimitExec`, and none with a computed expression swaps past a
/// `RepartitionExec`.

Comment on lines +2144 to +2146
/// `LocalLimitExec` is kept out of the host set. The corpus never strands a
/// leaf above one, but that is what the corpus shows, not a proof that nothing
/// can, so this pins the exclusion.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: "corpus" isn't defined anywhere, so a reader can't tell which queries it means. Suggested wording:

Suggested change
/// `LocalLimitExec` is kept out of the host set. The corpus never strands a
/// leaf above one, but that is what the corpus shows, not a proof that nothing
/// can, so this pins the exclusion.
/// `LocalLimitExec` is left out of `split_host_child`: no SQL query strands a
/// leaf expression above one today. This pins that choice.

Comment on lines +2263 to +2264
/// Six numbers have to come back in the right places, not two. The corpus
/// reaches a chain this deep but never with a leaf to move.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional cleanup: same "corpus" wording here. The snapshot already shows the chain, so the comment only needs to say what is checked:

Suggested change
/// Six numbers have to come back in the right places, not two. The corpus
/// reaches a chain this deep but never with a leaf to move.
/// Each of the three limits keeps its own `skip` and `fetch` after the rebuild.

…tisfy

Round 2 of review feedback, all four comment-only:

- `split_host_child` had no doc after the previous cut, so nothing said what a
  node has to satisfy to be listed. A `SortExec` arm would compile and no test
  would catch it, because the arms are an allowlist and an unlisted node is
  never reached. The doc also states that satisfying the condition is necessary
  rather than sufficient: `LocalLimitExec`, `CooperativeExec` and `BufferExec`
  all satisfy it and are deliberately left out.
- `try_split_leaf_projection` lost the reason the split exists at all, which
  main documented before this PR. Restored in one sentence that covers both the
  repartition and the limit case.
- the two test doc comments said "corpus", which is not defined anywhere, and
  one of them claimed a chain depth the test corpus was never measured for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cool9850311

Copy link
Copy Markdown
Contributor Author

@cool9850311 thanks for cutting the comments down, the input/output snapshots read much better! My approval stands. The four inline suggestions are optional comment tweaks: two put back a reason the cuts removed, two replace "corpus".

Done.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand the need for this PR.

If you want to proceed, please clean up the ticket to explain the symptom you are solving, not all the internal details of what you think is a problem

The #27131 claims the query fails

COPY (SELECT value AS column1 FROM range(0, 20000))
TO '/tmp/df_leaf_repro/big.parquet' STORED AS PARQUET;

CREATE EXTERNAL TABLE big
STORED AS PARQUET LOCATION '/tmp/df_leaf_repro/big.parquet';

SELECT file_row_index(), column1 + 1 FROM big;   -- Execution error

However, when I ran it it works find

SELECT file_row_index(), column1 + 1 FROM big;
+------------------+------------------------+
| file_row_index() | big.column1 + Int64(1) |
+------------------+------------------------+
| 0                | 1                      |
| 1                | 2                      |
| 2                | 3                      |
| 3                | 4                      |
| 4                | 5                      |
| 5                | 6                      |
| 6                | 7                      |
| 7                | 8                      |
| 8                | 9                      |
| 9                | 10                     |
| 10               | 11                     |
| 11               | 12                     |
| 12               | 13                     |
| 13               | 14                     |
| 14               | 15                     |
| 15               | 16                     |
| 16               | 17                     |
| 17               | 18                     |
| 18               | 19                     |
| 19               | 20                     |
| 20               | 21                     |
| 21               | 22                     |
| 22               | 23                     |
| 23               | 24                     |
| 24               | 25                     |
| 25               | 26                     |
| 26               | 27                     |
| 27               | 28                     |
| 28               | 29                     |
| 29               | 30                     |
| 30               | 31                     |
| 31               | 32                     |
| 32               | 33                     |
| 33               | 34                     |
| 34               | 35                     |
| 35               | 36                     |
| 36               | 37                     |
| 37               | 38                     |
| 38               | 39                     |
| 39               | 40                     |
| .                                         |
| .                                         |
| .                                         |
+------------------+------------------------+
20000 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 0.004 seconds.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cool9850311 thanks for putting those two reasons back. alamb's last review asks for the symptom rather than the internals, and the PR description still opens with the rule-by-rule walk-through, line numbers and mutation tables (about 10k characters). Could you cut it down to the template's sections, along the lines below? I checked both issues' repros: on this PR's base SELECT file_row_index() FROM small OFFSET 1 LIMIT 2 errors and s['a'] under OFFSET 5 LIMIT 3 scans projection=[s]. On this branch the first returns 1, 2 and the scan reads projection=[get_field(s@0, a) ...].

Suggested shorter PR description
## Which issue does this PR close?

Closes #27494.
Closes #27495.

## Rationale for this change

With `OFFSET n LIMIT m` on a listing table, `file_row_index()` and `input_file_name()` fail, and `s['a']` makes the scan read the whole struct:

```sql
SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
-- Execution error: file_row_index() is source dependent and cannot be evaluated directly
```

The skip pushed into the scan puts a `GlobalLimitExec` (and, with several files, a `CoalescePartitionsExec`) between the scan and the projection holding these expressions, and no projection moves below a `GlobalLimitExec`.

## What changes are included in this PR?

The leaf-projection split that #27152 added for a round-robin `RepartitionExec` now also moves past `GlobalLimitExec` and `CoalescePartitionsExec`, including several stacked. Only the leaf expressions move below the limit; computed ones stay above it, so the rows the `OFFSET` skips don't evaluate them.

## Are these changes tested?

Yes: value and `EXPLAIN` queries in `file_row_index.slt` and `projection_pushdown.slt`, and unit tests in `core/tests/physical_optimizer/projection_pushdown.rs` showing each input and output plan.

## Are there any user-facing changes?

These queries now return rows instead of an error, and `s['a']` under `OFFSET` reads only that field. No API change.

@jayzhan211
jayzhan211 dismissed their stale review November 17, 2026 13:10

Dismissing my earlier approval; see my latest review comment.

@cool9850311

Copy link
Copy Markdown
Contributor Author

@cool9850311 thanks for putting those two reasons back. alamb's last review asks for the symptom rather than the internals, and the PR description still opens with the rule-by-rule walk-through, line numbers and mutation tables (about 10k characters). Could you cut it down to the template's sections, along the lines below? I checked both issues' repros: on this PR's base SELECT file_row_index() FROM small OFFSET 1 LIMIT 2 errors and s['a'] under OFFSET 5 LIMIT 3 scans projection=[s]. On this branch the first returns 1, 2 and the scan reads projection=[get_field(s@0, a) ...].

Suggested shorter PR description

## Which issue does this PR close?

Closes #27494.
Closes #27495.

## Rationale for this change

With `OFFSET n LIMIT m` on a listing table, `file_row_index()` and `input_file_name()` fail, and `s['a']` makes the scan read the whole struct:

```sql
SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
-- Execution error: file_row_index() is source dependent and cannot be evaluated directly

The skip pushed into the scan puts a GlobalLimitExec (and, with several files, a CoalescePartitionsExec) between the scan and the projection holding these expressions, and no projection moves below a GlobalLimitExec.

What changes are included in this PR?

The leaf-projection split that #27152 added for a round-robin RepartitionExec now also moves past GlobalLimitExec and CoalescePartitionsExec, including several stacked. Only the leaf expressions move below the limit; computed ones stay above it, so the rows the OFFSET skips don't evaluate them.

Are these changes tested?

Yes: value and EXPLAIN queries in file_row_index.slt and projection_pushdown.slt, and unit tests in core/tests/physical_optimizer/projection_pushdown.rs showing each input and output plan.

Are there any user-facing changes?

These queries now return rows instead of an error, and s['a'] under OFFSET reads only that field. No API change.

Done.

@alamb

alamb commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Thanks, we are getting better here.

Now that we have broken the symptoms into two separate tickets, can you pleaes break this PR into two separate / more focused PRs

One for the fix / feature #27494 and the other for the new pushdown feature

The goal is to make it as easy for you and us to review this PR as possible

@cool9850311

Copy link
Copy Markdown
Contributor Author

Thanks, we are getting better here.

Now that we have broken the symptoms into two separate tickets, can you pleaes break this PR into two separate / more focused PRs

One for the fix / feature #27494 and the other for the new pushdown feature

The goal is to make it as easy for you and us to review this PR as possible

It's a little bit tricky, the root cause of them are actually the same, that's why this PR close both issues.
what should I do if the root cause of them are the same?

@cool9850311

Copy link
Copy Markdown
Contributor Author

Thanks, we are getting better here.

Now that we have broken the symptoms into two separate tickets, can you pleaes break this PR into two separate / more focused PRs

One for the fix / feature #27494 and the other for the new pushdown feature

The goal is to make it as easy for you and us to review this PR as possible

@alamb
The two issues are one bug seen twice. Both plans come out node-for-node identical:

-- #27494  SELECT file_row_index() AS r FROM small OFFSET 1 LIMIT 2
ProjectionExec: expr=[file_row_index() as r]
  GlobalLimitExec: skip=1, fetch=2
    DataSourceExec: file_groups={1 group: [[small.parquet]]}, limit=3

-- #27495  SELECT s['a'] AS v FROM t OFFSET 5 LIMIT 3
ProjectionExec: expr=[get_field(s@0, a) as v]
  GlobalLimitExec: skip=5, fetch=3
    DataSourceExec: file_groups={1 group: [[st.parquet]]}, projection=[s], limit=8

The leaf projection is planted on the TableScan, physical planning inserts a GlobalLimitExec between them for the pushed-down skip, and no projection can move below that node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OFFSET n LIMIT m stops get_field() from being pushed into the file scan file_row_index() / input_file_name() fail under OFFSET n LIMIT m

3 participants