Skip to content

[branch-54] Disable memory-limited (spilling) execution of NestedLoopJoinExec - #27562

Open
kamilk-pal wants to merge 2 commits into
apache:branch-54from
kamilk-pal:disable-nlj-memory-limited-execution-branch-54
Open

kamilk-pal wants to merge 2 commits into
apache:branch-54from
kamilk-pal:disable-nlj-memory-limited-execution-branch-54

Conversation

@kamilk-pal

@kamilk-pal kamilk-pal commented Nov 18, 2026 •

Copy link
Copy Markdown

Part of #27608

Rationale for this change

The memory-limited (spilling) fallback of NestedLoopJoinExec, added in 54.0.0 by #21448, can panic or silently return incorrect results on branch-54. Fixing it on 54.x would take at least three bug fixes:

These sit on top of continued rework of the feature on main: #21636, #21833, #22038, #25004, #25533, #25542. More reports keep arriving, e.g. #25701 (an empty result instead of spilling, reported on 55.1.0). All of it is first released in 56.0.0.

Rather than backporting that series, this PR disables the fallback on branch-54. A nested loop join whose build side does not fit in memory fails with ResourcesExhausted, as it did before 54.0.0.

What changes are included in this PR?

  • ENABLE_MEMORY_LIMITED_EXECUTION = false in nested_loop_join.rs, checked where execute decides whether it may fall back. The fallback code itself is unchanged.
  • Tests that need the fallback are #[ignore]d, a new test checks that every join type fails with ResourcesExhausted, and nested_loop_join_spill.slt expects the error.

Are there any user-facing changes?

Yes. On 54.x, a nested loop join whose build side exceeds the memory limit fails with ResourcesExhausted instead of spilling.

kamilk-pal and others added 2 commits November 18, 2026 17:41
…JoinExec

The memory-limited fallback added in apache#21448 returns incorrect results or
panics: it re-executes the build side (apache#24676), emits wrong unmatched rows
for LEFT-type joins with a multi-partition probe side (apache#22641), and drops
unmatched probe-side rows (apache#24745). The fixes (apache#24677, apache#24675, apache#24746) build
on further rework of the feature on main and are first released in 56.0.0.

Instead of backporting that series, disable the fallback on branch-54: a
build side that does not fit in memory fails with ResourcesExhausted, as
before 54.0.0. Tests that need the fallback are ignored, and
nested_loop_join_spill.slt now expects the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@hareshkh hareshkh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me given this can produce incorrect data! This can stay cherry-picked and become a part of the next release if any is cut from branch-54

@kamilk-pal
kamilk-pal marked this pull request as ready for review November 19, 2026 11:29
@kamilk-pal

Copy link
Copy Markdown
Author

Hey @alamb, I wonder if you could review?

I'm conscious that a minor release 54.2.0 off branch-54 was just released this week: #23805 . Preferably we'd have a new 54.x release with this fix; just merging the fix to branch-54 without cutting a new release is valuable too.

@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

I'm conscious that a minor release 54.2.0 off branch-54 was just released this week: #23805 . Preferably we'd have a new 54.x release with this fix; just merging the fix to branch-54 without cutting a new release is valuable too.

Thank you -- if you want to make a new minor release, can you please file a ticket to track it following the existing convention?

I agree backporting stuff to branch-54 is valuable

@alamb

alamb commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Rather than backporting that series, this PR disables the fallback on branch-54. A nested loop join whose build side does not fit in memory fails with ResourcesExhausted, as it did before 54.0.0.

That seems like a non trivial change in my mind and I worry some users woudl experience this as a regression. Perhaps we can add a specific knob disable spilling just in this case so that users who are hitting issues can turn it off

@kamilk-pal

Copy link
Copy Markdown
Author

Created a github issue for a new branch-54 release: Release DataFusion 54.3.0 (minor)

@kamilk-pal

Copy link
Copy Markdown
Author

That seems like a non trivial change in my mind and I worry some users woudl experience this as a regression. Perhaps we can add a specific knob disable spilling just in this case so that users who are hitting issues can turn it off

On 54.2.0, when the build side doesn’t fit and the join spills, the result is one of:

  1. correct: e.g. an INNER join over an in-memory build side (it still executes the build side twice)
  2. panic: e.g. a build side with a hash RepartitionExec, which panics with partition not used yet when executed a second time
  3. incorrect, with no error: e.g. a multi-file Parquet build side, which loses rows on the second execution

So it’s a regression for the first case, a different error for the second, and a correctness fix for the third. Because of the third case I believe it's best to disable the feature by default.

Regarding configurability: unfortunately it'd result in a semver breaking change. I created a github issue to discuss it further: config_namespace! structs lost #[non_exhaustive], so adding a config option is a breaking change. Given this context, I wonder what's the recommended next step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants