Repository navigation
[branch-54] Disable memory-limited (spilling) execution of NestedLoopJoinExec - #27562
kamilk-pal wants to merge 2 commits into
Conversation
…JoinExec The memory-limited fallback added in apache#21448 returns incorrect results or panics: it re-executes the build side (apache#24676), emits wrong unmatched rows for LEFT-type joins with a multi-partition probe side (apache#22641), and drops unmatched probe-side rows (apache#24745). The fixes (apache#24677, apache#24675, apache#24746) build on further rework of the feature on main and are first released in 56.0.0. Instead of backporting that series, disable the fallback on branch-54: a build side that does not fit in memory fails with ResourcesExhausted, as before 54.0.0. Tests that need the fallback are ignored, and nested_loop_join_spill.slt now expects the error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hareshkh
left a comment
There was a problem hiding this comment.
This makes sense to me given this can produce incorrect data! This can stay cherry-picked and become a part of the next release if any is cut from branch-54
Thank you -- if you want to make a new minor release, can you please file a ticket to track it following the existing convention? I agree backporting stuff to branch-54 is valuable |
That seems like a non trivial change in my mind and I worry some users woudl experience this as a regression. Perhaps we can add a specific knob disable spilling just in this case so that users who are hitting issues can turn it off |
|
Created a github issue for a new branch-54 release: Release DataFusion 54.3.0 (minor) |
On 54.2.0, when the build side doesn’t fit and the join spills, the result is one of:
So it’s a regression for the first case, a different error for the second, and a correctness fix for the third. Because of the third case I believe it's best to disable the feature by default. Regarding configurability: unfortunately it'd result in a semver breaking change. I created a github issue to discuss it further: config_namespace! structs lost #[non_exhaustive], so adding a config option is a breaking change. Given this context, I wonder what's the recommended next step. |
Part of #27608
Rationale for this change
The memory-limited (spilling) fallback of
NestedLoopJoinExec, added in 54.0.0 by #21448, can panic or silently return incorrect results onbranch-54. Fixing it on 54.x would take at least three bug fixes:RepartitionExecand loses rows from file scans (NestedLoopJoin memory-limited fallback re-executes the build side, losing rows for non-replayable inputs #24676)These sit on top of continued rework of the feature on
main: #21636, #21833, #22038, #25004, #25533, #25542. More reports keep arriving, e.g. #25701 (an empty result instead of spilling, reported on 55.1.0). All of it is first released in 56.0.0.Rather than backporting that series, this PR disables the fallback on
branch-54. A nested loop join whose build side does not fit in memory fails withResourcesExhausted, as it did before 54.0.0.What changes are included in this PR?
ENABLE_MEMORY_LIMITED_EXECUTION = falseinnested_loop_join.rs, checked whereexecutedecides whether it may fall back. The fallback code itself is unchanged.#[ignore]d, a new test checks that every join type fails withResourcesExhausted, andnested_loop_join_spill.sltexpects the error.Are there any user-facing changes?
Yes. On 54.x, a nested loop join whose build side exceeds the memory limit fails with
ResourcesExhaustedinstead of spilling.