Repository navigation
minor: speedup one slow aggregate unit test - #26925
Merged
Merged
Conversation
2010YOUY01
commented
Oct 30, 2026
alamb
approved these changes
Oct 30, 2026
alamb
left a comment
Contributor
There was a problem hiding this comment.
Thank you @2010YOUY01 -- this makes sense to me
abstractdog
pushed a commit
to abstractdog/datafusion
that referenced
this pull request
Nov 19, 2026
> [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close? Part of apache#26759 and apache#13813. ## Rationale for this change After apache#26925 scaled smaller batch sizes in `clustered_keys_match_hash_grouping_across_batches_and_emits` to `129.max(batch_size + 1)`, `batch_size = 1024` still builds `1025` rows and the test still runs in the `hash-collisions` CI job. Under `force_hash_collisions`, the hash-based `GroupValuesColumn` reference becomes quadratic even though `GroupValuesClustered` does no hashing. In regular runs, each sort-direction case also evaluates every key type sequentially on a single thread. ## What changes are included in this PR? Skip `clustered_keys_match_hash_grouping_across_batches_and_emits` and its exclusive helper and imports when `force_hash_collisions` is enabled, matching apache#26759. Other clustered grouping unit tests remain active in collision mode. Split the key types into deterministic shards so the test harness schedules them in parallel during regular and extended runs. Every key type, nullability, sort direction, batch size, and emit limit combination is preserved. ## What is the testing strategy for this PR? On current `main` (after apache#26925) on a shared Linux host: - With `force_hash_collisions` enabled across `datafusion-physical-plan` and `datafusion-common` (matching `cargo xtask ci step test hash-collisions`), runtime for the `clustered` module drops from 14.4s to 0.06s while keeping the remaining clustered unit tests active. - Regular `--test-threads=8` wall time for `clustered_keys_match_hash_grouping_across_batches_and_emits` drops from 3.0s to 0.8s. Run both modes with: ```shell cargo test --profile ci -p datafusion-physical-plan --lib clustered_keys_match_hash_grouping_across_batches_and_emits -- --test-threads=8 cargo test --profile ci -p datafusion-physical-plan --lib --features force_hash_collisions,datafusion-common/force_hash_collisions aggregates::group_values::multi_group_by::clustered::tests ``` ## Are there any user-facing changes? None. This changes test scheduling and skips redundant collision-mode reference hashing. AI-assisted contribution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Fix one slow test by scaling it down, while preserve all existing test goal.
Some more high-level thoughts beyond this PR:
I've found recently there are many very expensive UTs added, it would be better we can notify them in CI, and try to either
What changes are included in this PR?
The actual change is 2 lines, though the diff is larger. The test is a many layer nested loop to cover different combinations:
for batch_size inloop upperrows, it's decided based onbatch_sizeno
assert!changed so the testing goal has been preserved.What is the testing strategy for this PR?
Are there any user-facing changes?