Skip to content

test: parallelize clustered grouping test coverage - #26839

Merged
kosiew merged 2 commits into
apache:mainfrom
bvolpato:bvolpato/clustered-test-collision-scope
Nov 19, 2026
Merged

kosiew merged 2 commits into
apache:mainfrom
bvolpato:bvolpato/clustered-test-collision-scope

Conversation

@bvolpato

@bvolpato bvolpato commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Important

Fork CI setup: For pull requests from a fork to main, enable workflows in your fork's Actions tab before you push.
See the fork CI setup guide.

Which issue does this PR close?

Part of #26759 and #13813.

Rationale for this change

After #26925 scaled smaller batch sizes in clustered_keys_match_hash_grouping_across_batches_and_emits to 129.max(batch_size + 1), batch_size = 1024 still builds 1025 rows and the test still runs in the hash-collisions CI job. Under force_hash_collisions, the hash-based GroupValuesColumn reference becomes quadratic even though GroupValuesClustered does no hashing. In regular runs, each sort-direction case also evaluates every key type sequentially on a single thread.

What changes are included in this PR?

Skip clustered_keys_match_hash_grouping_across_batches_and_emits and its exclusive helper and imports when force_hash_collisions is enabled, matching #26759. Other clustered grouping unit tests remain active in collision mode.

Split the key types into deterministic shards so the test harness schedules them in parallel during regular and extended runs. Every key type, nullability, sort direction, batch size, and emit limit combination is preserved.

What is the testing strategy for this PR?

On current main (after #26925) on a shared Linux host:

  • With force_hash_collisions enabled across datafusion-physical-plan and datafusion-common (matching cargo xtask ci step test hash-collisions), runtime for the clustered module drops from 14.4s to 0.06s while keeping the remaining clustered unit tests active.
  • Regular --test-threads=8 wall time for clustered_keys_match_hash_grouping_across_batches_and_emits drops from 3.0s to 0.8s.

Run both modes with:

cargo test --profile ci -p datafusion-physical-plan --lib clustered_keys_match_hash_grouping_across_batches_and_emits -- --test-threads=8
cargo test --profile ci -p datafusion-physical-plan --lib --features force_hash_collisions,datafusion-common/force_hash_collisions aggregates::group_values::multi_group_by::clustered::tests

Are there any user-facing changes?

None. This changes test scheduling and skips redundant collision-mode reference hashing.

AI-assisted contribution.

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Oct 28, 2026
@adriangb

Copy link
Copy Markdown
Contributor

happy to review this once it's ready

@bvolpato
bvolpato force-pushed the bvolpato/clustered-test-collision-scope branch from 982238c to 175a936 Compare October 29, 2026 00:09
@bvolpato
bvolpato marked this pull request as ready for review October 30, 2026 16:08
@bvolpato

Copy link
Copy Markdown
Contributor Author

Marked this ready for review now that #26840 has landed. Thanks.

@bvolpato
bvolpato force-pushed the bvolpato/clustered-test-collision-scope branch from 175a936 to d426aab Compare November 14, 2026 22:42

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bvolpato, thanks for addressing the clustered grouping test overhead. The change appropriately partitions the existing key-type matrix into independently schedulable cases while preserving default-build coverage and assertions.

The feature gate is justified because GroupValuesClustered does not hash, whereas the GroupValuesColumn reference incurs quadratic costs under forced collisions. Excluding this comparison avoids unrelated CI overhead without changing production behavior or public APIs. Other clustered grouping tests remain active in collision builds.

I found no blocking issues. Focused tests pass in both configurations, and all 54 key types are covered exactly once per sort-direction case. The scheduling improvement is structurally sound, although no before/after speedup was measured. I have one optional iteration simplification below. Approving.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bvolpato, thanks for the updates. The test sharding preserves coverage while reducing runtime, and excluding the comparison from forced-collision builds is appropriate.

The follow-up also resolves my earlier suggestion. Approved.

@kosiew

kosiew commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

🚀
@bvolpato
Thank you for your contribution.

@kosiew
kosiew added this pull request to the merge queue Nov 19, 2026
Merged via the queue into apache:main with commit 7d34f56 Nov 19, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants