Repository navigation
test: parallelize clustered grouping test coverage - #26839
Conversation
|
happy to review this once it's ready |
982238c to
175a936
Compare
|
Marked this ready for review now that #26840 has landed. Thanks. |
175a936 to
d426aab
Compare
kosiew
left a comment
There was a problem hiding this comment.
@bvolpato, thanks for addressing the clustered grouping test overhead. The change appropriately partitions the existing key-type matrix into independently schedulable cases while preserving default-build coverage and assertions.
The feature gate is justified because GroupValuesClustered does not hash, whereas the GroupValuesColumn reference incurs quadratic costs under forced collisions. Excluding this comparison avoids unrelated CI overhead without changing production behavior or public APIs. Other clustered grouping tests remain active in collision builds.
I found no blocking issues. Focused tests pass in both configurations, and all 54 key types are covered exactly once per sort-direction case. The scheduling improvement is structurally sound, although no before/after speedup was measured. I have one optional iteration simplification below. Approving.
kosiew
left a comment
There was a problem hiding this comment.
@bvolpato, thanks for the updates. The test sharding preserves coverage while reducing runtime, and excluding the comparison from forced-collision builds is appropriate.
The follow-up also resolves my earlier suggestion. Approved.
|
🚀 |
Important
Fork CI setup: For pull requests from a fork to
main, enable workflows in your fork's Actions tab before you push.See the fork CI setup guide.
Which issue does this PR close?
Part of #26759 and #13813.
Rationale for this change
After #26925 scaled smaller batch sizes in
clustered_keys_match_hash_grouping_across_batches_and_emitsto129.max(batch_size + 1),batch_size = 1024still builds1025rows and the test still runs in thehash-collisionsCI job. Underforce_hash_collisions, the hash-basedGroupValuesColumnreference becomes quadratic even thoughGroupValuesClustereddoes no hashing. In regular runs, each sort-direction case also evaluates every key type sequentially on a single thread.What changes are included in this PR?
Skip
clustered_keys_match_hash_grouping_across_batches_and_emitsand its exclusive helper and imports whenforce_hash_collisionsis enabled, matching #26759. Other clustered grouping unit tests remain active in collision mode.Split the key types into deterministic shards so the test harness schedules them in parallel during regular and extended runs. Every key type, nullability, sort direction, batch size, and emit limit combination is preserved.
What is the testing strategy for this PR?
On current
main(after #26925) on a shared Linux host:force_hash_collisionsenabled acrossdatafusion-physical-plananddatafusion-common(matchingcargo xtask ci step test hash-collisions), runtime for theclusteredmodule drops from 14.4s to 0.06s while keeping the remaining clustered unit tests active.--test-threads=8wall time forclustered_keys_match_hash_grouping_across_batches_and_emitsdrops from 3.0s to 0.8s.Run both modes with:
Are there any user-facing changes?
None. This changes test scheduling and skips redundant collision-mode reference hashing.
AI-assisted contribution.