Skip to content

minor: speedup one slow aggregate unit test - #26925

Merged
alamb merged 2 commits into
apache:mainfrom
2010YOUY01:test-speed/clustered-grouping
Oct 30, 2026
Merged

alamb merged 2 commits into
apache:mainfrom
2010YOUY01:test-speed/clustered-grouping

Conversation

@2010YOUY01

Copy link
Copy Markdown
Member

Which issue does this PR close?

  • Closes #.

Rationale for this change

Fix one slow test by scaling it down, while preserve all existing test goal.

cargo nextest run -p datafusion-physical-plan --lib --test-threads 1 clustered_keys_match_hash_grouping_across_batches_and_emits

Sorted:   25.180s → 3.782s
Unsorted: 25.497s → 3.766s

Some more high-level thoughts beyond this PR:

I've found recently there are many very expensive UTs added, it would be better we can notify them in CI, and try to either

  • scale down
  • move to extended tests
  • make the fuzzing configurable by time budget

What changes are included in this PR?

The actual change is 2 lines, though the diff is larger. The test is a many layer nested loop to cover different combinations:

  1. move the for batch_size in loop upper
  2. scale down rows, it's decided based on batch_size

no assert! changed so the testing goal has been preserved.

What is the testing strategy for this PR?

Are there any user-facing changes?

@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label Oct 30, 2026

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @2010YOUY01 -- this makes sense to me

@alamb
alamb added this pull request to the merge queue Oct 30, 2026
Merged via the queue into apache:main with commit 0ad5049 Oct 30, 2026
44 checks passed
abstractdog pushed a commit to abstractdog/datafusion that referenced this pull request Nov 19, 2026
> [!IMPORTANT]
>
> **Fork CI setup:** For pull requests from a fork to `main`, enable
workflows in your fork's **Actions** tab before you push.
> See the [fork CI setup
guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks).

## Which issue does this PR close?

Part of apache#26759 and apache#13813.

## Rationale for this change

After apache#26925 scaled smaller batch sizes in
`clustered_keys_match_hash_grouping_across_batches_and_emits` to
`129.max(batch_size + 1)`, `batch_size = 1024` still builds `1025` rows
and the test still runs in the `hash-collisions` CI job. Under
`force_hash_collisions`, the hash-based `GroupValuesColumn` reference
becomes quadratic even though `GroupValuesClustered` does no hashing. In
regular runs, each sort-direction case also evaluates every key type
sequentially on a single thread.

## What changes are included in this PR?

Skip `clustered_keys_match_hash_grouping_across_batches_and_emits` and
its exclusive helper and imports when `force_hash_collisions` is
enabled, matching apache#26759. Other clustered grouping unit tests remain
active in collision mode.

Split the key types into deterministic shards so the test harness
schedules them in parallel during regular and extended runs. Every key
type, nullability, sort direction, batch size, and emit limit
combination is preserved.

## What is the testing strategy for this PR?

On current `main` (after apache#26925) on a shared Linux host:

- With `force_hash_collisions` enabled across `datafusion-physical-plan`
and `datafusion-common` (matching `cargo xtask ci step test
hash-collisions`), runtime for the `clustered` module drops from 14.4s
to 0.06s while keeping the remaining clustered unit tests active.
- Regular `--test-threads=8` wall time for
`clustered_keys_match_hash_grouping_across_batches_and_emits` drops from
3.0s to 0.8s.

Run both modes with:

```shell
cargo test --profile ci -p datafusion-physical-plan --lib clustered_keys_match_hash_grouping_across_batches_and_emits -- --test-threads=8
cargo test --profile ci -p datafusion-physical-plan --lib --features force_hash_collisions,datafusion-common/force_hash_collisions aggregates::group_values::multi_group_by::clustered::tests
```

## Are there any user-facing changes?

None. This changes test scheduling and skips redundant collision-mode
reference hashing.

AI-assisted contribution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants