Skip to content

test: split leaf_pushdown_fuzz_extended into parallel sub-tests (#26759) - #26902

Merged
blaginin merged 4 commits into
apache:mainfrom
Sruhvx-jpg:perf/leaf-pushdown-fuzz-split
Nov 18, 2026
Merged

blaginin merged 4 commits into
apache:mainfrom
Sruhvx-jpg:perf/leaf-pushdown-fuzz-split

Conversation

@Sruhvx-jpg

@Sruhvx-jpg Sruhvx-jpg commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes part of #26759.

Rationale for this change

In CI extended runs, leaf_pushdown_fuzz_extended bottlenecks runners for ~230 seconds by sequentially running 5,000 in-memory cases followed by 1,000 Parquet cases on a single thread.

Per maintainer feedback in #26843, this PR splits the extended test into smaller, parallel sub-tests rather than adding internal concurrency machinery.

What changes are included in this PR?

  • Splits the 5,000 in-memory cases of leaf_pushdown_fuzz_extended into 4 partitioned test targets (1,250 cases each with disjoint seed ranges).
  • Isolates the 1,000 Parquet cases into leaf_pushdown_fuzz_extended_parquet.
  • Relies on Cargo's native test runner to execute the sub-tests concurrently across available cores.
  • async fn run(...) remains unchanged. Total case count and deterministic seed reproduction are fully preserved.

Are these changes tested?

Yes, tested via cargo test --features extended_tests --test extended_tests -- leaf_pushdown.

@github-actions github-actions Bot added the core Core DataFusion crate label Oct 29, 2026
@Sruhvx-jpg

Sruhvx-jpg commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor Author

CC @2010YOUY01 what are ur opinion's?

@Sruhvx-jpg
Sruhvx-jpg marked this pull request as draft November 10, 2026 20:06
@blaginin
blaginin self-requested a review November 18, 2026 17:57
@blaginin
blaginin marked this pull request as ready for review November 18, 2026 17:57
@blaginin blaginin self-assigned this Nov 18, 2026
@blaginin

Copy link
Copy Markdown
Member

this seems important because that's one of the slowest test which sometimes result in the CI bottleneck

@blaginin
blaginin enabled auto-merge November 18, 2026 18:20
@blaginin
blaginin added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit 91da0ec Nov 18, 2026
42 checks passed
0xdeafbeef pushed a commit to 0xdeafbeef/datafusion that referenced this pull request Nov 19, 2026
extended tests were unnecessarily
[slow](https://github.com/apache/datafusion/actions/runs/44326404502/job/134878743321#step:11:812)
because of just two tests

<img width="1090" height="476"
alt="file-0b04021c474b008e850fda968749cfe6"
src="https://github.com/user-attachments/assets/5aa8cefd-ab40-42cf-9016-b25525cdba36"
/>


One was fixed in apache#26902

This PR fixes the other one



Before

<img width="340" height="69" alt="dyn-ca6f6b73039acc8437b1a680875da51e"
src="https://github.com/user-attachments/assets/1ece97ba-96b1-4667-bdcf-7b52e4d3d68d"
/>


After

<img width="341" height="64" alt="dyn-c899d24cd91ec98c3da6e7dd60769886"
src="https://github.com/user-attachments/assets/4a3f9d3d-8d8d-44ef-9a86-02dab0bd7902"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants