Skip to content

feat(spark): add grouped accumulators for collect_list and collect_set - #348

Open
AnuragRaut08 wants to merge 1 commit into
apache:mainfrom
AnuragRaut08:feat/339-spark-collect-groups-accumulator
Open

AnuragRaut08 wants to merge 1 commit into
apache:mainfrom
AnuragRaut08:feat/339-spark-collect-groups-accumulator

Conversation

@AnuragRaut08

Copy link
Copy Markdown

Which issue does this PR close?

Closes #339

Rationale for this change

Spark-compatible collect_list and collect_set currently use the scalar accumulator implementation without native grouped accumulator support.

As a result, grouped aggregation falls back to DataFusion's generic GroupsAccumulatorAdapter, which can introduce significant overhead, particularly when aggregating high-cardinality data with many groups.

This change provides native grouped accumulator implementations while preserving Spark's expected behavior:

  • collect_list retains duplicate values.
  • collect_set removes duplicate values.
  • NULL input values are ignored.
  • Groups with no retained values, including groups containing only NULL values or rows excluded by a FILTER clause, return an empty list ([]) rather than NULL.

What changes are included in this PR?

  • Implement groups_accumulator_supported() and create_groups_accumulator() for both SparkCollectList and SparkCollectSet.
  • Reuse DataFusion's native grouped array_agg implementation, configuring NULL handling and distinct semantics to match Spark.
  • Add a grouped accumulator wrapper that converts NULL evaluation results for empty groups into valid empty lists without modifying the intermediate aggregation state.
  • Preserve state handling through state(), merge_batch(), and convert_to_state() to support partial aggregation and merging.
  • Add Rust unit tests covering duplicate handling, NULL-only and unseen groups, filtered aggregation, partial-state merging, state conversion, incremental batches, group growth, partial emission, string values, and non-nullable inputs.
  • Add SQLLogicTests covering grouped collect_list and filtered grouped collect_list/collect_set.
  • Add a Criterion benchmark for both functions using 2 million rows and workloads with 1,000 and 200,000 groups.

Are these changes tested?

Yes.

  • cargo test -p datafusion-spark — passed (431 tests).
  • cargo test -p datafusion-spark-sqllogictest — passed (252 test files).
  • cargo clippy --all-targets --all-features -- -D warnings — passed.
  • cargo fmt --all -- --check — passed.
  • cargo bench -p datafusion-spark --bench collect --no-run — passed.

The added SQLLogicTests verify grouped NULL handling, duplicate preservation, and FILTER behavior. Rust unit tests additionally exercise the grouped accumulator's state and emission paths, which are difficult to cover directly through SQLLogicTests.

Expected behavior is based on Spark's documented collect_list and collect_set semantics. No separate PySpark version comparison was performed.

Are there any user-facing changes?

No SQL syntax or function signatures change.

This improves the execution path for grouped collect_list and collect_set and ensures groups with no retained values evaluate to empty lists rather than NULL, consistent with Spark behavior.

Benchmark

Benchmarked through SQL GROUP BY with 2 million rows, 10% NULL values, a single-threaded Tokio runtime, and 10 Criterion samples. Before and after measurements were taken on the same machine.

Results:

  • collect_list (1,000 groups): 425 ms → 44 ms (~9.7x faster)
  • collect_set (1,000 groups): 339 ms → 101 ms (~3.4x faster)
  • collect_list (200,000 groups): 4.82 s → 110 ms (~43.8x faster)
  • collect_set (200,000 groups): 7.03 s → 270 ms (~26x faster)

These results compare the previous and new implementations only; there is no Spark baseline.

The grouped array_agg implementation emits NULL for groups with no retained values, with empty offset ranges for those entries. Removing the null buffer therefore produces valid empty lists without orphaned values. A debug_assert! checks this invariant, and a unit test covers the failure case.

Known Limitations

  • Nested-type element behavior is not explicitly covered by the added tests.
  • No SQL test explicitly forces a multi-phase aggregation plan or the skip-partial path. Rust unit tests exercise state(), merge_batch(), and convert_to_state() directly.

AI Assistance

I used an AI assistant during development. I reviewed the implementation and ran the tests and benchmarks myself manually

@AnuragRaut08 AnuragRaut08 changed the title feat(spark): add grouped accumulators for collect_list and collect_set feat(spark): add grouped accumulators for collect_list and collect_set Nov 17, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @AnuragRaut08. The wrapper is small and the unit tests are thorough. I like that make_acc deliberately sets ignore_nulls and is_distinct to the wrong values, and that EmitTo::First, filters and the result type are all covered.

I checked out the PR and compared it with main on a set of queries run with target_partitions=4 and batch_size=4. These covered collect_set(a ORDER BY ...), collect_list(DISTINCT a), nested List elements, struct elements and dictionary-encoded strings. Results were identical to main except for the element order of collect_set, which I mention below. I also ran a 2000-row table with a small batch size, 4 partitions and the skip-partial thresholds forced. The plan was Partial then FinalPartitioned, and the per-group counts matched count(a). So the merge and convert_to_state paths work in practice. cargo clippy and cargo fmt --check are clean locally.

I also read the pinned DataFusion branch-56 code. ArrayAggGroupsAccumulator::evaluate and DistinctArrayAggGroupsAccumulator::evaluate both emit NULL only for groups with a count of zero, and those entries have an empty offset range. So dropping the null buffer is correct for the version we pin.

1. Make the invariant check real in release builds

That invariant belongs to DataFusion and not to this crate. If a future DataFusion bump emitted a NULL entry with a non-empty range, release builds would silently return wrong lists, because the check in NullToEmptyListGroupsAccumulator::evaluate is only a debug_assert!. The unit test is also #[cfg(debug_assertions)], so it never runs in release. The PR description says the invariant is checked, which is only true for debug builds.

The loop is O(groups), which is cheap next to the O(rows) work in evaluate. Would you consider turning it into a real check that returns internal_err!? The test could then run everywhere.

2. Extra slt coverage

The description lists multi-phase aggregation, skip-partial and nested types as untested. These can be covered from SQL. Something like this worked for me:

statement ok
set datafusion.execution.target_partitions = 4;

statement ok
set datafusion.execution.batch_size = 16;

statement ok
set datafusion.execution.skip_partial_aggregation_probe_rows_threshold = 1;

statement ok
set datafusion.execution.skip_partial_aggregation_probe_ratio_threshold = 0.0;

statement ok
CREATE TABLE big AS SELECT v % 7 AS g, CASE WHEN v % 5 = 0 THEN NULL ELSE v % 11 END AS a FROM generate_series(1, 2000) AS t(v);

query IIII
SELECT g, array_length(collect_list(a)), array_length(collect_set(a)), count(a) FROM big GROUP BY g ORDER BY g;

It would also be good to add a case with make_array(1, NULL) elements and one with named_struct elements, including an all-NULL group. That locks in the behavior the description says is untested. Please remember to reset the session settings at the end of the file if other tests in it depend on the defaults.

3. Description and benchmark notes

  • The element order of collect_set changes. On main I see hash order ([2, 1, 3] for my test input) and with this PR I see insertion order ([1, 3, 2]). Spark leaves it unspecified and the existing slt files already wrap it in array_sort, so this looks fine. It is user-visible, though, so a line in the description would help.
  • The benchmark uses a single-partition MemTable on a current-thread runtime. That is fine for the claim, but could you say it doesn't exercise repartitioning or merging of partial states?

CI had not run yet when I looked, so a maintainer may need to approve the workflow.

@andygrove

Copy link
Copy Markdown
Member

Hi @AnuragRaut08, just checking in on the points above when you have a moment. The branch also conflicts with main now. It's only the new [[bench]] entry in crates/spark/Cargo.toml, so a rebase should sort it out. I ran the collect.slt file through PySpark 4.2.0 and the new grouped cases match Spark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SparkCollectList / SparkCollectSet have no GroupsAccumulator

2 participants