Repository navigation
feat(spark): add grouped accumulators for collect_list and collect_set - #348
AnuragRaut08 wants to merge 1 commit into
Conversation
collect_list and collect_set
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this, @AnuragRaut08. The wrapper is small and the unit tests are thorough. I like that make_acc deliberately sets ignore_nulls and is_distinct to the wrong values, and that EmitTo::First, filters and the result type are all covered.
I checked out the PR and compared it with main on a set of queries run with target_partitions=4 and batch_size=4. These covered collect_set(a ORDER BY ...), collect_list(DISTINCT a), nested List elements, struct elements and dictionary-encoded strings. Results were identical to main except for the element order of collect_set, which I mention below. I also ran a 2000-row table with a small batch size, 4 partitions and the skip-partial thresholds forced. The plan was Partial then FinalPartitioned, and the per-group counts matched count(a). So the merge and convert_to_state paths work in practice. cargo clippy and cargo fmt --check are clean locally.
I also read the pinned DataFusion branch-56 code. ArrayAggGroupsAccumulator::evaluate and DistinctArrayAggGroupsAccumulator::evaluate both emit NULL only for groups with a count of zero, and those entries have an empty offset range. So dropping the null buffer is correct for the version we pin.
1. Make the invariant check real in release builds
That invariant belongs to DataFusion and not to this crate. If a future DataFusion bump emitted a NULL entry with a non-empty range, release builds would silently return wrong lists, because the check in NullToEmptyListGroupsAccumulator::evaluate is only a debug_assert!. The unit test is also #[cfg(debug_assertions)], so it never runs in release. The PR description says the invariant is checked, which is only true for debug builds.
The loop is O(groups), which is cheap next to the O(rows) work in evaluate. Would you consider turning it into a real check that returns internal_err!? The test could then run everywhere.
2. Extra slt coverage
The description lists multi-phase aggregation, skip-partial and nested types as untested. These can be covered from SQL. Something like this worked for me:
statement ok
set datafusion.execution.target_partitions = 4;
statement ok
set datafusion.execution.batch_size = 16;
statement ok
set datafusion.execution.skip_partial_aggregation_probe_rows_threshold = 1;
statement ok
set datafusion.execution.skip_partial_aggregation_probe_ratio_threshold = 0.0;
statement ok
CREATE TABLE big AS SELECT v % 7 AS g, CASE WHEN v % 5 = 0 THEN NULL ELSE v % 11 END AS a FROM generate_series(1, 2000) AS t(v);
query IIII
SELECT g, array_length(collect_list(a)), array_length(collect_set(a)), count(a) FROM big GROUP BY g ORDER BY g;
It would also be good to add a case with make_array(1, NULL) elements and one with named_struct elements, including an all-NULL group. That locks in the behavior the description says is untested. Please remember to reset the session settings at the end of the file if other tests in it depend on the defaults.
3. Description and benchmark notes
- The element order of
collect_setchanges. OnmainI see hash order ([2, 1, 3]for my test input) and with this PR I see insertion order ([1, 3, 2]). Spark leaves it unspecified and the existing slt files already wrap it inarray_sort, so this looks fine. It is user-visible, though, so a line in the description would help. - The benchmark uses a single-partition
MemTableon a current-thread runtime. That is fine for the claim, but could you say it doesn't exercise repartitioning or merging of partial states?
CI had not run yet when I looked, so a maintainer may need to approve the workflow.
|
Hi @AnuragRaut08, just checking in on the points above when you have a moment. The branch also conflicts with |
Which issue does this PR close?
Closes #339
Rationale for this change
Spark-compatible
collect_listandcollect_setcurrently use the scalar accumulator implementation without native grouped accumulator support.As a result, grouped aggregation falls back to DataFusion's generic
GroupsAccumulatorAdapter, which can introduce significant overhead, particularly when aggregating high-cardinality data with many groups.This change provides native grouped accumulator implementations while preserving Spark's expected behavior:
collect_listretains duplicate values.collect_setremoves duplicate values.FILTERclause, return an empty list ([]) rather thanNULL.What changes are included in this PR?
groups_accumulator_supported()andcreate_groups_accumulator()for bothSparkCollectListandSparkCollectSet.array_aggimplementation, configuring NULL handling and distinct semantics to match Spark.state(),merge_batch(), andconvert_to_state()to support partial aggregation and merging.collect_listand filtered groupedcollect_list/collect_set.Are these changes tested?
Yes.
cargo test -p datafusion-spark— passed (431 tests).cargo test -p datafusion-spark-sqllogictest— passed (252 test files).cargo clippy --all-targets --all-features -- -D warnings— passed.cargo fmt --all -- --check— passed.cargo bench -p datafusion-spark --bench collect --no-run— passed.The added SQLLogicTests verify grouped NULL handling, duplicate preservation, and
FILTERbehavior. Rust unit tests additionally exercise the grouped accumulator's state and emission paths, which are difficult to cover directly through SQLLogicTests.Expected behavior is based on Spark's documented
collect_listandcollect_setsemantics. No separate PySpark version comparison was performed.Are there any user-facing changes?
No SQL syntax or function signatures change.
This improves the execution path for grouped
collect_listandcollect_setand ensures groups with no retained values evaluate to empty lists rather than NULL, consistent with Spark behavior.Benchmark
Benchmarked through SQL
GROUP BYwith 2 million rows, 10% NULL values, a single-threaded Tokio runtime, and 10 Criterion samples. Before and after measurements were taken on the same machine.Results:
collect_list(1,000 groups): 425 ms → 44 ms (~9.7x faster)collect_set(1,000 groups): 339 ms → 101 ms (~3.4x faster)collect_list(200,000 groups): 4.82 s → 110 ms (~43.8x faster)collect_set(200,000 groups): 7.03 s → 270 ms (~26x faster)These results compare the previous and new implementations only; there is no Spark baseline.
The grouped
array_aggimplementation emits NULL for groups with no retained values, with empty offset ranges for those entries. Removing the null buffer therefore produces valid empty lists without orphaned values. Adebug_assert!checks this invariant, and a unit test covers the failure case.Known Limitations
state(),merge_batch(), andconvert_to_state()directly.AI Assistance
I used an AI assistant during development. I reviewed the implementation and ran the tests and benchmarks myself manually