Repository navigation
fix: UDF, UDAF, UDWF with_alias(..) should wrap the inner function fully - #12098
Conversation
| /// requirement aggregators return `Ok(None)`. | ||
| fn with_beneficial_ordering( | ||
| self: Arc<Self>, | ||
| &self, |
There was a problem hiding this comment.
The Arc<Self> was hard to call from the aliased version. Is there a reason why it was Arc'd?
There was a problem hiding this comment.
I think the idea is that passing through the &Arc allows the result to be created without a deep copy., though maybe it is being overly optimized 🤔
There was a problem hiding this comment.
Hmm, dunno if there is a difference between passing an Arc and &self perf wise - neither should require copy. But I got it work with the Arc<> (by adding a .clone() there, but that should only clone the pointer and not the self so I think that's fine as well), so then I don't need to change this and that's probably safer 😅
fixed in 33b89ce
There was a problem hiding this comment.
I agree the Arc in the API is wonky -- it definitely might be better to remove it. Thank you for considering doing so in a follow on PR
| /// requirement aggregators return `Ok(None)`. | ||
| fn with_beneficial_ordering( | ||
| self: Arc<Self>, | ||
| &self, |
There was a problem hiding this comment.
I think the idea is that passing through the &Arc allows the result to be created without a deep copy., though maybe it is being overly optimized 🤔
| &self.aliases | ||
| } | ||
|
|
||
| fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<Field>> { |
There was a problem hiding this comment.
👍 nice - to avoid similar bugs in the future maybe we should also add a comment to AggregateUDFImpl that tries to remind people to add an implementation here.
It would be cool if we could tell Rust not to create a default implementation for a particular impl so the compiler could do the check for us, but I don't know how to express that in Rust
There was a problem hiding this comment.
Added notes in 175b8d2. And yep - it would be nice to have a way of enforcing this, but I also couldn't figure out any way to do that :/
|
I took the liberty of pushing a commit to flx clippy |
|
Compilation is failing due to #12108 |
Thanks for the review, fixing clippy, and fixing main! :) |
…sed (apache#26480) ## Which issue does this PR close? - Closes apache#26466. ## Rationale for this change Giving an aggregate function an extra name with `AggregateUDF::with_aliases` broke that function for every window frame that does not start at `UNBOUNDED PRECEDING`. After registering an aliased `sum`, this query ```sql SELECT my_sum(i) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) FROM t ``` failed with ``` NotImplemented("Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: sum(t.i) ORDER BY [t.i ASC NULLS LAST] ROWS BETWEEN 1 PRECEDING AND CURRENT ROW") ``` instead of returning `1, 3, 5, 7`. Aliased `min` and `max` failed the same way. The original name was affected too. `AliasedAggregateUDFImpl::name()` forwards to the inner function, so registering an aliased `sum` replaces the built-in under the name `sum` as well, and a plain `sum(i)` over a sliding frame then failed in that session even though the user never touched the alias. ## What changes are included in this PR? One line in `AliasedAggregateUDFImpl::create_sliding_accumulator`. It delegated to the inner function's `accumulator` rather than to the inner function's `create_sliding_accumulator`, so an aliased function was handed the plain accumulator, which does not implement `retract_batch`. `sum`, `min`, `max` and `count` all override `create_sliding_accumulator`, so forwarding to it is enough to restore the correct behaviour. The `DISTINCT` paths come along for free, because each function decides on `AccumulatorArgs::is_distinct` internally; the wrapper only has to ask the right method. Since apache#12098 set out to forward every method on this wrapper to the inner function, this looks like a slip rather than a deliberate choice. The `#[warn(clippy::missing_trait_methods)]` attribute on the impl cannot catch it, because the method does exist and only its body is wrong. It is also easy to miss by eye: the wrong body is exactly the trait's own default implementation of `create_sliding_accumulator`. ## What is the testing strategy for this PR? `datafusion/expr/src/udaf.rs` adds `test_aliased_udaf_forwards_create_sliding_accumulator`, a unit test built on a UDAF double whose two accumulator methods are deliberately distinguishable: the plain one returns an error, the sliding one succeeds. Asserting both directions pins the delegation itself, independent of how any real aggregate implements its accumulators, so the test keeps its meaning even if `sum`'s sliding accumulator is rewritten. `datafusion/core/tests/user_defined/user_defined_aggregates.rs` adds three end-to-end tests, placed next to the existing `test_user_defined_functions_with_alias`: - aliased `sum`, `min` and `max` over `ROWS BETWEEN 1 PRECEDING AND CURRENT ROW`, asserting the values from the issue; - the same query under the original `sum` name, covering the registry-overwrite case described above; - an `UNBOUNDED PRECEDING` control, which takes the plain-accumulator path and passes both before and after this change. **Why these are not `.slt` tests**, even though the queries themselves are plain SQL: reproducing the bug needs `with_aliases` plus `register_udaf`, and `datafusion/sqllogictest` has no way to register a user-defined aggregate, so an aliased UDAF cannot be constructed from SQL at all. That puts this change at the second tier of the testing guide, end-to-end tests through the public API, rather than the first. Each new test was checked to fail without the one-line change, with the error message quoted above, and the control test to pass either way. ## Are there any user-facing changes? Yes, as a bug fix. Aliased `sum`, `min` and `max` now work in window frames that do not start at `UNBOUNDED PRECEDING`, and a session that registers an aliased aggregate no longer breaks the built-in function of the same name in those frames. No public API signature changes, so no `api change` label: `AliasedAggregateUDFImpl` is private, and `create_sliding_accumulator` was already a defaulted method on `AggregateUDFImpl`. ## AI usage This PR was prepared with Claude Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Which issue does this PR close?
Rationale for this change
While calling
with_aliasonMakeArray, we noticed some weird type failures:That turned out to be because the AliasedScalarUDFImpl doesn't implement
coerce_types, so instead of usingMakeArray::coerce_typeswe'd end up using the default implementation of ScalarUDFImpl which just throws (that throw is then ignored later on, making this a bit hard to debug).What changes are included in this PR?
Implement the full set of functions in Aliased[Scalar/Aggregate/Window]FunctionUDFImpl to pass through to inner UDF.
Are these changes tested?
No.. any ideas on how to smartly test these? I can add a simple test for the case we hit somewhere, but is there any generic way of guaranteeing that the aliased versions re-implement all of the *UDFImpl methods?
Are there any user-facing changes?
Mostly just fixes