Skip to content

fix: UDF, UDAF, UDWF with_alias(..) should wrap the inner function fully - #12098

Merged
alamb merged 5 commits into
apache:mainfrom
Blizzara:avo/with-alias
Aug 22, 2024
Merged

alamb merged 5 commits into
apache:mainfrom
Blizzara:avo/with-alias

Conversation

@Blizzara

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

While calling with_alias on MakeArray, we noticed some weird type failures:

Error: Error(Inner { cause: Context("type_coercion", Plan("Coercion from [Int32, Int32, Int32] to the signature OneOf([UserDefined, Any(0)]) failed.

That turned out to be because the AliasedScalarUDFImpl doesn't implement coerce_types, so instead of using MakeArray::coerce_types we'd end up using the default implementation of ScalarUDFImpl which just throws (that throw is then ignored later on, making this a bit hard to debug).

What changes are included in this PR?

Implement the full set of functions in Aliased[Scalar/Aggregate/Window]FunctionUDFImpl to pass through to inner UDF.

Are these changes tested?

No.. any ideas on how to smartly test these? I can add a simple test for the case we hit somewhere, but is there any generic way of guaranteeing that the aliased versions re-implement all of the *UDFImpl methods?

Are there any user-facing changes?

Mostly just fixes

Comment thread datafusion/expr/src/udaf.rs Outdated
/// requirement aggregators return `Ok(None)`.
fn with_beneficial_ordering(
self: Arc<Self>,
&self,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Arc<Self> was hard to call from the aliased version. Is there a reason why it was Arc'd?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea is that passing through the &Arc allows the result to be created without a deep copy., though maybe it is being overly optimized 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, dunno if there is a difference between passing an Arc and &self perf wise - neither should require copy. But I got it work with the Arc<> (by adding a .clone() there, but that should only clone the pointer and not the self so I think that's fine as well), so then I don't need to change this and that's probably safer 😅

fixed in 33b89ce

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree the Arc in the API is wonky -- it definitely might be better to remove it. Thank you for considering doing so in a follow on PR

@github-actions github-actions Bot added logical-expr Logical plan and expressions functions Changes to functions implementation labels Aug 21, 2024
@Blizzara
Blizzara marked this pull request as ready for review August 21, 2024 14:11

@alamb alamb left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thanks @Blizzara -- the only question I have is related to the change of the Arc in the trait.

Perhaps we can contemplate that change as a separate PR so we can merge this one in?

Comment thread datafusion/expr/src/udaf.rs Outdated
/// requirement aggregators return `Ok(None)`.
fn with_beneficial_ordering(
self: Arc<Self>,
&self,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea is that passing through the &Arc allows the result to be created without a deep copy., though maybe it is being overly optimized 🤔

&self.aliases
}

fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<Field>> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 nice - to avoid similar bugs in the future maybe we should also add a comment to AggregateUDFImpl that tries to remind people to add an implementation here.

It would be cool if we could tell Rust not to create a default implementation for a particular impl so the compiler could do the check for us, but I don't know how to express that in Rust

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added notes in 175b8d2. And yep - it would be nice to have a way of enforcing this, but I also couldn't figure out any way to do that :/

@github-actions github-actions Bot removed the functions Changes to functions implementation label Aug 22, 2024
@alamb

alamb commented Aug 22, 2024

Copy link
Copy Markdown
Contributor

I took the liberty of pushing a commit to flx clippy

@alamb

alamb commented Aug 22, 2024

Copy link
Copy Markdown
Contributor

Compilation is failing due to #12108

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thanks @Blizzara

@alamb
alamb merged commit 502ce4b into apache:main Aug 22, 2024
@Blizzara
Blizzara deleted the avo/with-alias branch August 22, 2024 13:43
@Blizzara

Copy link
Copy Markdown
Contributor Author

Looks good to me -- thanks @Blizzara

Thanks for the review, fixing clippy, and fixing main! :)

vera-rubin pushed a commit to vera-rubin/datafusion that referenced this pull request Oct 18, 2026
…sed (apache#26480)

## Which issue does this PR close?

- Closes apache#26466.

## Rationale for this change

Giving an aggregate function an extra name with
`AggregateUDF::with_aliases` broke that function for every window frame
that does not start at `UNBOUNDED PRECEDING`. After registering an
aliased `sum`, this query

```sql
SELECT my_sum(i) OVER (ORDER BY i ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) FROM t
```

failed with

```
NotImplemented("Aggregate can not be used as a sliding accumulator because `retract_batch` is not implemented: sum(t.i) ORDER BY [t.i ASC NULLS LAST] ROWS BETWEEN 1 PRECEDING AND CURRENT ROW")
```

instead of returning `1, 3, 5, 7`. Aliased `min` and `max` failed the
same way.

The original name was affected too. `AliasedAggregateUDFImpl::name()`
forwards to the inner function, so registering an aliased `sum` replaces
the built-in under the name `sum` as well, and a plain `sum(i)` over a
sliding frame then failed in that session even though the user never
touched the alias.

## What changes are included in this PR?

One line in `AliasedAggregateUDFImpl::create_sliding_accumulator`. It
delegated to the inner function's `accumulator` rather than to the inner
function's `create_sliding_accumulator`, so an aliased function was
handed the plain accumulator, which does not implement `retract_batch`.

`sum`, `min`, `max` and `count` all override
`create_sliding_accumulator`, so forwarding to it is enough to restore
the correct behaviour. The `DISTINCT` paths come along for free, because
each function decides on `AccumulatorArgs::is_distinct` internally; the
wrapper only has to ask the right method.

Since apache#12098 set out to forward every method on this wrapper to the
inner function, this looks like a slip rather than a deliberate choice.
The `#[warn(clippy::missing_trait_methods)]` attribute on the impl
cannot catch it, because the method does exist and only its body is
wrong. It is also easy to miss by eye: the wrong body is exactly the
trait's own default implementation of `create_sliding_accumulator`.

## What is the testing strategy for this PR?

`datafusion/expr/src/udaf.rs` adds
`test_aliased_udaf_forwards_create_sliding_accumulator`, a unit test
built on a UDAF double whose two accumulator methods are deliberately
distinguishable: the plain one returns an error, the sliding one
succeeds. Asserting both directions pins the delegation itself,
independent of how any real aggregate implements its accumulators, so
the test keeps its meaning even if `sum`'s sliding accumulator is
rewritten.

`datafusion/core/tests/user_defined/user_defined_aggregates.rs` adds
three end-to-end tests, placed next to the existing
`test_user_defined_functions_with_alias`:

- aliased `sum`, `min` and `max` over `ROWS BETWEEN 1 PRECEDING AND
CURRENT ROW`, asserting the values from the issue;
- the same query under the original `sum` name, covering the
registry-overwrite case described above;
- an `UNBOUNDED PRECEDING` control, which takes the plain-accumulator
path and passes both before and after this change.

**Why these are not `.slt` tests**, even though the queries themselves
are plain SQL: reproducing the bug needs `with_aliases` plus
`register_udaf`, and `datafusion/sqllogictest` has no way to register a
user-defined aggregate, so an aliased UDAF cannot be constructed from
SQL at all. That puts this change at the second tier of the testing
guide, end-to-end tests through the public API, rather than the first.

Each new test was checked to fail without the one-line change, with the
error message quoted above, and the control test to pass either way.

## Are there any user-facing changes?

Yes, as a bug fix. Aliased `sum`, `min` and `max` now work in window
frames that do not start at `UNBOUNDED PRECEDING`, and a session that
registers an aliased aggregate no longer breaks the built-in function of
the same name in those frames. No public API signature changes, so no
`api change` label: `AliasedAggregateUDFImpl` is private, and
`create_sliding_accumulator` was already a defaulted method on
`AggregateUDFImpl`.

## AI usage

This PR was prepared with Claude Code.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants