You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Improved performance for FixedSizeBinary grouping #24267
@maxburke added native support for FixedSizeBinary grouping keys (for example uuids) and the initial support is faster than the previous implementation. While testing the PR, I found it can still be optimzied more, so I wanted to record the ideas in this ticket.
Here is a way to test:
COPY (
SELECT
arrow_cast(decode(md5((value % 5000000)::varchar), 'hex'), 'FixedSizeBinary(16)') AS uuid,
(value % 10) AS id
FROM generate_series(1, 200000000)
) TO 'uuids.parquet';
This looks like this
andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ du -s -h uuids.parquet
3.0G uuids.parquet
As a follow on, this can likely be optimized more -- for example, we could have a special case loop for when the inputs are known to have no nulls (likely a common case for things like UUIDs)
We could also move to using get_unchecked to skip the bounds check and try to make this lookup loop even more performant
Activity
dhruvxvaishnav commentedon Aug 12, 2026
take