Skip to content

Reconsider hashing of nulls #822

Closed
Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The create_hash function is responsible for hashing values in arrays. At the moment, however, it (effectively) hashes NULL values to 0 for all types, which likely leads to sub optimial behavior such as @Dandandan observed in #812 (comment) that NULL,1 and 1,NULL will hash to the same value.

Describe the solution you'd like
TBD

Describe alternatives you've considered
@jorgecarleitao 's comment (copied below) from #790 (comment) offers a few alternatives:

From the hashing side, an unknown to me atm is how to efficiently hash values+validity. I.e. given V = ["a", "", "c"] and N = [true, false, true], I see some options:

  • hash(V) ^ !N + unique * N where unique is a unique sentinel value exclusive for null values. If hash is vectorized, this operation is vectorized.

  • concat(hash(value), is_valid) for value, is_valid in zip(V,N)

  • split the array between nulls and not nulls, i.e. N -> (non-null indices, null indices), perform hashing over valid indices only, and then, at the very end, append all values for the nulls. We do this in the sort kernel, to reduce the number of slots to perform comparisons over.

If we could write the code in a way that we could "easily" switch between implementations (during dev only, not a conf parameter), we could bench whether one wins over the other, or under which circumstances.

Additional context
Add any other context or screenshots about the feature request here.

Activity

  1. alamb commented on Mar 10, 2026

    @alamb
    ContributorAuthor

    This is not clearly actionable so let's close this one as done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions