Skip to content

implement Set Operations UNION #998

Closed

Description

@xudong963

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
From README, I found Set Operations UNION isn't implemented. So I want to implement it.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Activity

  1. xudong963 commented on Sep 20, 2021

    @xudong963
    MemberAuthor

    I am blocked at dedup Vec<RecordBatch>. Do you have some good ideas? @houqp @alamb

  2. Dandandan commented on Sep 20, 2021

    @Dandandan
    Contributor

    Union distinct effectively means building a structure to deduplicate the same rows. This means:

    • Sort and deduplicate based on equivalent consecutive rows (might be fast if data is already sorted)
    • Hash all column keys, store in hash table.

    My proposal to implement Union is to translate [a] union [b] into the equivalent select distinct * from ([a] union all [b]). This currently uses the hash aggregate based implementation for deduplication. Later it could switch to whatever implementation is more efficient for the particular query.

  3. xudong963 commented on Sep 20, 2021

    @xudong963
    MemberAuthor

    @Dandandan Thanks for your idea. I have implemented a version. I will consider your idea later.

  4. xudong963 commented on Sep 20, 2021

    @xudong963
    MemberAuthor

    I am blocked at dedup Vec<RecordBatch>. Do you have some good ideas? @houqp @alamb

    @houqp @alamb The obstacle solved, but may not be the best.

  5. alamb commented on Sep 20, 2021

    @alamb
    Contributor

    Another potential way to implement UNION might be to rewrite the logical plan to the equivalent of:

    SELECT x from foo UNION select x from bar
    -->
    SELECT distinct (select x from foo UNION ALL select x from bar)

    So as to avoid having to reimplement any physical operators

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions