You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
From README, I found Set Operations UNION isn't implemented. So I want to implement it.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Union distinct effectively means building a structure to deduplicate the same rows. This means:
Sort and deduplicate based on equivalent consecutive rows (might be fast if data is already sorted)
Hash all column keys, store in hash table.
My proposal to implement Union is to translate [a] union [b] into the equivalent select distinct * from ([a] union all [b]). This currently uses the hash aggregate based implementation for deduplication. Later it could switch to whatever implementation is more efficient for the particular query.
Activity
xudong963 commentedon Sep 20, 2021
I am blocked at dedup
Vec<RecordBatch>. Do you have some good ideas? @houqp @alambDandandan commentedon Sep 20, 2021
Union distincteffectively means building a structure to deduplicate the same rows. This means:My proposal to implement
Unionis to translate[a] union [b]into the equivalentselect distinct * from ([a] union all [b]). This currently uses the hash aggregate based implementation for deduplication. Later it could switch to whatever implementation is more efficient for the particular query.xudong963 commentedon Sep 20, 2021
@Dandandan Thanks for your idea. I have implemented a version. I will consider your idea later.
xudong963 commentedon Sep 20, 2021
@houqp @alamb The obstacle solved, but may not be the best.
alamb commentedon Sep 20, 2021
Another potential way to implement
UNIONmight be to rewrite the logical plan to the equivalent of:SELECT x from foo UNION select x from bar-->
SELECT distinct (select x from foo UNION ALL select x from bar)So as to avoid having to reimplement any physical operators