Skip to content

Implement select distinct - #262

Merged
Dandandan merged 10 commits into
apache:masterfrom
Dandandan:implement_select_distinct
May 5, 2021
Merged

Dandandan merged 10 commits into
apache:masterfrom
Dandandan:implement_select_distinct

Conversation

@Dandandan

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #250

Rationale for this change

Fix behavior and a simple implementation of select distinct.

What changes are included in this PR?

An implementation of select distinct.

Are there any user-facing changes?

A working select distinct

&self,
input: &LogicalPlan,
expr: Vec<Expr>,
force: bool,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small drive-by simplification, an unused option

@Dandandan
Dandandan requested a review from alamb May 4, 2021 20:54
@codecov-commenter

codecov-commenter commented May 4, 2021 •

Copy link
Copy Markdown

Codecov Report

Merging #262 (6ea9c77) into master (5f6024d) will increase coverage by 0.06%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #262      +/-   ##
==========================================
+ Coverage   76.80%   76.87%   +0.06%     
==========================================
  Files         133      133              
  Lines       23284    23350      +66     
==========================================
+ Hits        17884    17950      +66     
  Misses       5400     5400              
Impacted Files Coverage Δ
datafusion/src/sql/planner.rs 83.70% <100.00%> (+0.05%) ⬆️
datafusion/tests/sql.rs 99.89% <100.00%> (+<0.01%) ⬆️
datafusion/src/datasource/csv.rs 73.07% <0.00%> (-0.26%) ⬇️
...tafusion/src/physical_plan/distinct_expressions.rs 90.80% <0.00%> (-0.11%) ⬇️
datafusion/src/physical_plan/csv.rs 79.01% <0.00%> (-0.09%) ⬇️
datafusion/src/physical_plan/parquet.rs 87.55% <0.00%> (-0.02%) ⬇️
datafusion/src/scalar.rs 54.36% <0.00%> (+0.22%) ⬆️
datafusion/src/physical_plan/group_scalar.rs 58.82% <0.00%> (+1.17%) ⬆️
datafusion/src/physical_plan/common.rs 86.20% <0.00%> (+2.20%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5f6024d...6ea9c77. Read the comment docs.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great @Dandandan thank you ❤️

Comment thread datafusion/tests/sql.rs Resolved
Comment thread datafusion/tests/sql.rs
let sql = "SELECT DISTINCT c1 FROM aggregate_simple order by c1";
let actual = execute(&mut ctx, sql).await;

let expected = vec![

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I doubled checked that the input has duplicates (to show that the distinct has removed them): https://github.com/apache/arrow-datafusion/blob/master/datafusion/tests/aggregate_simple.csv

👍

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@Dandandan
Dandandan merged commit 9e84f15 into apache:master May 5, 2021
@houqp houqp added api change Changes the API exposed to users of the crate datafusion enhancement New feature or request labels Jul 29, 2021
HairstonE pushed a commit to HairstonE/datafusion that referenced this pull request Oct 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api change Changes the API exposed to users of the crate enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorrect answers with SELECT DISTINCT queries

5 participants