Skip to content

simplify unqualified col name aliases - #3882

Closed
comphead wants to merge 5 commits into
apache:mainfrom
comphead:scalar_function_col_name
Closed

comphead wants to merge 5 commits into
apache:mainfrom
comphead:scalar_function_col_name

Conversation

@comphead

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3722 .

Rationale for this change

Use simple names in schema - use the function name in this example and do not add the parameters.

What changes are included in this PR?

Use simple names in schema - use the function name in this example and do not add the parameters.

Are there any user-facing changes?

The unqualified column name will be changed

@github-actions github-actions Bot added core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules labels Oct 18, 2022
@comphead

Copy link
Copy Markdown
Contributor Author

@andygrove please check this PR

alamb
alamb previously approved these changes Oct 19, 2022

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it looks great -- thank you @comphead

"| false |",
"| false |",
"+-------------------------------------------------------------+",
"+------------------------------+",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan changes sure look nicer 😍

@Dandandan

Copy link
Copy Markdown
Contributor

Not sure I'm a fan. What happens with the alias if you use a function on more than one column?
E.g. select trim(a), trim(b) from table?

@Dandandan Dandandan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when using the same function more than once?

@alamb

alamb commented Oct 19, 2022

Copy link
Copy Markdown
Contributor

Here is what postgres does:

postgres=# select * from foo;
 val | time 
-----+------
 A   |  1.0
 B   |  2.0
 C   |  3.0
(3 rows)

postgres=# select trim(val), trim(val) from foo;
 btrim | btrim 
-------+-------
 A     | A
 B     | B
 C     | C
(3 rows)

postgres=# 

Here is what datafusion-cli does from master:

DataFusion CLI v13.0.0
❯ create table foo as select column1 as val, column2 as time from (values ('A', 1.0), ('B', 2.0), ('C', 3.0)) as sq;

0 rows in set. Query took 0.023 seconds.
❯ select trim(val), trim(val) from foo;
Plan("Projections require unique expression names but the expression \"trim(foo.val)\" at position 0 and \"trim(foo.val)\" at position 1 have the same name. Consider aliasing (\"AS\") one of them.")

@comphead

comphead commented Oct 19, 2022 •

Copy link
Copy Markdown
Contributor Author

What happens when using the same function more than once?

@Dandandan thanks for you comment, the engine historically fails if the alias are the same, like if you run
let sql = "SELECT COALESCE(NULL, 'test'), COALESCE(NULL, 'test')";
it will fail even now.

supposingly we need a separate ticket to try to do postgres behaviuor, or add some unique postfix like
coalesce_0, coalesce_1. I'm happy to do this if needed

@comphead
comphead requested a review from Dandandan October 19, 2022 22:43
@Dandandan

Copy link
Copy Markdown
Contributor

Ok, but the current branch now also fails with
select trim(a), trim(b) from table? As the two are now giving the same alias. This is my main worry to merge this. As suffix that @comphead suggests seems a solution.

@comphead

comphead commented Oct 20, 2022 •

Copy link
Copy Markdown
Contributor Author

Ok, but the current branch now also fails with select trim(a), trim(b) from table? As the two are now giving the same alias. This is my main worry to merge this. As suffix that @comphead suggests seems a solution.

This is correct, as we shortened the column name the collision will happen more often. But this gives us more motivation to make unique readable aliases.

Moreover user should use custom aliases, not sure its super comfy to refer the column from outer sql to nested subquery which contains not friendly alias like btrim(test.a,Utf8(\"ab\"))

@alamb
alamb dismissed their stale review October 20, 2022 12:51

Regression now that we can not do trim(a) and trim(b) from table

@alamb

alamb commented Oct 20, 2022

Copy link
Copy Markdown
Contributor

BTW the way postgres handles this, as I recall, is to refer to each output created by a node using its index the output schema rather than a string

I don't recommend going this way as debugging errors with mixed up output indexes was both painful and common

@comphead

Copy link
Copy Markdown
Contributor Author

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

@alamb

alamb commented Oct 21, 2022

Copy link
Copy Markdown
Contributor

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

I am not sure -- perhaps @andygrove can weigh in here as he filed #3722

@comphead

Copy link
Copy Markdown
Contributor Author

@alamb @Dandandan should I put unique col aliasing like trim_0, trim_1 as part of this PR, or create a new one?

I am not sure -- perhaps @andygrove can weigh in here as he filed #3722

@andygrove please help here until the code rots

@andygrove

Copy link
Copy Markdown
Member

I will catch up on this and other pending PRs tomorrow.

# Conflicts:
#	datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs
@alamb

alamb commented Oct 27, 2022

Copy link
Copy Markdown
Contributor

Marking this as a draft as it is waiting on feedback and not ready for review or merge

@alamb
alamb marked this pull request as draft October 27, 2022 14:27
@andygrove

Copy link
Copy Markdown
Member

Postgres allows duplicate column names in a schema but then throws an error if you try and select one of those columns. We currently fail on creating a schema with duplicate columns.

I guess the first question is, do we want to be more like Postgres here?

@comphead

Copy link
Copy Markdown
Contributor Author

Postgres allows duplicate column names in a schema but then throws an error if you try and select one of those columns. We currently fail on creating a schema with duplicate columns.

I guess the first question is, do we want to be more like Postgres here?

Thanks @andygrove for the feedback.

Most likely we should avoid current naming for not qualified columns as its not user friendly and hard to use in outer subqueries.

And perhaps its a bigger topic to discuss naming standard, to be like a Postgres or something else, but we can create a discussion ticket on this first, to get all options from contributors.

If you ask me, I like how its done in Trino

select t.*, 'col_again' from ( select 'col', 2 a, count(1) ) t

_col0|a|_col2|_col3    |
-----+-+-----+---------+
col  |2|    1|col_again|

@alamb

alamb commented Nov 28, 2023

Copy link
Copy Markdown
Contributor

Closing as this PR is over a year old. Please feel free to reopen it / rebase it if you plan to keep working on it

@alamb alamb closed this Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use simpler schema names for expressions

4 participants