Skip to content

Support NULL literals in where clause - #11266

Merged
alamb merged 2 commits into
apache:mainfrom
xinlifoobar:dev/xinli/null_lit
Jul 9, 2024
Merged

alamb merged 2 commits into
apache:mainfrom
xinlifoobar:dev/xinli/null_lit

Conversation

@xinlifoobar

@xinlifoobar xinlifoobar commented Jul 4, 2024 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #11248

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) labels Jul 4, 2024
@xinlifoobar xinlifoobar changed the title Try fix where clause incorrectly reject NULL literal [POC] Try fix where clause incorrectly reject NULL literal Jul 4, 2024
@xinlifoobar xinlifoobar changed the title [POC] Try fix where clause incorrectly reject NULL literal Try fix where clause incorrectly reject NULL literal Jul 5, 2024
},
Err(_) => {
let Ok(null_array) = as_null_array(&array) else {
return internal_err!("Cannot create filter_array from non-boolean predicates, unable to continute");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previous it will throw a downcast_error, which mess up the debugging logs.

----
foo (empty) NULL

# Where clause accept NULL literal

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From postgres,

postgres=# select 1 where null and 1=1;
 ?column? 
----------
(0 rows)

postgres=# select 1 where null and 1=1;
 ?column? 
----------
(0 rows)

postgres=# select 1 where null or 1=1;
 ?column? 
----------
        1
(1 row)

@alamb alamb changed the title Try fix where clause incorrectly reject NULL literal Support NULL literals in where clause Jul 8, 2024

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @xinlifoobar -- this makes sense to me

Another potential way to fix this issue would be to cast / coerce NULL literals in the WHERE clause to boolean

Basically we could add the equivalent of NULL::boolean

For example, given the input

SELECT ... FROM foo WHERE NULL and x = 5;

We could transform it to

SELECT ... FROM foo WHERE NULL::boolean and x = 5;

That might result in simpler code as FilterExec would not have to be aware of DataType::null

Perhaps it would fit in https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/analyzer/type_coercion.rs

However, I think this PR is better than main and we could try to see if the alternate approach loked better as a future work (or never)

@alamb
alamb merged commit fa01917 into apache:main Jul 9, 2024
@alamb

alamb commented Jul 9, 2024

Copy link
Copy Markdown
Contributor

Thanks again @xinlifoobar

@xinlifoobar

Copy link
Copy Markdown
Contributor Author

Thanks @xinlifoobar -- this makes sense to me

Another potential way to fix this issue would be to cast / coerce NULL literals in the WHERE clause to boolean

Basically we could add the equivalent of NULL::boolean

For example, given the input

SELECT ... FROM foo WHERE NULL and x = 5;

We could transform it to

SELECT ... FROM foo WHERE NULL::boolean and x = 5;

That might result in simpler code as FilterExec would not have to be aware of DataType::null

Perhaps it would fit in https://github.com/apache/datafusion/blob/main/datafusion/optimizer/src/analyzer/type_coercion.rs

However, I think this PR is better than main and we could try to see if the alternate approach loked better as a future work (or never)

This is a lot... Let me looking into this.

findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
* Try fix where clause incorrectly reject NULL literal

* check null in filter
xinlifoobar added a commit to xinlifoobar/datafusion that referenced this pull request Jul 16, 2024
xinlifoobar added a commit to xinlifoobar/datafusion that referenced this pull request Jul 18, 2024
* Try fix where clause incorrectly reject NULL literal

* check null in filter
alamb pushed a commit that referenced this pull request Jul 18, 2024
…11491)

* Revert "Support `NULL` literals in where clause  (#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
Lordworms pushed a commit to Lordworms/arrow-datafusion that referenced this pull request Jul 23, 2024
…pache#11491)

* Revert "Support `NULL` literals in where clause  (apache#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
wiedld pushed a commit to influxdata/arrow-datafusion that referenced this pull request Jul 31, 2024
…pache#11491)

* Revert "Support `NULL` literals in where clause  (apache#11266)"

This reverts commit fa01917.

* Followup Support NULL literals in where clause

* misc err change

* adopt comparison_coercion

* Fix comments

* Fix comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

where clause incorrectly reject NULL literal (by SQLancer-NoREC)

2 participants