Skip to content

Optimizer regressions in unwrap_cast_in_comparison #3690

Closed

Description

@andygrove

Describe the bug
I am testing the latest code from master with some Dask SQL queries and seeing some new regressions that were caused by #3662.

To Reproduce

Error casting to unsigned types

#[test]
fn unsigned_target_type() -> Result<()> {
    let sql = "SELECT * FROM test WHERE col_uint32 > 0";
    let plan = test_sql(sql)?;
    let expected = "TBD";
    assert_eq!(expected, format!("{:?}", plan));
    Ok(())
}
Error: Internal("Optimizer rule 'unwrap_cast_in_comparison' failed due to unexpected error: Internal error: Error target data type UInt32. This was likely caused by a bug in DataFusion's code and we would welcome that you file an bug report in our issue tracker")

CASE WHEN regression

#[test]
fn case_when() -> Result<()> {
    let sql = "SELECT CASE WHEN col_int32 > 0 THEN 1 ELSE 0 END FROM test";
    let plan = test_sql(sql)?;
    let expected = "TBD";
    assert_eq!(expected, format!("{:?}", plan));
    Ok(())
}
Error: Internal("Optimizer rule 'unwrap_cast_in_comparison' failed due to unexpected error: Schema error: No field named 'test.col_int32'. Valid fields are 'CASE WHEN test.col_int32 > Int64(0) THEN Int64(1) ELSE Int64(0) END'.")

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

Activity

  1. self-assigned this
    on Oct 3, 2022
  2. liukun4515 commented on Oct 4, 2022

    @liukun4515
    Contributor

    @andygrove
    Does the failed about the case when caused by this issue?

    I will check them.

  3. liukun4515 commented on Oct 4, 2022

    @liukun4515
    Contributor

    For the case when case

    #[test]
    fn case_when() -> Result<()> {
        let sql = "SELECT CASE WHEN col_int32 > 0 THEN 1 ELSE 0 END FROM test";
        let plan = test_sql(sql)?;
        let expected = "Projection: CASE WHEN CAST(#test.col_int32 AS Int64) > Int64(0) THEN Int64(1) ELSE Int64(0) END\
        \n  TableScan: test projection=[col_int32]";
        assert_eq!(expected, format!("{:?}", plan));
        Ok(())
    }
    

    In the rule of unwrap_cast_in_comparison, we can't find the filed type of 'test.col_int32' for the projection plan.

    Because the schema for the projection is:

    DFSchema { fields: [DFField { qualifier: None, field: Field { name: "CASE WHEN test.col_int32 > Int64(0) THEN Int64(1) ELSE Int64(0) END", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None } }], metadata: {} }
    

    and the schema of table scan is:

    DFSchema { fields: [DFField { qualifier: Some("test"), field: Field { name: "col_int32", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("test"), field: Field { name: "col_uint32", data_type: UInt32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("test"), field: Field { name: "col_utf8", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("test"), field: Field { name: "col_date32", data_type: Date32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }, DFField { qualifier: Some("test"), field: Field { name: "col_date64", data_type: Date64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: None } }], metadata: {} }
    

    I am confused about the difference between these two plan.

    Why the schema for the projection plan is the output expr for the plan?

    @alamb @andygrove

    I think the schema should only contain the fields from the table, but I find the output expr for the projection plan.

  4. liukun4515 commented on Oct 4, 2022

    @liukun4515
    Contributor

    I think the schema for the logical plan is very important, and we can extract and generate any information from the basic schema.

    For example:

    table:
    c1 int32,
    c2 int64,
    c3 utf8
    

    sql:

    select c1,c2+1 from table where c1>12
    

    The plan should be

    projection:  c1,c2+1; the schema is [c1,c2]
        filter: c1>12; the schema is [c1,c2]
           table scan: [c1,c2] with the projection; the schema is [c1,c2]
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions