Skip to content

hash_join panics when join keys have different data types #2877

Closed

Description

@andygrove

Describe the bug

There are several places where hash_join can panic in version 9.0.0 when I run SQL queries.

thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-9.0.0/src/physical_plan/hash_join.rs:973:17
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-9.0.0/src/physical_plan/hash_join.rs:979:17
thread 'tokio-runtime-worker' panicked at 'called `Option::unwrap()` on a `None` value', /home/andy/.cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-9.0.0/src/physical_plan/hash_join.rs:1045:17

To Reproduce
I am experimenting with SQL query fuzzing. Here is one query that failed:

SELECT _c438, _c439, _c440, _c441, _c442, _c443
FROM (
  (
    (SELECT test1.c0 AS _c438, test1.c1 AS _c439, test1.c2 AS _c440, test1.c3 AS _c441, test1.c4 AS _c442, test1.c5 AS _c443
      FROM (test1))
    FULL JOIN
    (SELECT test1.c0 AS _c444, test1.c1 AS _c445, test1.c2 AS _c446, test1.c3 AS _c447, test1.c4 AS _c448, test1.c5 AS _c449
      FROM (test1))
    ON _c438 = _c446)
  RIGHT JOIN
  (SELECT test0.c0 AS _c450, test0.c1 AS _c451, test0.c2 AS _c452, test0.c3 AS _c453, test0.c4 AS _c454, test0.c5 AS _c455
    FROM (test0))
  ON _c441 = _c451);
ArrowError(ExternalError("Arrow error: External error: Execution error: Join Error: task 5672 panicked"))

Data files are here - https://github.com/andygrove/sqlfuzz/tree/main/testdata

Expected behavior
Should not panic

Additional context

Activity

  1. andygrove commented on Jul 12, 2022

    @andygrove
    MemberAuthor

    I think the issue is that we don't have type coercion for join keys so joining a number column to a string can cause this. in fn equal_rows we are making assumptions that left and right columns have the same data type.

  2. changed the title [-]hash_join panics ('called `Option::unwrap()` on a `None` value)[/-] [+]hash_join panics when join keys have different data types[/+] on Jul 12, 2022
  3. comphead commented on Jul 18, 2022

    @comphead
    Contributor

    @andygrove I'd like to investigate this as part of #2910

  4. comphead commented on Aug 2, 2022

    @comphead
    Contributor

    Rev engineered data from sqlfuzz, quick test to reproduce the behavior

    #[tokio::test]
        async fn hj_non_compat() -> Result<()> {
            let schema = Arc::new(Schema::new(vec![
                arrow::datatypes::Field::new(
                "column_1",
                DataType::Int32,
                false,
            ),
            arrow::datatypes::Field::new(
                "column_2",
                DataType::Int8,
                false,
            )
            
            ]));
    
            let data = RecordBatch::try_new(
                schema.clone(),
                vec![
                    Arc::new(arrow::array::Int32Array::from(vec![
                        Some(1),
                    ])),
                    Arc::new(arrow::array::Int8Array::from(vec![
                        Some(1),
                    ])),
                ],
            )?;
    
            let table = crate::datasource::MemTable::try_new(schema, vec![vec![data]])?;
    
            let ctx = SessionContext::new();
            ctx.register_table("test", Arc::new(table))?;
    
            let sql = r#"
            SELECT 
                *
            FROM 
                test a right join test b on a.column_1 = b.column_2"#;
    
            let df = ctx.sql(sql).await.unwrap();
            df.show_limit(10).await.unwrap();
    
            Ok(())
        }
    

    Right, the problem is non compat downcast

  5. modified the milestone: 14.0.0 on Oct 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions