Skip to content

Debug impl Expr is lossy #6677

Closed
Closed

Description

@crepererum

Describe the bug

The Debug impl for Expr is a human-readable "nice" output:

https://github.com/apache/arrow-datafusion/blob/6872535358c2c6b70350484e6cc7c058c7a806ae/datafusion/expr/src/expr.rs#L872-L883

However it should be as detailed as possible. E.g. there's no way to tell from that output if f(x) is a function call or a column named f(x) which some of the optimizer passes do. This makes leads to an extremely frustrating debugging experience, esp. when some pass or step complains that a certain column wasn't found.

To Reproduce

No response

Expected behavior

Just auto-derive Debug.

Additional context

No response

Activity

  1. alamb commented on Jun 15, 2023

    @alamb
    Contributor

    I believe there was some historical reason for this but I can't recall the exact details

    I suggest we try:

    1. Move the current implementation of impl Debug to impl Display (as it is needed for naming columns)
    2. Use an automatic #[derive(Debug)] for Expr as suggested by @crepererum
  2. alamb commented on Jun 15, 2023

    @alamb
    Contributor

    This would be a great first issue, at least to see what happens if you implement the suggestion in #6677 (comment)

    It may end up with tests that fail that need to be updated or places in the code that use the Debug impl (e.g. {:?} rather than the Display impl (e.g. {}) but I think finding / tracking them down would be a good way to learn about (and improve) the codebase

  3. parkma99 commented on Jun 15, 2023

    @parkma99
    Contributor

    I will take it.

  4. parkma99 commented on Jun 17, 2023

    @parkma99
    Contributor

    I create this #6708 PR . Now it still has some error like this:

    External error: query result mismatch:
    [SQL] EXPLAIN
    select *, join_t1.t1_id + 11
    from join_t1, join_t2
    where join_t1.t1_id + 11 = join_t2.t2_id
    [Diff] (-expected|+actual)
        logical_plan
        Projection: join_t1.t1_id, join_t1.t1_name, join_t1.t1_int, join_t2.t2_id, join_t2.t2_name, join_t2.t2_int, CAST(join_t1.t1_id AS Int64) + Int64(11)
        --Inner Join: CAST(join_t1.t1_id AS Int64) + Int64(11) = CAST(join_t2.t2_id AS Int64)
        ----TableScan: join_t1 projection=[t1_id, t1_name, t1_int]
        ----TableScan: join_t2 projection=[t2_id, t2_name, t2_int]
        physical_plan
        ProjectionExec: expr=[t1_id@0 as t1_id, t1_name@1 as t1_name, t1_int@2 as t1_int, t2_id@3 as t2_id, t2_name@4 as t2_name, t2_int@5 as t2_int, CAST(t1_id@0 AS Int64) + 11 as join_t1.t1_id + Int64(11)]
        --ProjectionExec: expr=[t1_id@0 as t1_id, t1_name@1 as t1_name, t1_int@2 as t1_int, t2_id@4 as t2_id, t2_name@5 as t2_name, t2_int@6 as t2_int]
        ----CoalesceBatchesExec: target_batch_size=4096
    -   ------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "join_t1.t1_id + Int64(11)", index: 3 }, Column { name: "CAST(join_t2.t2_id AS Int64)", index: 3 })]
    +   ------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "join_t1.t1_id + Int64(11)", index: 3 }, Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"join_t2\" }), name: \"t2_id\" }), data_type: Int64 })", index: 3 })]
        --------CoalescePartitionsExec
        ----------ProjectionExec: expr=[t1_id@0 as t1_id, t1_name@1 as t1_name, t1_int@2 as t1_int, CAST(t1_id@0 AS Int64) + 11 as join_t1.t1_id + Int64(11)]
        ------------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
        --------------MemoryExec: partitions=1, partition_sizes=[1]
    -   --------ProjectionExec: expr=[t2_id@0 as t2_id, t2_name@1 as t2_name, t2_int@2 as t2_int, CAST(t2_id@0 AS Int64) as CAST(join_t2.t2_id AS Int64)]
    +   --------ProjectionExec: expr=[t2_id@0 as t2_id, t2_name@1 as t2_name, t2_int@2 as t2_int, CAST(t2_id@0 AS Int64) as Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "join_t2" }), name: "t2_id" }), data_type: Int64 })]
        ----------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
        ------------MemoryExec: partitions=1, partition_sizes=[1]
    at tests/sqllogictests/test_files/joins.slt:1430
    
    External error: query result mismatch:
    [SQL] explain
    (
        SELECT id, name FROM t1
        EXCEPT
        SELECT id, name FROM t2
    )
    UNION ALL
    (
        SELECT id, name FROM t2
        EXCEPT
        SELECT id, name FROM t1
    )
    [Diff] (-expected|+actual)
        logical_plan
        Union
        --LeftAnti Join: t1.id = CAST(t2.id AS Int32), t1.name = t2.name
        ----Aggregate: groupBy=[[t1.id, t1.name]], aggr=[[]]
        ------TableScan: t1 projection=[id, name]
        ----TableScan: t2 projection=[id, name]
        --Projection: CAST(t2.id AS Int32) AS id, t2.name
        ----LeftAnti Join: CAST(t2.id AS Int32) = t1.id, t2.name = t1.name
        ------Aggregate: groupBy=[[t2.id, t2.name]], aggr=[[]]
        --------TableScan: t2 projection=[id, name]
        ------TableScan: t1 projection=[id, name]
        physical_plan
        UnionExec
        --ProjectionExec: expr=[id@0 as id, name@1 as name]
        ----CoalesceBatchesExec: target_batch_size=8192
    -   ------HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(Column { name: "id", index: 0 }, Column { name: "CAST(t2.id AS Int32)", index: 2 }), (Column { name: "name", index: 1 }, Column { name: "name", index: 1 })]
    +   ------HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(Column { name: "id", index: 0 }, Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"t2\" }), name: \"id\" }), data_type: Int32 })", index: 2 }), (Column { name: "name", index: 1 }, Column { name: "name", index: 1 })]
        --------AggregateExec: mode=FinalPartitioned, gby=[id@0 as id, name@1 as name], aggr=[]
        ----------CoalesceBatchesExec: target_batch_size=8192
        ------------RepartitionExec: partitioning=Hash([Column { name: "id", index: 0 }, Column { name: "name", index: 1 }], 4), input_partitions=4
        --------------AggregateExec: mode=Partial, gby=[id@0 as id, name@1 as name], aggr=[]
        ----------------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
        --------CoalesceBatchesExec: target_batch_size=8192
    -   ----------RepartitionExec: partitioning=Hash([Column { name: "CAST(t2.id AS Int32)", index: 2 }, Column { name: "name", index: 1 }], 4), input_partitions=4
    -   ------------ProjectionExec: expr=[id@0 as id, name@1 as name, CAST(id@0 AS Int32) as CAST(t2.id AS Int32)]
    +   ----------RepartitionExec: partitioning=Hash([Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"t2\" }), name: \"id\" }), data_type: Int32 })", index: 2 }, Column { name: "name", index: 1 }], 4), input_partitions=4
    +   ------------ProjectionExec: expr=[id@0 as id, name@1 as name, CAST(id@0 AS Int32) as Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t2" }), name: "id" }), data_type: Int32 })]
        --------------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
        --ProjectionExec: expr=[CAST(id@0 AS Int32) as id, name@1 as name]
        ----ProjectionExec: expr=[id@0 as id, name@1 as name]
        ------CoalesceBatchesExec: target_batch_size=8192
    -   --------HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(Column { name: "CAST(t2.id AS Int32)", index: 2 }, Column { name: "id", index: 0 }), (Column { name: "name", index: 1 }, Column { name: "name", index: 1 })]
    +   --------HashJoinExec: mode=Partitioned, join_type=LeftAnti, on=[(Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"t2\" }), name: \"id\" }), data_type: Int32 })", index: 2 }, Column { name: "id", index: 0 }), (Column { name: "name", index: 1 }, Column { name: "name", index: 1 })]
        ----------CoalesceBatchesExec: target_batch_size=8192
    -   ------------RepartitionExec: partitioning=Hash([Column { name: "CAST(t2.id AS Int32)", index: 2 }, Column { name: "name", index: 1 }], 4), input_partitions=4
    -   --------------ProjectionExec: expr=[id@0 as id, name@1 as name, CAST(id@0 AS Int32) as CAST(t2.id AS Int32)]
    +   ------------RepartitionExec: partitioning=Hash([Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"t2\" }), name: \"id\" }), data_type: Int32 })", index: 2 }, Column { name: "name", index: 1 }], 4), input_partitions=4
    +   --------------ProjectionExec: expr=[id@0 as id, name@1 as name, CAST(id@0 AS Int32) as Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t2" }), name: "id" }), data_type: Int32 })]
        ----------------AggregateExec: mode=FinalPartitioned, gby=[id@0 as id, name@1 as name], aggr=[]
        ------------------CoalesceBatchesExec: target_batch_size=8192
        --------------------RepartitionExec: partitioning=Hash([Column { name: "id", index: 0 }, Column { name: "name", index: 1 }], 4), input_partitions=4
        ----------------------AggregateExec: mode=Partial, gby=[id@0 as id, name@1 as name], aggr=[]
        ------------------------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
        ----------CoalesceBatchesExec: target_batch_size=8192
        ------------RepartitionExec: partitioning=Hash([Column { name: "id", index: 0 }, Column { name: "name", index: 1 }], 4), input_partitions=4
        --------------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
    at tests/sqllogictests/test_files/union.slt:178
    
    Error: Execution("2 failures")
    error: test failed, to rerun pass `-p datafusion --test sqllogictests`

    I am not sure to change the slt file or change the code. Thanks for your answer @crepererum

  5. alamb commented on Jun 18, 2023

    @alamb
    Contributor
    +   ------HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(Column { name: "join_t1.t1_id + Int64(11)", index: 3 }, Column { name: "Cast(Cast { expr: Column(Column { relation: Some(Bare { table: \"join_t2\" }), name: \"t2_id\" }), data_type: Int64 })", index: 3 })]
    

    I think this difference is due to this code using "{:?}" as a format string on the on exprs:

    https://github.com/apache/arrow-datafusion/blob/867ce6ee492e9473dca268e728567d7ba765b202/datafusion/core/src/physical_plan/joins/hash_join.rs#L429-L430

    Perhaps we could change the HashJoin formatting code to use {} as the format string (aka Display)?

  6. parkma99 commented on Jun 19, 2023

    @parkma99
    Contributor

    Perhaps we could change the HashJoin formatting code to use {} as the format string (aka Display)?

    I don't know using {} is means Debug impl Expr is lossy

  7. alamb commented on Jun 19, 2023

    @alamb
    Contributor

    I think the ticket from @crepererum was trying to say that formatting Exprs with {:?} (the Debug Impl) is lossy as it isn't in normal Rust style like

    Expr::InList(
      InList { expr: .. }
    )

    Instead, the debug formatting is more like SQL style IN (x, y).

    Thus, the idea here is to change the Debug impl to be the normal Rust style, and leave the Display impl the SQL style

    However, given that Debug and Display, on the main branch, are currently the same, there will be some places that use the Debug style (e.g. {:?} that really should be using the Display style, such as the Hash Join output). Previously this didn't matter because the Debug and Display styles are the same

  8. parkma99 commented on Jun 19, 2023

    @parkma99
    Contributor

    Thus, the idea here is to change the Debug impl to be the normal Rust style, and leave the Display impl the SQL style
    Thanks @alamb

  9. parkma99 commented on Jun 19, 2023

    @parkma99
    Contributor

    Hello @alamb , I update the code, got two kinds error:

    [SQL] explain SELECT t1_id, (SELECT sum(t2_int) FROM t2 WHERE t2.t2_id = t1.t1_id) as t2_sum from t1
    [Diff] (-expected|+actual)
        logical_plan
        Projection: t1.t1_id, __scalar_sq_2.SUM(t2.t2_int) AS t2_sum
        --Left Join: t1.t1_id = __scalar_sq_2.t2_id
        ----TableScan: t1 projection=[t1_id]
        ----SubqueryAlias: __scalar_sq_2
        ------Projection: SUM(t2.t2_int), t2.t2_id
        --------Aggregate: groupBy=[[t2.t2_id]], aggr=[[SUM(t2.t2_int)]]
        ----------TableScan: t2 projection=[t2_id, t2_int]
        physical_plan
        ProjectionExec: expr=[t1_id@0 as t1_id, SUM(t2.t2_int)@1 as t2_sum]
        --CoalesceBatchesExec: target_batch_size=8192
    -   ----HashJoinExec: mode=Partitioned, join_type=Left, on=[(Column { name: "t1_id", index: 0 }, Column { name: "t2_id", index: 1 })]
    +   ----HashJoinExec: mode=Partitioned, join_type=Left, on=[(t1_id@0, t2_id@1)]
        ------CoalesceBatchesExec: target_batch_size=8192
    -   --------RepartitionExec: partitioning=Hash([Column { name: "t1_id", index: 0 }], 4), input_partitions=4
    +   --------RepartitionExec: partitioning=Hash([t1_id@0], 4), input_partitions=4
        ----------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
        ------ProjectionExec: expr=[SUM(t2.t2_int)@1 as SUM(t2.t2_int), t2_id@0 as t2_id]
        --------AggregateExec: mode=FinalPartitioned, gby=[t2_id@0 as t2_id], aggr=[SUM(t2.t2_int)]
        ----------CoalesceBatchesExec: target_batch_size=8192
    -   ------------RepartitionExec: partitioning=Hash([Column { name: "t2_id", index: 0 }], 4), input_partitions=4
    +   ------------RepartitionExec: partitioning=Hash([t2_id@0], 4), input_partitions=4
        --------------AggregateExec: mode=Partial, gby=[t2_id@0 as t2_id], aggr=[SUM(t2.t2_int)]
        ----------------MemoryExec: partitions=4, partition_sizes=[1, 0, 0, 0]
    at tests/sqllogictests/test_files/subquery.slt:112
    [SQL] explain select * from hashjoin_datatype_table_t1 t1 right join hashjoin_datatype_table_t2 t2 on t1.c3 = t2.c3
    [Diff] (-expected|+actual)
        logical_plan
        Right Join: CAST(t1.c3 AS Decimal128(10, 2)) = t2.c3
        --SubqueryAlias: t1
        ----TableScan: hashjoin_datatype_table_t1 projection=[c1, c2, c3, c4]
        --SubqueryAlias: t2
        ----TableScan: hashjoin_datatype_table_t2 projection=[c1, c2, c3, c4]
        physical_plan
        ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, c1@5 as c1, c2@6 as c2, c3@7 as c3, c4@8 as c4]
    -   --SortMergeJoin: join_type=Right, on=[(Column { name: "CAST(t1.c3 AS Decimal128(10, 2))", index: 4 }, Column { name: "c3", index: 2 })]
    -   ----SortExec: expr=[CAST(t1.c3 AS Decimal128(10, 2))@4 ASC]
    +   --SortMergeJoin: join_type=Right, on=[(Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4, c3@2)]
    +   ----SortExec: expr=[Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4 ASC]
        ------CoalesceBatchesExec: target_batch_size=4096
    -   --------RepartitionExec: partitioning=Hash([Column { name: "CAST(t1.c3 AS Decimal128(10, 2))", index: 4 }], 2), input_partitions=2
    -   ----------ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, CAST(c3@2 AS Decimal128(10, 2)) as CAST(t1.c3 AS Decimal128(10, 2))]
    +   --------RepartitionExec: partitioning=Hash([Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4], 2), input_partitions=2
    +   ----------ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, CAST(c3@2 AS Decimal128(10, 2)) as Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })]
        ------------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
        --------------MemoryExec: partitions=1, partition_sizes=[1]
        ----SortExec: expr=[c3@2 ASC]
        ------CoalesceBatchesExec: target_batch_size=4096
        --------RepartitionExec: partitioning=Hash([c3@2], 2), input_partitions=2
        ----------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
        ------------MemoryExec: partitions=1, partition_sizes=[1]
    at tests/sqllogictests/test_files/joins.slt:2672

    Case 1: I think changing the slt file is a good option.
    Case 2: I failed to find where used {:?}😭. I think It caused by building the Column Struct by passing name is Debug String.

  10. crepererum commented on Jun 20, 2023

    @crepererum
    ContributorAuthor

    @parkma99 I think you have to fix the Rust code to not use Debug but Display for for the plan printers (fmt_as):

    • SortMergeJoin
    • SortExec
    • RepartitionExec
    • ProjectionExec

    Here is one example fix:

    https://github.com/apache/arrow-datafusion/blob/d29ab931506f23d8ef8fcd48ae523adfd3c07d6a/datafusion/core/src/physical_plan/repartition/mod.rs#L447-L462

    partitioning={:?} should be partitioning={}. However Partitioning doesn't implement Display yet, so you probably wanna hand-roll that.

    Similar fixes are required for the other exec nodes.

  11. parkma99 commented on Jun 20, 2023

    @parkma99
    Contributor

    Thanks @crepererum , I already implement Display for Partiitioning. The main error is Cast Expr building the Column Struct by passing name is Debug String. Could you tell me physical_plan where build Column?

  12. crepererum commented on Jun 20, 2023

    @crepererum
    ContributorAuthor

    I'll have a look at your PR and debug it from there. Let's continue the conversation in there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions