You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
However it should be as detailed as possible. E.g. there's no way to tell from that output if f(x) is a function call or a column named f(x) which some of the optimizer passes do. This makes leads to an extremely frustrating debugging experience, esp. when some pass or step complains that a certain column wasn't found.
This would be a great first issue, at least to see what happens if you implement the suggestion in #6677 (comment)
It may end up with tests that fail that need to be updated or places in the code that use the Debug impl (e.g. {:?} rather than the Display impl (e.g. {}) but I think finding / tracking them down would be a good way to learn about (and improve) the codebase
I think the ticket from @crepererum was trying to say that formatting Exprs with {:?} (the Debug Impl) is lossy as it isn't in normal Rust style like
Expr::InList(InList{expr: .. })
Instead, the debug formatting is more like SQL style IN (x, y).
Thus, the idea here is to change the Debug impl to be the normal Rust style, and leave the Display impl the SQL style
However, given that Debug and Display, on the main branch, are currently the same, there will be some places that use the Debug style (e.g. {:?} that really should be using the Display style, such as the Hash Join output). Previously this didn't matter because the Debug and Display styles are the same
[SQL] explain select* from hashjoin_datatype_table_t1 t1 right join hashjoin_datatype_table_t2 t2 on t1.c3 = t2.c3
[Diff] (-expected|+actual)
logical_plan
Right Join: CAST(t1.c3 AS Decimal128(10, 2)) = t2.c3
--SubqueryAlias: t1
----TableScan: hashjoin_datatype_table_t1 projection=[c1, c2, c3, c4]
--SubqueryAlias: t2
----TableScan: hashjoin_datatype_table_t2 projection=[c1, c2, c3, c4]
physical_plan
ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, c1@5 as c1, c2@6 as c2, c3@7 as c3, c4@8 as c4]
- --SortMergeJoin: join_type=Right, on=[(Column { name: "CAST(t1.c3 AS Decimal128(10, 2))", index: 4 }, Column { name: "c3", index: 2 })]
- ----SortExec: expr=[CAST(t1.c3 AS Decimal128(10, 2))@4 ASC]
+ --SortMergeJoin: join_type=Right, on=[(Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4, c3@2)]
+ ----SortExec: expr=[Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4 ASC]
------CoalesceBatchesExec: target_batch_size=4096
- --------RepartitionExec: partitioning=Hash([Column { name: "CAST(t1.c3 AS Decimal128(10, 2))", index: 4 }], 2), input_partitions=2
- ----------ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, CAST(c3@2 AS Decimal128(10, 2)) as CAST(t1.c3 AS Decimal128(10, 2))]
+ --------RepartitionExec: partitioning=Hash([Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })@4], 2), input_partitions=2
+ ----------ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, c3@2 as c3, c4@3 as c4, CAST(c3@2 AS Decimal128(10, 2)) as Cast(Cast { expr: Column(Column { relation: Some(Bare { table: "t1" }), name: "c3" }), data_type: Decimal128(10, 2) })]
------------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
--------------MemoryExec: partitions=1, partition_sizes=[1]
----SortExec: expr=[c3@2 ASC]
------CoalesceBatchesExec: target_batch_size=4096
--------RepartitionExec: partitioning=Hash([c3@2], 2), input_partitions=2
----------RepartitionExec: partitioning=RoundRobinBatch(2), input_partitions=1
------------MemoryExec: partitions=1, partition_sizes=[1]
at tests/sqllogictests/test_files/joins.slt:2672
Case 1: I think changing the slt file is a good option.
Case 2: I failed to find where used {:?}😭. I think It caused by building the Column Struct by passing name is Debug String.
Thanks @crepererum , I already implement Display for Partiitioning. The main error is Cast Expr building the Column Struct by passing name is Debug String. Could you tell me physical_plan where build Column?
Activity
alamb commentedon Jun 15, 2023
I believe there was some historical reason for this but I can't recall the exact details
I suggest we try:
impl Debugtoimpl Display(as it is needed for naming columns)#[derive(Debug)]for Expr as suggested by @crepererumalamb commentedon Jun 15, 2023
This would be a great first issue, at least to see what happens if you implement the suggestion in #6677 (comment)
It may end up with tests that fail that need to be updated or places in the code that use the Debug impl (e.g.
{:?}rather than the Display impl (e.g.{}) but I think finding / tracking them down would be a good way to learn about (and improve) the codebaseparkma99 commentedon Jun 15, 2023
I will take it.
parkma99 commentedon Jun 17, 2023
I create this #6708 PR . Now it still has some error like this:
I am not sure to change the
sltfile or change the code. Thanks for your answer @crepererumalamb commentedon Jun 18, 2023
I think this difference is due to this code using
"{:?}"as a format string on theonexprs:https://github.com/apache/arrow-datafusion/blob/867ce6ee492e9473dca268e728567d7ba765b202/datafusion/core/src/physical_plan/joins/hash_join.rs#L429-L430
Perhaps we could change the HashJoin formatting code to use
{}as the format string (akaDisplay)?parkma99 commentedon Jun 19, 2023
I don't know using
{}is meansDebug impl Expr is lossyalamb commentedon Jun 19, 2023
I think the ticket from @crepererum was trying to say that formatting
Exprs with{:?}(the Debug Impl) is lossy as it isn't in normal Rust style likeInstead, the debug formatting is more like SQL style
IN (x, y).Thus, the idea here is to change the
Debugimpl to be the normal Rust style, and leave theDisplayimpl the SQL styleHowever, given that Debug and Display, on the main branch, are currently the same, there will be some places that use the
Debugstyle (e.g.{:?}that really should be using theDisplaystyle, such as the Hash Join output). Previously this didn't matter because the Debug and Display styles are the sameparkma99 commentedon Jun 19, 2023
parkma99 commentedon Jun 19, 2023
Hello @alamb , I update the code, got two kinds error:
Case 1: I think changing the slt file is a good option.
Case 2: I failed to find where used
{:?}😭. I think It caused by building theColumnStruct by passingnameisDebug String.crepererum commentedon Jun 20, 2023
@parkma99 I think you have to fix the Rust code to not use
DebugbutDisplayfor for the plan printers (fmt_as):SortMergeJoinSortExecRepartitionExecProjectionExecHere is one example fix:
https://github.com/apache/arrow-datafusion/blob/d29ab931506f23d8ef8fcd48ae523adfd3c07d6a/datafusion/core/src/physical_plan/repartition/mod.rs#L447-L462
partitioning={:?}should bepartitioning={}. HoweverPartitioningdoesn't implementDisplayyet, so you probably wanna hand-roll that.Similar fixes are required for the other exec nodes.
parkma99 commentedon Jun 20, 2023
Thanks @crepererum , I already implement
DisplayforPartiitioning. The main error isCast Exprbuilding theColumnStruct by passing name isDebugString. Could you tell mephysical_planwhere buildColumn?crepererum commentedon Jun 20, 2023
I'll have a look at your PR and debug it from there. Let's continue the conversation in there.