Skip to content

Join Statement: Schema contains duplicate unqualified field name #155

Closed

Description

@alamb

Activity

  1. alamb commented on Apr 26, 2021

    @alamb
    ContributorAuthor

    Comment from GANG LIAO(gangliao) @ 2021-01-29T20:09:31.033+0000:

    join statement cannot distinguish two columns with the same name from two tables. [https://github.com/apache/arrow/issues/9307]
    
    [~jorgecarleitao]  

    Comment from R J(TurnOfACard) @ 2021-02-18T11:19:34.301+0000:

    When building a schema using `datafusion::logical_plan::plan::LogicalPlan::schema()` the returned schema lacks a table qualifier. I think when building the TableScan, it allocates a table_name, but the returned`datafusion::logical_plan::dfschema::DFField`'s have no reference to the provided table name. Attempting to join the schemas (in `datafusion::logical_plan::dfschema::DFSchema::new`) then results in a list of fields with conflicting names as they lack qualifiers.
    
    I am new to DataFusion but would be happy to add some unit tests and try to look at this on the weekend.

    Comment from R J(TurnOfACard) @ 2021-02-20T22:59:14.507+0000:

    I'm not sure there is an easy fix without making breaking changes to the public API. When building a join schema, it checks if the join set is valid (physical_plan::hash_utils::check_join_set_is_valid), which has a parent public API call (physical_plan::hash_utils::check_join_is_valid). This join is unaware of the registered name (CSV or parquet) as it is performed with arrow schemas rather than DataFusion schemas.
    
     
    
    EDIT:
    
    It could be my lack of knowledge of the DataFusion codebase, but it appears it would need a lot of changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions