Skip to content

Improve schema merging #4223

Open
Open

Description

@andygrove

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I am trying to work with the nyctaxi parquet data set which has one file per month. Over time, some of the types have changed. For example passenger_count started out as Int64 and was later changed to Float64.

Arrow-rs can not merge these schemas.

Other solutions (such as DuckDB) will merge these schemas and pick the least restrictive type (Float64).

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

  1. tustvold commented on Nov 15, 2022

    @tustvold
    Contributor

    I would have thought this logic would exist within the query engine, i.e. DataFusion, not the compute engine? In particular I would have thought it would be a TableProvider detail, that would generate plans with the relevant schema coercion logic?

  2. andygrove commented on Nov 15, 2022

    @andygrove
    MemberAuthor

    I'm fine with implementing this in DataFusion. It currently delegates to Schema::try_merge in this repo, though, so it would likely mean duplicating some of this code in DF. I'll transfer this issue.

  3. transferred this issue fromapache/arrow-rson Nov 15, 2022
  4. added this to the 15.0.0 milestone on Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions