You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I am trying to work with the nyctaxi parquet data set which has one file per month. Over time, some of the types have changed. For example passenger_count started out as Int64 and was later changed to Float64.
Arrow-rs can not merge these schemas.
Other solutions (such as DuckDB) will merge these schemas and pick the least restrictive type (Float64).
I would have thought this logic would exist within the query engine, i.e. DataFusion, not the compute engine? In particular I would have thought it would be a TableProvider detail, that would generate plans with the relevant schema coercion logic?
I'm fine with implementing this in DataFusion. It currently delegates to Schema::try_merge in this repo, though, so it would likely mean duplicating some of this code in DF. I'll transfer this issue.
Activity
tustvold commentedon Nov 15, 2022
I would have thought this logic would exist within the query engine, i.e. DataFusion, not the compute engine? In particular I would have thought it would be a
TableProviderdetail, that would generate plans with the relevant schema coercion logic?andygrove commentedon Nov 15, 2022
I'm fine with implementing this in DataFusion. It currently delegates to
Schema::try_mergein this repo, though, so it would likely mean duplicating some of this code in DF. I'll transfer this issue.