You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make table partitioning accessible in register/read APIS #1185
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The ability to read partitioned tables is added the the ListingTable in #1141. This is a very common feature that would be interesting to expose in the highest level APIs such as ExecutionContext.read_xxx and ExecutionContext.register_xxx
Describe the solution you'd like
The cleanest solution would probably to have a partiotioning field in CsvReadOptions/AvroReadOptions and also make read_parquet use a ParquetReadOptions
Hi guys, coming back to this now, since we can already find partition in paths in the object store, and looking at the API, I think what is missing is the ability to infer ListingOptions::table_partition_cols from the base path given to the TableProvider.
The table provider should then be able to easily merge the two schemas (the object store paths and the file format) for the exec.
Am I getting this right ?
@Igosuki I think the idea is to extend these public apis to allow users explicitly specify table partition columns (i.e. the table_partition_cols field) using either dataframe or sql interface. The auto partitioning discovery feature can be enabled when no explicit partitioning is specified by the user.
I think there is value in making partition column statically typed as well instead of only treating them as strings. But this is out of scope for this issue.
Activity
jimexist commentedon Nov 10, 2021
@rdettai i closed #1220 and maybe we can think of a way to adapt two styles
Igosuki commentedon Dec 15, 2021
Hi guys, coming back to this now, since we can already find partition in paths in the object store, and looking at the API, I think what is missing is the ability to infer ListingOptions::table_partition_cols from the base path given to the TableProvider.
The table provider should then be able to easily merge the two schemas (the object store paths and the file format) for the exec.
Am I getting this right ?
houqp commentedon Dec 20, 2021
@Igosuki I think the idea is to extend these public apis to allow users explicitly specify table partition columns (i.e. the
table_partition_colsfield) using either dataframe or sql interface. The auto partitioning discovery feature can be enabled when no explicit partitioning is specified by the user.I think there is value in making partition column statically typed as well instead of only treating them as strings. But this is out of scope for this issue.
rdettai commentedon Dec 26, 2021
@houqp not sure I would tag this as
good first issueas it requires to change the API and a fair amount of plumbing (i.e touch quite a few files)alamb commentedon Feb 3, 2025
I believe
ListingOptions::table_partition_colsnow implements what is described in this ticke so closing and claiming successhttps://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingOptions.html#structfield.table_partition_cols
feat: Make native shuffle compression configurable and respect `spark…
feat: add support for array_contains expression (apache#1163)