Skip to content

Make table partitioning accessible in register/read APIS #1185

Closed

Description

@rdettai

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The ability to read partitioned tables is added the the ListingTable in #1141. This is a very common feature that would be interesting to expose in the highest level APIs such as ExecutionContext.read_xxx and ExecutionContext.register_xxx

Describe the solution you'd like
The cleanest solution would probably to have a partiotioning field in CsvReadOptions/AvroReadOptions and also make read_parquet use a ParquetReadOptions

Activity

  1. jimexist commented on Nov 10, 2021

    @jimexist
    Member

    @rdettai i closed #1220 and maybe we can think of a way to adapt two styles

  2. Igosuki commented on Dec 15, 2021

    @Igosuki
    Contributor

    Hi guys, coming back to this now, since we can already find partition in paths in the object store, and looking at the API, I think what is missing is the ability to infer ListingOptions::table_partition_cols from the base path given to the TableProvider.
    The table provider should then be able to easily merge the two schemas (the object store paths and the file format) for the exec.
    Am I getting this right ?

  3. houqp commented on Dec 20, 2021

    @houqp
    Member

    @Igosuki I think the idea is to extend these public apis to allow users explicitly specify table partition columns (i.e. the table_partition_cols field) using either dataframe or sql interface. The auto partitioning discovery feature can be enabled when no explicit partitioning is specified by the user.

    I think there is value in making partition column statically typed as well instead of only treating them as strings. But this is out of scope for this issue.

  4. rdettai commented on Dec 26, 2021

    @rdettai
    ContributorAuthor

    @houqp not sure I would tag this as good first issue as it requires to change the API and a fair amount of plumbing (i.e touch quite a few files)

  5. alamb commented on Feb 3, 2025

    @alamb
    Contributor

    I believe ListingOptions::table_partition_cols now implements what is described in this ticke so closing and claiming success

    https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingOptions.html#structfield.table_partition_cols

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions