Skip to content

[C++][Dataset] arrow::dataset::Partitioning::Default() can't be used for writing dataset #15256

Closed

Description

@kou

Describe the bug, including details regarding any error messages, version, and platform.

Because arrow::dataset::DefaultPartitioning::Format() isn't implemented:

Result<PartitionPathFormat> Format(const compute::Expression& expr) const override {
return Status::NotImplemented("formatting paths from ", type_name(),
" Partitioning");
}

It's required in WriteBatch():

ARROW_ASSIGN_OR_RAISE(destination,
write_options.partitioning->Format(partition_expression));

Is it expected that we can't use arrow::dataset::Partitioning::Default() for writing dataset?
If it's expected, how about removing arrow::dataset::Partitioning::Default() because it's useless?
If it's not expected, how about implementing arrow::dataset::DefaultPartitioning::Format() like the following?

diff --git a/cpp/src/arrow/dataset/partition.cc b/cpp/src/arrow/dataset/partition.cc
index 46cdf9023c..13add35fb8 100644
--- a/cpp/src/arrow/dataset/partition.cc
+++ b/cpp/src/arrow/dataset/partition.cc
@@ -90,8 +90,7 @@ std::shared_ptr<Partitioning> Partitioning::Default() {
     }
 
     Result<PartitionPathFormat> Format(const compute::Expression& expr) const override {
-      return Status::NotImplemented("formatting paths from ", type_name(),
-                                    " Partitioning");
+      return PartitionPathFormat{"", ""};
     }
 
     Result<PartitionedBatches> Partition(

Component(s)

C++

Activity

  1. westonpace commented on Jan 10, 2023

    @westonpace
    Member
  2. kou commented on Jan 11, 2023

    @kou
    MemberAuthor

    Thanks! I didn't notice it.

    It seems that we want to use arrow::dataset::Partitioning::Default() for no partition write. (All data are written to the base directory.)

  3. westonpace commented on Jan 11, 2023

    @westonpace
    Member

    It seems that we want to use arrow::dataset::Partitioning::Default() for no partition write. (All data are written to the base directory.)

    I agree. Either Partitioning::Default means "no partitioning" or we get rid of Partitioning::Default and make partitioning optional anywhere it is specified. I think your suggestion (Partitioning::Default means "all in one directory") is the easiest.

  4. added a commit that references this issue on Apr 5, 2023
  5. added this to the 12.0.0 milestone on Apr 5, 2023
  6. added a commit that references this issue on May 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions