Skip to content

write_parquet() / write_csv_arrow() cannot stream a dataset object back to S3 #30615

Closed

Description

@asfimport

It seems we cannot send a dataset (table) object opened with open_dataset() into write_parquet() or write_csv_arrow().  I realize that the former is a 'multi-file' representation and the latter two are not, but as both represent a single table (single schema) it seems like this should still work.  (It also looks like write_csv_arrow() gives us more control over formatting details than write_dataset() )

Reporter: Carl Boettiger / @cboettig

Related issues:

Note: This issue was originally created as ARROW-15104. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Dec 14, 2021

    @asfimport
    Author

    Nicola Crane / @thisisnic:
    Hi @cboettig , thanks for reporting this!  I totally agree that this functionality should be implemented - and we have a couple of tickets open which when completed will allow this, so I'm going to close this ticket as a duplicate (though always good to know that there is user interest in this getting prioritised)

  2. asfimport commented on Dec 14, 2021

    @asfimport
    Author

    Carl Boettiger / @cboettig:
    Thanks much! sorry for the duplicate and I'll keep an eye on those issues you linked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions