You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
write_parquet() / write_csv_arrow() cannot stream a dataset object back to S3 #30615
It seems we cannot send a dataset (table) object opened with open_dataset() into write_parquet() or write_csv_arrow(). I realize that the former is a 'multi-file' representation and the latter two are not, but as both represent a single table (single schema) it seems like this should still work. (It also looks like write_csv_arrow() gives us more control over formatting details than write_dataset() )
Nicola Crane / @thisisnic:
Hi @cboettig , thanks for reporting this! I totally agree that this functionality should be implemented - and we have a couple of tickets open which when completed will allow this, so I'm going to close this ticket as a duplicate (though always good to know that there is user interest in this getting prioritised)
Activity
asfimport commentedon Dec 14, 2021
Nicola Crane / @thisisnic:
Hi @cboettig , thanks for reporting this! I totally agree that this functionality should be implemented - and we have a couple of tickets open which when completed will allow this, so I'm going to close this ticket as a duplicate (though always good to know that there is user interest in this getting prioritised)
asfimport commentedon Dec 14, 2021
Carl Boettiger / @cboettig:
Thanks much! sorry for the duplicate and I'll keep an eye on those issues you linked.