You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enable updating Parquet key-value metadata of opened ParquetWriter before closing #41608
In pyarrow, it would be great to have the ability to update Parquet key-value metadata of an existing ParquetWriter instance before closing the file. For example:
This is akin to the arrow-rs ArrowWriter::append_key_value_metadata, which lets you mutate an existing writer to add key-value metadata before writing the final metadata.
My use case for this is in writing to GeoParquet efficiently. Part of the GeoParquet metadata (stored as Parquet key-value metadata) includes an optional bounding box of all geometries in the file. But many applications might need a pass over the dataset to infer the bounding box of the data. If this key-value metadata can only be defined when opening the ParquetWriter, then streaming applications (such as converting other file formats to GeoParquet) might need two passes over the data: one to infer the bounding box and construct the schema to pass to the ParquetWriter constructor, and other to actually write the batches.
Activity
mapleFU commentedon May 13, 2024
C++ has this API, but this might not integrated to Python SDK. Would you like to support this?
This means exporting this api ( 3bd57e3#diff-0042b97c44521fde7da4ce3b4446cd7bc85bbc25eae63d2b4896ce58241e3b94 ) to Python SDK
kylebarron commentedon May 13, 2024
That would be great. I don't know C++ myself so I can't help out. But I may be able to test a branch if necessary
mapleFU commentedon May 13, 2024
I would finish it in this week
GH-41608: [C++][Python] Extends the add_key_value to parquet::arrow a…
mapleFU commentedon Jun 4, 2024
Issue resolved by pull request 41633
#41633
mapleFU commentedon Jun 4, 2024
Sorry for delay, this would be in 17.0.0