Skip to content

Enable updating Parquet key-value metadata of opened ParquetWriter before closing #41608

Closed

Description

@kylebarron

Describe the enhancement requested

In pyarrow, it would be great to have the ability to update Parquet key-value metadata of an existing ParquetWriter instance before closing the file. For example:

import pyarrow.parquet as pq

with pq.ParquetWriter("output.parquet", schema) as writer:
    writer.write_batch(...)
    writer.add_key_value_metadata({b"hello": "world"})

This is akin to the arrow-rs ArrowWriter::append_key_value_metadata, which lets you mutate an existing writer to add key-value metadata before writing the final metadata.

My use case for this is in writing to GeoParquet efficiently. Part of the GeoParquet metadata (stored as Parquet key-value metadata) includes an optional bounding box of all geometries in the file. But many applications might need a pass over the dataset to infer the bounding box of the data. If this key-value metadata can only be defined when opening the ParquetWriter, then streaming applications (such as converting other file formats to GeoParquet) might need two passes over the data: one to infer the bounding box and construct the schema to pass to the ParquetWriter constructor, and other to actually write the batches.

Component(s)

Python

Activity

  1. mapleFU commented on May 13, 2024

    @mapleFU
    Member

    C++ has this API, but this might not integrated to Python SDK. Would you like to support this?

    This means exporting this api ( 3bd57e3#diff-0042b97c44521fde7da4ce3b4446cd7bc85bbc25eae63d2b4896ce58241e3b94 ) to Python SDK

  2. kylebarron commented on May 13, 2024

    @kylebarron
    MemberAuthor

    That would be great. I don't know C++ myself so I can't help out. But I may be able to test a branch if necessary

  3. mapleFU commented on May 13, 2024

    @mapleFU
    Member

    I would finish it in this week

  4. added a commit that references this issue on Jun 4, 2024
  5. added this to the 17.0.0 milestone on Jun 4, 2024
  6. mapleFU commented on Jun 4, 2024

    @mapleFU
    Member

    Issue resolved by pull request 41633
    #41633

  7. mapleFU commented on Jun 4, 2024

    @mapleFU
    Member

    Sorry for delay, this would be in 17.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions