Skip to content

[Format] Feather V2 based on Arrow IPC file format, with compression support #21958

Closed

Description

@asfimport

The initial Feather file format is a minimal subset of the Arrow IPC format. It has a number of limitations (see https://wesmckinney.com/blog/feather-arrow-future/). 

We want to retain "feather" as the name of the on-disk representation of Arrow memory, so in order to support everything that Arrow supports, we need a "feather 2.0" format.

IIUC, defining the file format is "done" (dump the memory to disk). Remaining issues include upgrading "feather" readers and writers in all languages to support both feather 1.0 and feather 2.0. (e.g. https://issues.apache.org/jira/browse/ARROW-5501)

Reporter: Neal Richardson / @nealrichardson
Assignee: Wes McKinney / @wesm

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-5510. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Feb 17, 2020

    @asfimport
    Author

    Wes McKinney / @wesm:
    I started looking at this. As preliminaries I'm thinking about how to compress individual buffers in RecordBatch message bodies in the least invasive way

  2. asfimport commented on Mar 30, 2020

    @asfimport
    Author

    Wes McKinney / @wesm:
    Issue resolved by pull request 6694
    #6694

  3. added this to the 0.17.0 milestone on Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions