Skip to content

[C++/Python] Feather format cannot accommodate string columns containing more than a total of 2GB of data #17898

Closed

Description

@asfimport

This can be fixed by migrating Feather to using the Arrow file/stream format internally. As soon as we have R bindings for Arrow, we can do this

See wesm/feather#325

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

Related issues:

Note: This issue was originally created as ARROW-1907. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Apr 27, 2018

    @asfimport
    Author

    Sritanu Chakraborty:
    What's the current state of this issue? I am trying to convert a pandas dataframe of 10 Gb(approx) to feather. I am getting the same error:

    TypeError: Cannot convert pyarrow.lib.ChunkedArray to pyarrow.lib.Array

     

  2. asfimport commented on Aug 21, 2019

    @asfimport
    Author

    Wes McKinney / @wesm:
    Now that the R library is available on CRAN, tackling this issue should be a little bit easier

  3. asfimport commented on Aug 21, 2019

    @asfimport
    Author

    Neal Richardson / @nealrichardson:
    Do ARROW-3308 and ARROW-3317 cover this, or is there more involved?

  4. asfimport commented on Feb 18, 2020

    @asfimport
    Author

    Wes McKinney / @wesm:
    This will be handled by the Feather V2 project (since the IPC protocol will be used to represent the data on disk)

  5. added this to the 0.17.0 milestone on Jan 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions