This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Promote _to_arrow_iterable to public method #1072
Description
Is your feature request related to a problem? Please describe.
Currently, it is possible to stream RecordBatches using RowIterator._to_arrow_iterable and this is done internally in the to_arrow method. There is a public to_dataframe_iterable method and so it feels like there should be a matching to_arrow_iterable method for the same reasons the Pandas version exists.
Describe the solution you'd like
Promote _to_arrow_iterable to to_arrow_iterable
Activity
tswast commentedon Nov 19, 2021
Great suggestion! I'm personally curious how you will be using this method but have no problems with adding it. I think we left it private to begin with because we weren't sure there would be demand for it.
judahrand commentedon Nov 19, 2021
I'm currently contributing to Feast (the feature store project) and at the moment everything which gets streamed from BigQuery is converted from Arrow to a Protobuf representation (hopefully this changes soon) and then pushed to a key value store.
Without the
to_arrow_iterablemethod there is no choice but to wait for the whole dataset to load before beginning to convert and load into the online store. This can be rather a lot of data - I ran into a case where it took 1 hour to load the dataset yesterday (not 100% how much of this was query time and download time but it was multiple GB).I'm keen to use the iterable version to improve this - allowing quicker debugging and lower memory usage.