You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems there is a regression in version 22.0.0 related to pyarrow.flight.GeneratorStream and this change #47115 (based on the error message).
When using the GeneratorStream backed with anything other than pyarrow.RecordBatch iterable (like pyarrow.Table iterable or pyarrow.RecordBatchReader iterable) , it now fails with
Unknown error: Writer should be initialized before reading Next batches
Here is a minimal reproducer:
fromcollections.abcimportGeneratorimportpyarrowimportpyarrow.flight_TABLE=pyarrow.Table.from_pydict({
"city": ["Prague", "Paris", "London"],
"population": [1_397_880, 2_048_472, 9_089_736]
})
defbatch_iterator(table: pyarrow.Table) ->Generator[pyarrow.RecordBatch, None, None]:
forbatchintable.to_batches():
yieldbatch_TABLE_TICKET=pyarrow.flight.Ticket(b"table")
_BATCH_READER_TICKET=pyarrow.flight.Ticket(b"rbr")
_BATCH_TICKET=pyarrow.flight.Ticket(b"batch")
classFlightServer(pyarrow.flight.FlightServerBase):
defdo_get(self, context, ticket):
ifticket==_TABLE_TICKET:
returnpyarrow.flight.GeneratorStream(_TABLE.schema, [_TABLE])
elifticket==_BATCH_READER_TICKET:
returnpyarrow.flight.GeneratorStream(_TABLE.schema, [_TABLE.to_reader()])
else:
returnpyarrow.flight.GeneratorStream(_TABLE.schema, batch_iterator(_TABLE))
defmain():
server=FlightServer()
print("Serving on port", server.port)
client=pyarrow.flight.FlightClient(f"grpc+tcp://localhost:{server.port}")
# Does not work in pyarrow 22, works in 21# f = client.do_get(_TABLE_TICKET)# print(f.read_all())# Does not work in pyarrow 22, works in 21# f = client.do_get(_BATCH_READER_TICKET)# print(f.read_all())# Works in pyarrow 22 and 21f=client.do_get(_BATCH_TICKET)
print(f.read_all())
if__name__=='__main__':
main()
Only the iterator that goes batch-by-batch still works in 22.0.0, in older versions all three worked.
I am running on macOS 26.1 on M1 but I presume the bug is platform-agnostic.
I tried a naive fix in #48082 but no luck, the fix is going to be more complicated :-/ but the tests I added there should pass (and I they do pass on 21.0.0, they also pass on current main with #47115 reverted).
Activity
apacheGH-48076: [C++][Flight] fix GeneratorStream for Tables
apacheGH-48076: [C++][Flight] fix GeneratorStream for Tables
no23reason commentedon Nov 7, 2025
I tried a naive fix in #48082 but no luck, the fix is going to be more complicated :-/ but the tests I added there should pass (and I they do pass on 21.0.0, they also pass on current
mainwith #47115 reverted).apacheGH-48076: [C++][Flight] fix GeneratorStream for Tables
apacheGH-48076: [C++][Flight] fix GeneratorStream for Tables
apacheGH-48076: [C++][Flight] add more tests
apacheGH-48076: [C++][Flight] add more tests
GH-48076: [C++][Flight] fix GeneratorStream for Tables (#48082)
lidavidm commentedon Nov 14, 2025
Issue resolved by pull request 48082
#48082
apacheGH-48076: [C++][Flight] fix GeneratorStream for Tables (apache#…