Repository navigation
GH-45055: [C++][Flight] Update Flight Server RecordBatchStreamImpl to reuse ipc::RecordBatchWriter with custom IpcPayloadWriter instead of manually generating FlightPayload - #47115
Conversation
|
|
…e flight payloads for RecordBatches" This reverts commit c1eae76.
… payloads for RecordBatches
…multiple dictionaries) and possibly drop changes to schema
|
@lidavidm I've tried to play some tricks to reuse the I found a couple of special cases from our tests that surprised me because they diverge from my expectations over IPC. I am not adding more tests because with the change on the Python test I am already covering the original issue for the replacement dict on DoGet and with the previous tests we cover possible regressions. Let me know if you would like other test cases to be tackled. |
| // Check if writer is already initialized or if schema has changed. | ||
| // To recreate the writer. |
There was a problem hiding this comment.
I'm not actually sure what that test is about...we don't support changing schemas on the same stream (this was discussed on the ML a while back and discarded for lack of anyone to push it forward with all the bikeshedding)
There was a problem hiding this comment.
The test is TestFlightSqlServer.TestCommandGetTablesWithIncludedSchemas and failed with:
/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/example/sqlite_server.cc:395: GetTables include_schema=1
/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/example/sqlite_server.cc:409: GetTables: SELECT 'main' as catalog_name, null as schema_name, name as table_name, type as table_type FROM sqlite_master where 1=1 and table_name LIKE 'int%' order by table_name
Reading next payload from data stream: 1
/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/server_test.cc:325: Failure
Failed
'_error_or_value62.status()' failed with Invalid: Tried to write record batch with different schema
/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:339 writer_->WriteRecordBatch(*batch)
/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:415 impl_->Next(&payload)
/home/raulcd/code/arrow/cpp/src/arrow/flight/transport_server.cc:291 data_stream->Next(). gRPC client debug context: {"created":"@1753261907.620461787","description":"Error received from peer ipv4:127.0.0.1:40331","file":"/home/raulcd/code/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Tried to write record batch with different schema\n/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:339 writer_->WriteRecordBatch(*batch)\n/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:415 impl_->Next(&payload)\n/home/raulcd/code/arrow/cpp/src/arrow/flight/transport_server.cc:291 data_stream->Next()","grpc_status":3}. Client context: OK
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:219 stream_->Finish(Status::OK())
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:256 Next()
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:116 ToRecordBatches(stop_token)The schemas difference was table_schema from binary to binary not null:
## Batch schema:
catalog_name: string
db_schema_name: string
table_name: string not null
table_type: string not null
table_schema: binary
## Reader Schema:
catalog_name: string
db_schema_name: string
table_name: string not null
table_type: string not null
table_schema: binary not nullI've found the issue and fixed here: d4305a8
|
Appveyor is timing out cloning the repository for the last builds (not only on this PR). It is not related to this PR and was successful on previous commits so CI is green. |
|
Thanks for tackling this Raúl! |
|
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 25f8f00. There weren't enough matching historic benchmark results to make a call on whether there were regressions. The full Conbench report has more details. |
|
For the record, does this also enable dictionary deltas or just dictionary replacement? |
|
Both as it uses the underlying arrow/cpp/src/arrow/ipc/writer.cc Lines 1272 to 1340 in 71a7b55 It will generate the dictionary delta as we do on the Ipc RecordBatchWriter. Should I add a new test for that? edit: is the first time I am modifying this code base but I am pretty sure this is what happens :) |
If that's easy to do in Python then why not. The IPC reader exposes some useful |
|
that's interesting, I hacked something quick and dirty locally to validate dictionary deltas with flight just exposing the @lidavidm what do you think about exposing some |
|
I think we can just use the regular stats |
|
Though, we should clarify what kind of stats Rusty actually wants |
After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. So to fix this, remove the assertion and instead initialize the writer on first access.
After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. So to fix this, remove the assertion and instead initialize the writer on first access.
After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. So to fix this, remove the assertion and instead initialize the writer on first access. Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there is of the SCHEMA type and we want RECORD_BATCH one.
After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. So to fix this, remove the assertion and instead initialize the writer on first access. Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there is of the SCHEMA type and we want RECORD_BATCH one.
### Rationale for this change After the changes in #47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. ### What changes are included in this PR? Removal of the problematic assertion and initialization of the writer object when it is needed first. Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there is of the SCHEMA type and we want RECORD_BATCH or DICTIONARY_BATCH one. ### Are these changes tested? Yes, via CI. Tests for the GeneratorStreams were extended so that they test GeneratorStreams backed by Tables and RecordBatchReaders, not just RecordBatches. ### Are there any user-facing changes? No, just a fix for a regression restoring the functionality from version 21.0.0 and earlier. * GitHub Issue: #48076 Authored-by: Dan Homola <dan.homola@gooddata.com> Signed-off-by: David Li <li.davidm96@gmail.com>
…48082) ### Rationale for this change After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders. This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader. ### What changes are included in this PR? Removal of the problematic assertion and initialization of the writer object when it is needed first. Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there is of the SCHEMA type and we want RECORD_BATCH or DICTIONARY_BATCH one. ### Are these changes tested? Yes, via CI. Tests for the GeneratorStreams were extended so that they test GeneratorStreams backed by Tables and RecordBatchReaders, not just RecordBatches. ### Are there any user-facing changes? No, just a fix for a regression restoring the functionality from version 21.0.0 and earlier. * GitHub Issue: apache#48076 Authored-by: Dan Homola <dan.homola@gooddata.com> Signed-off-by: David Li <li.davidm96@gmail.com>
Rationale for this change
Currently Dictionary replacement or deltas are not supported in DoGet as seen on the original issue.
We could replicate the
ipc::WriteDictionariescode but instead of replicating this logic we can also update our flight serverRecordBatchStreamImplto reuseipc::RecordBatchWriterand just update the payload to match the flight payload.What changes are included in this PR?
Updated logic on
arrow::flight::RecordBatchStream::RecordBatchStreamImplto use anipc::RecordBatchWriterand a customipc::IpcPayloadWriterinstead of manually buidling the payloads.Are these changes tested?
Yes via CI. Test data for some Python tests has been updated to validate dictionary replacement is working now. On main the updated test fails with:
Are there any user-facing changes?
No