Skip to content

GH-45055: [C++][Flight] Update Flight Server RecordBatchStreamImpl to reuse ipc::RecordBatchWriter with custom IpcPayloadWriter instead of manually generating FlightPayload - #47115

Merged
lidavidm merged 13 commits into
apache:mainfrom
raulcd:GH-45055
Jul 25, 2025

Conversation

@raulcd

@raulcd raulcd commented Jul 15, 2025 •

Copy link
Copy Markdown
Member

Rationale for this change

Currently Dictionary replacement or deltas are not supported in DoGet as seen on the original issue.
We could replicate the ipc::WriteDictionaries code but instead of replicating this logic we can also update our flight server RecordBatchStreamImpl to reuse ipc::RecordBatchWriter and just update the payload to match the flight payload.

What changes are included in this PR?

Updated logic on arrow::flight::RecordBatchStream::RecordBatchStreamImpl to use an ipc::RecordBatchWriter and a custom ipc::IpcPayloadWriter instead of manually buidling the payloads.

Are these changes tested?

Yes via CI. Test data for some Python tests has been updated to validate dictionary replacement is working now. On main the updated test fails with:

>               assert data.equals(table)
E               assert False
E                +  where False = equals(pyarrow.Table\nsome_dicts: dictionary<values=string, indices=int64, ordered=0>\n----\nsome_dicts: [  -- dictionary:\n["foo...\n[1,0,null],  -- dictionary:\n["foo","baz","quux"]  -- indices:\n[2,1],  -- dictionary:\n["bar","qux"]  -- indices:\n[0,1]])
E                +    where equals = pyarrow.Table\nsome_dicts: dictionary<values=string, indices=int64, ordered=0>\n----\nsome_dicts: [  -- dictionary:\n["foo...ull],  -- dictionary:\n["foo","baz","quux"]  -- indices:\n[2,1],  -- dictionary:\n["foo","baz","quux"]  -- indices:\n[0,1]].equals

pyarrow/tests/test_flight.py:1319: AssertionError
================================================================================== short test summary info ===================================================================================
FAILED pyarrow/tests/test_flight.py::test_flight_do_get_dicts - assert False
FAILED pyarrow/tests/test_flight.py::test_flight_domain_socket - assert False

Are there any user-facing changes?

No

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #45055 has been automatically assigned in GitHub to PR creator.

@raulcd raulcd changed the title GH-45055: [C++][Flight] Support dictionary replacement in DoGet GH-45055: [C++][Flight] Update Flight Server RecordBatchStreamImpl to reuse ipc::RecordBatchWriter with custom IpcPayloadWriter instead of manually generating FlightPayload Jul 21, 2025
Comment thread cpp/src/arrow/flight/server.cc Resolved
@raulcd
raulcd marked this pull request as ready for review July 21, 2025 14:08
@raulcd
raulcd requested review from AlenkaF, lidavidm and rok as code owners July 21, 2025 14:08
@raulcd

raulcd commented Jul 21, 2025

Copy link
Copy Markdown
Member Author

@lidavidm I've tried to play some tricks to reuse the ipc::RecordBatchWriter with a new IpcPayloadWriter, similar to what we do on the client. This is the first time I am doing anything with Flight so I might have completely missed something, please let me know if that's the case :)

I found a couple of special cases from our tests that surprised me because they diverge from my expectations over IPC.
There's a flight sql test that seems to be changing the schema mid stream, is this expected? Can we return FlightPayloads that doesn't match the returned schema?

I am not adding more tests because with the change on the Python test I am already covering the original issue for the replacement dict on DoGet and with the previous tests we cover possible regressions. Let me know if you would like other test cases to be tackled.

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Left some remarks

Comment thread cpp/src/arrow/flight/server.cc Resolved
Comment thread cpp/src/arrow/flight/server.cc Resolved
Comment thread cpp/src/arrow/flight/server.cc Outdated
Comment on lines +330 to +331
// Check if writer is already initialized or if schema has changed.
// To recreate the writer.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not actually sure what that test is about...we don't support changing schemas on the same stream (this was discussed on the ML a while back and discarded for lack of anyone to push it forward with all the bikeshedding)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is TestFlightSqlServer.TestCommandGetTablesWithIncludedSchemas and failed with:

/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/example/sqlite_server.cc:395: GetTables include_schema=1
/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/example/sqlite_server.cc:409: GetTables: SELECT 'main' as catalog_name, null as schema_name, name as table_name, type as table_type FROM sqlite_master where 1=1 and table_name LIKE 'int%' order by table_name
Reading next payload from data stream: 1
/home/raulcd/code/arrow/cpp/src/arrow/flight/sql/server_test.cc:325: Failure
Failed
'_error_or_value62.status()' failed with Invalid: Tried to write record batch with different schema
/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:339  writer_->WriteRecordBatch(*batch)
/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:415  impl_->Next(&payload)
/home/raulcd/code/arrow/cpp/src/arrow/flight/transport_server.cc:291  data_stream->Next(). gRPC client debug context: {"created":"@1753261907.620461787","description":"Error received from peer ipv4:127.0.0.1:40331","file":"/home/raulcd/code/arrow/cpp/build/grpc_ep-prefix/src/grpc_ep/src/core/lib/surface/call.cc","file_line":952,"grpc_message":"Tried to write record batch with different schema\n/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:339  writer_->WriteRecordBatch(*batch)\n/home/raulcd/code/arrow/cpp/src/arrow/flight/server.cc:415  impl_->Next(&payload)\n/home/raulcd/code/arrow/cpp/src/arrow/flight/transport_server.cc:291  data_stream->Next()","grpc_status":3}. Client context: OK
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:219  stream_->Finish(Status::OK())
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:256  Next()
/home/raulcd/code/arrow/cpp/src/arrow/flight/client.cc:116  ToRecordBatches(stop_token)

The schemas difference was table_schema from binary to binary not null:

## Batch schema:
catalog_name: string
db_schema_name: string
table_name: string not null
table_type: string not null
table_schema: binary
## Reader Schema:
catalog_name: string
db_schema_name: string
table_name: string not null
table_type: string not null
table_schema: binary not null

I've found the issue and fixed here: d4305a8

Comment thread cpp/src/arrow/flight/server.cc Resolved
Comment thread cpp/src/arrow/flight/server.cc Resolved
Comment thread cpp/src/arrow/flight/server.cc Resolved
@raulcd

raulcd commented Jul 24, 2025

Copy link
Copy Markdown
Member Author

Appveyor is timing out cloning the repository for the last builds (not only on this PR). It is not related to this PR and was successful on previous commits so CI is green.

@lidavidm

Copy link
Copy Markdown
Member

Thanks for tackling this Raúl!

@raulcd
raulcd deleted the GH-45055 branch July 25, 2025 10:14
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 25f8f00.

There weren't enough matching historic benchmark results to make a call on whether there were regressions.

The full Conbench report has more details.

@pitrou

pitrou commented Aug 18, 2025

Copy link
Copy Markdown
Member

For the record, does this also enable dictionary deltas or just dictionary replacement?

@raulcd

raulcd commented Aug 18, 2025 •

Copy link
Copy Markdown
Member Author

Both as it uses the underlying RecordBatchWriter here:

Status WriteDictionaries(const RecordBatch& batch) {
ARROW_ASSIGN_OR_RAISE(const auto dictionaries, CollectDictionaries(batch, mapper_));
const auto equal_options = EqualOptions().nans_equal(true);
for (const auto& pair : dictionaries) {
int64_t dictionary_id = pair.first;
const auto& dictionary = pair.second;
// If a dictionary with this id was already emitted, check if it was the same.
auto* last_dictionary = &last_dictionaries_[dictionary_id];
const bool dictionary_exists = (*last_dictionary != nullptr);
int64_t delta_start = 0;
if (dictionary_exists) {
if ((*last_dictionary)->data() == dictionary->data()) {
// Fast shortcut for a common case.
// Same dictionary data by pointer => no need to emit it again
continue;
}
const int64_t last_length = (*last_dictionary)->length();
const int64_t new_length = dictionary->length();
if (new_length == last_length &&
((*last_dictionary)->Equals(dictionary, equal_options))) {
// Same dictionary by value => no need to emit it again
// (while this can have a CPU cost, this code path is required
// for the IPC file format)
continue;
}
// (the read path doesn't support outer dictionary deltas, don't emit them)
if (new_length > last_length && options_.emit_dictionary_deltas &&
!HasNestedDict(*dictionary->data()) &&
((*last_dictionary)
->RangeEquals(dictionary, 0, last_length, 0, equal_options))) {
// New dictionary starts with the current dictionary
delta_start = last_length;
}
if (is_file_format_ && !delta_start) {
return Status::Invalid(
"Dictionary replacement detected when writing IPC file format. "
"Arrow IPC files only support a single non-delta dictionary for "
"a given field across all batches.");
}
}
IpcPayload payload;
if (delta_start) {
RETURN_NOT_OK(GetDictionaryPayload(dictionary_id, /*is_delta=*/true,
dictionary->Slice(delta_start), options_,
&payload));
} else {
RETURN_NOT_OK(
GetDictionaryPayload(dictionary_id, dictionary, options_, &payload));
}
RETURN_NOT_OK(WritePayload(payload));
++stats_.num_dictionary_batches;
if (dictionary_exists) {
if (delta_start) {
++stats_.num_dictionary_deltas;
} else {
++stats_.num_replaced_dictionaries;
}
}
// Remember dictionary for next batches
*last_dictionary = dictionary;
}
return Status::OK();
}

It will generate the dictionary delta as we do on the Ipc RecordBatchWriter.

Should I add a new test for that?

edit: is the first time I am modifying this code base but I am pretty sure this is what happens :)

@pitrou

pitrou commented Aug 18, 2025

Copy link
Copy Markdown
Member

Should I add a new test for that?

If that's easy to do in Python then why not. The IPC reader exposes some useful ReadStats, but AFAICT the Flight client does not currently allows getting these stats. Perhaps it can be added?

@raulcd

raulcd commented Aug 18, 2025 •

Copy link
Copy Markdown
Member Author

that's interesting, I hacked something quick and dirty locally to validate dictionary deltas with flight just exposing the virtual arrow::ipc::ReadStats stats() const = 0; to the FlightStreamReader and was able to validate deltas work (of course using the correct IpcWriteOptions.emit_dictionary_deltas:

pyarrow/tests/test_flight.py Stats: {'num_messages': 7, 'num_record_batches': 3, 'num_dictionary_batches': 3, 'num_dictionary_deltas': 2, 'num_replaced_dictionaries': 0}

@lidavidm what do you think about exposing some ReadStats on the FlightStreamReader, Should this be it's own FlightStats instead of arrow::ipc::ReadStats?
I recall there were some requests to have a method exposing the stats like, which we can use to do this:

@lidavidm

Copy link
Copy Markdown
Member

I think we can just use the regular stats

@lidavidm

Copy link
Copy Markdown
Member

Though, we should clarify what kind of stats Rusty actually wants

no23reason added a commit to no23reason/arrow that referenced this pull request Nov 7, 2025
After the changes in apache#47115, GeneratorStreams backed by anything else
than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the
RecordBatchStream#GetSchemaPayload would always get called, which
is not the case when the GeneratorStream is backed by a Table
or a RecordBatchReader.

So to fix this, remove the assertion and instead initialize the writer
on first access.
no23reason added a commit to no23reason/arrow that referenced this pull request Nov 7, 2025
After the changes in apache#47115, GeneratorStreams backed by anything else
than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the
RecordBatchStream#GetSchemaPayload would always get called, which
is not the case when the GeneratorStream is backed by a Table
or a RecordBatchReader.

So to fix this, remove the assertion and instead initialize the writer
on first access.
no23reason added a commit to no23reason/arrow that referenced this pull request Nov 10, 2025
After the changes in apache#47115, GeneratorStreams backed by anything else
than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the
RecordBatchStream#GetSchemaPayload would always get called, which
is not the case when the GeneratorStream is backed by a Table
or a RecordBatchReader.

So to fix this, remove the assertion and instead initialize the writer
on first access. Also, to accommodate for this case, drop the incoming
message when initializing the writer in Next, as the message there
is of the SCHEMA type and we want RECORD_BATCH one.
no23reason added a commit to no23reason/arrow that referenced this pull request Nov 10, 2025
After the changes in apache#47115, GeneratorStreams backed by anything else
than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the
RecordBatchStream#GetSchemaPayload would always get called, which
is not the case when the GeneratorStream is backed by a Table
or a RecordBatchReader.

So to fix this, remove the assertion and instead initialize the writer
on first access. Also, to accommodate for this case, drop the incoming
message when initializing the writer in Next, as the message there
is of the SCHEMA type and we want RECORD_BATCH one.
lidavidm pushed a commit that referenced this pull request Nov 14, 2025
### Rationale for this change

After the changes in #47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader.

### What changes are included in this PR?

Removal of the problematic assertion and initialization of the writer object when it is needed first.
Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there
is of the SCHEMA type and we want RECORD_BATCH or DICTIONARY_BATCH one.

### Are these changes tested?

Yes, via CI. Tests for the GeneratorStreams were extended so that they test GeneratorStreams backed by Tables and RecordBatchReaders, not just RecordBatches. 

### Are there any user-facing changes?

No, just a fix for a regression restoring the functionality from version 21.0.0 and earlier.

* GitHub Issue: #48076

Authored-by: Dan Homola <dan.homola@gooddata.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Mottl pushed a commit to Mottl/arrow that referenced this pull request May 26, 2026
…48082)

### Rationale for this change

After the changes in apache#47115, GeneratorStreams backed by anything else than RecordBatches failed. This includes Tables and RecordBatchReaders.

This was caused by a too strict assumption that the RecordBatchStream#GetSchemaPayload would always get called, which is not the case when the GeneratorStream is backed by a Table or a RecordBatchReader.

### What changes are included in this PR?

Removal of the problematic assertion and initialization of the writer object when it is needed first.
Also, to accommodate for this case, drop the incoming message when initializing the writer in Next, as the message there
is of the SCHEMA type and we want RECORD_BATCH or DICTIONARY_BATCH one.

### Are these changes tested?

Yes, via CI. Tests for the GeneratorStreams were extended so that they test GeneratorStreams backed by Tables and RecordBatchReaders, not just RecordBatches. 

### Are there any user-facing changes?

No, just a fix for a regression restoring the functionality from version 21.0.0 and earlier.

* GitHub Issue: apache#48076

Authored-by: Dan Homola <dan.homola@gooddata.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants