Skip to content

[C++][Flight] DoExchange doesn't support dictionary replacement #26727

Closed

Description

@asfimport

Looking at the server DoExchangeMessageWriter implementation, it seems to assume that the dictionary values for a given dictionary-encoded field will never change accross record batches.

Reporter: Antoine Pitrou / @pitrou

Related issues:

Note: This issue was originally created as ARROW-10787. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Dec 2, 2020

    @asfimport
    Author
  2. raulcd commented on Aug 19, 2025

    @raulcd
    Member

    The DoExchange code that generates the messages and the payloads does not reuse the code changed on server.cc here:

    It uses its own implementation here:

    // TODO(ARROW-10787): this should use the same writer/ipc trick as client
    class TransportMessageWriter final : public FlightMessageWriter {
    public:
    explicit TransportMessageWriter(ServerDataStream* stream)
    : stream_(stream), ipc_options_(::arrow::ipc::IpcWriteOptions::Defaults()) {}
    Status Begin(const std::shared_ptr<Schema>& schema,
    const ipc::IpcWriteOptions& options) override {
    if (started_) {
    return Status::Invalid("This writer has already been started.");
    }
    started_ = true;
    ipc_options_ = options;
    RETURN_NOT_OK(mapper_.AddSchemaFields(*schema));
    FlightPayload schema_payload;
    RETURN_NOT_OK(ipc::GetSchemaPayload(*schema, ipc_options_, mapper_,
    &schema_payload.ipc_message));
    return WritePayload(schema_payload);
    }
    Status WriteRecordBatch(const RecordBatch& batch) override {
    return WriteWithMetadata(batch, nullptr);
    }
    Status WriteMetadata(std::shared_ptr<Buffer> app_metadata) override {
    FlightPayload payload{};
    payload.app_metadata = app_metadata;
    return WritePayload(payload);
    }
    Status WriteWithMetadata(const RecordBatch& batch,
    std::shared_ptr<Buffer> app_metadata) override {
    RETURN_NOT_OK(CheckStarted());
    RETURN_NOT_OK(EnsureDictionariesWritten(batch));
    FlightPayload payload{};
    if (app_metadata) {
    payload.app_metadata = app_metadata;
    }
    RETURN_NOT_OK(ipc::GetRecordBatchPayload(batch, ipc_options_, &payload.ipc_message));
    RETURN_NOT_OK(WritePayload(payload));
    ++stats_.num_record_batches;
    return Status::OK();
    }
    Status Close() override {
    // It's fine to Close() without writing data
    return Status::OK();
    }
    ipc::WriteStats stats() const override { return stats_; }
    private:
    Status WritePayload(const FlightPayload& payload) {
    ARROW_ASSIGN_OR_RAISE(auto success, stream_->WriteData(payload));
    if (!success) {
    return MakeFlightError(FlightStatusCode::Internal,
    "Could not write metadata to stream (client disconnect?)");
    }
    ++stats_.num_messages;
    return Status::OK();
    }
    Status CheckStarted() {
    if (!started_) {
    return Status::Invalid("This writer is not started. Call Begin() with a schema");
    }
    return Status::OK();
    }
    Status EnsureDictionariesWritten(const RecordBatch& batch) {
    if (dictionaries_written_) {
    return Status::OK();
    }
    dictionaries_written_ = true;
    ARROW_ASSIGN_OR_RAISE(const auto dictionaries,
    ipc::CollectDictionaries(batch, mapper_));
    for (const auto& pair : dictionaries) {
    FlightPayload payload{};
    RETURN_NOT_OK(ipc::GetDictionaryPayload(pair.first, pair.second, ipc_options_,
    &payload.ipc_message));
    RETURN_NOT_OK(WritePayload(payload));
    ++stats_.num_dictionary_batches;
    }
    return Status::OK();
    }
    ServerDataStream* stream_;
    ::arrow::ipc::IpcWriteOptions ipc_options_;
    ipc::DictionaryFieldMapper mapper_;
    ipc::WriteStats stats_;
    bool started_ = false;
    bool dictionaries_written_ = false;
    };

    I think we can do the same trick we did for DoGet were we use an internal ipc::RecordBatchWriter and our own IpcPayloadWriter to generate the FlightPayload so we can get rid of the manual dictionary collection and generation.
    @lidavidm what do you think?

  3. lidavidm commented on Aug 20, 2025

    @lidavidm
    Member

    Yes, well, that's why there's a comment there saying we should do that :)

  4. added 2 commits that reference this issue on Aug 25, 2025
  5. added this to the 22.0.0 milestone on Aug 26, 2025
  6. raulcd commented on Aug 26, 2025

    @raulcd
    Member

    Issue resolved by pull request 47410
    #47410

  7. added a commit that references this issue on May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions