Skip to content

[C++][Flight] Fully support dictionary replacement in Flight #45056

Closed

Description

@amoeba

Describe the enhancement requested

As of writing Flight C++ currently only supports dictionary replacement in DoPut calls and not in DoGet or (partially) DoExchange.

Sub-issues:

Component(s)

C++, FlightRPC

Activity

  1. pitrou commented on Aug 19, 2025

    @pitrou
    Member

    Is DoExchange also handled by #47115 @raulcd ? If so, can we update this issue and the corresponding sub-issue?

  2. raulcd commented on Aug 19, 2025

    @raulcd
    Member

    Is DoExchange also handled by #47115

    I'm not familiar with DoExchange so not sure if it reuses the same code path. I'll have to set up a test and try, I can work on that and will update with findings.

  3. self-assigned this
    on Aug 19, 2025
  4. raulcd commented on Aug 19, 2025

    @raulcd
    Member

    DoExchange doesn't seem to be using the same code path as the returned table is not allowing for dictionary replacement nor dictionary deltas. Test reproducer:

    def test_flight_dictionary_deltas_do_exchange():
        class DeltaFlightServer(ConstantFlightServer):
            def do_exchange(self, context, descriptor, reader, writer):
                if descriptor.command == b'dict_deltas':
                    table = simple_dicts_table()
                    options = pa.ipc.IpcWriteOptions(emit_dictionary_deltas=True)
                    writer.begin(table.schema, options=options)
                    writer.write_table(table)
    
        with DeltaFlightServer() as server, \
                FlightClient(('localhost', server.port)) as client:
    
            descriptor = flight.FlightDescriptor.for_command(b"dict_deltas")
            writer, reader = client.do_exchange(descriptor,
                                                options=flight.FlightCallOptions(
                                                    write_options=pa.ipc.IpcWriteOptions(emit_dictionary_deltas=True)
                                                )
                                               )
    
            received_table = reader.read_all()
            expected_table = simple_dicts_table()
            assert received_table.equals(expected_table)

    The received table did not update the dictionary (neither deltas / or dictionary replacement)

    (Pdb) received_table
    pyarrow.Table
    some_dicts: dictionary<values=string, indices=int64, ordered=0>
    ----
    some_dicts: [  -- dictionary:
    ["foo","baz","quux"]  -- indices:
    [1,0,null],  -- dictionary:
    ["foo","baz","quux"]  -- indices:
    [2,1],  -- dictionary:
    ["foo","baz","quux"]  -- indices:
    [0,3]]
    (Pdb) expected_table
    pyarrow.Table
    some_dicts: dictionary<values=string, indices=int64, ordered=0>
    ----
    some_dicts: [  -- dictionary:
    ["foo","baz","quux"]  -- indices:
    [1,0,null],  -- dictionary:
    ["foo","baz","quux"]  -- indices:
    [2,1],  -- dictionary:
    ["foo","baz","quux","new"]  -- indices:
    [0,3]]
    
  5. added this to the 22.0.0 milestone on Aug 26, 2025
  6. raulcd commented on Aug 26, 2025

    @raulcd
    Member

    Both DoGet and DoExchange have been implemented, see the sub-issues above.

  7. amoeba commented on Aug 26, 2025

    @amoeba
    MemberAuthor

    Nice work @raulcd, thanks for taking this on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions