Skip to content

[C++][FlightRPC] Support non-grpc data planes #30773

Closed

Description

@asfimport

The goal is to decouple FlightData payload transmission from gRPC and provide a convenient framework to support other accelerated data planes (ucx, dpdk, xdp, etc.).

As a PoC, I implemented a shared memory data plane for local rpc, the performance is great compared with current best (grpc over unix socket).

Tested throughput and latency on xeon gold 5218, DoGet, batch size 128K.

|streams|grpc over unix socket|shared memory data plane|
|
|-|-|-|-|
|1|3324 MB/s, 35 us|7045 MB/s, 16 us|
|
|2|6289 MB/s, 38 us|13311 MB/s, 17 us|
|
|4|10037 MB/s, 44 us|25012 MB/s, 17 us|

Reporter: Yibo Cai / @cyb70289
Assignee: David Li / @lidavidm

Related issues:

PRs and other links:

Note: This issue was originally created as ARROW-15282. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Feb 17, 2022

    @asfimport
    Author

    Yibo Cai / @cyb70289:
    A better solution at ARROW-15706.

  2. asfimport commented on Feb 18, 2022

    @asfimport
    Author

    David Li / @lidavidm:
    If you don't mind I'm going to recycle this JIRA since I'd like to split the gRPC refactoring in ARROW-15706 from the UCX implementation itself

  3. asfimport commented on Feb 19, 2022

    @asfimport
    Author

    Yibo Cai / @cyb70289:
    Sure, no problem.

  4. asfimport commented on Mar 11, 2022

    @asfimport
    Author

    Yibo Cai / @cyb70289:
    Issue resolved by pull request 12465
    #12465

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions