Skip to content

[Flight] Add received bytes to ipc::ReadStats #47171

Closed as not planned
Closed as not planned

Description

@rustyconover

Describe the enhancement requested

Currently there isn't a method to return the read stats from the FlightStreamReader.

Component(s)

FlightRPC

Activity

  1. raulcd commented on Aug 19, 2025

    @raulcd
    Member

    Hi @rustyconover , a similar discussion was raised on the PR linked above about exposing the arrow::ipc::ReadStats to the FligthStreamReader with the changes we implemented to have dictionary replacement and dictionary deltas for Flight DoGet (DoExchange on the works). Are those the stats you were looking for, or were you referring to other type of stats?

    struct ReadStats {
    /// Number of IPC messages read.
    int64_t num_messages = 0;
    /// Number of record batches read.
    int64_t num_record_batches = 0;
    /// Number of dictionary batches read.
    ///
    /// Note: num_dictionary_batches >= num_dictionary_deltas + num_replaced_dictionaries
    int64_t num_dictionary_batches = 0;
    /// Number of dictionary deltas read.
    int64_t num_dictionary_deltas = 0;
    /// Number of replaced dictionaries (i.e. where a dictionary batch replaces
    /// an existing dictionary with an unrelated new dictionary).
    int64_t num_replaced_dictionaries = 0;
    };

  2. rustyconover commented on Aug 21, 2025

    @rustyconover
    CollaboratorAuthor

    Hi @raulcd,

    I'm mostly interested in measures of bytes received. That way I can surface it in the EXPLAIN ANALYZE output of Airport in DuckDB to show users how much data they read from the Flight server.

    Rusty

  3. raulcd commented on Aug 25, 2025

    @raulcd
    Member

    Thanks @rustyconover , as a first step I've created:

    To expose the existing ReadStats in FlightStreamReader.

    We can probably rename this issue to explicitly add bytes received to ReadStats ?

  4. raulcd commented on Sep 4, 2025

    @raulcd
    Member

    Exposing the existing ipc::ReadStats in the FlightStreamReader PR has been merged. We can update the title for this issue to specifically request for the total size in bytes of the data read to be added to ipc::ReadStats. It would be inline with the ipc::WriteStats were we already have:

    /// Total size in bytes of record batches emitted.
    /// The "raw" size counts the original buffer sizes, while the "serialized" size
    /// includes padding and (optionally) compression.
    int64_t total_raw_body_size = 0;
    int64_t total_serialized_body_size = 0;

  5. changed the title [-][Flight] Return read stats from FlightStreamReader[/-] [+][Flight] Add received bytes to ipc::ReadStats[/+] on Sep 27, 2025
  6. rustyconover commented on Sep 27, 2025

    @rustyconover
    CollaboratorAuthor

    @raulcd thanks for making this go forward. I'll be looking to try it out shortly and match catch up with you in Paris.

  7. github-actions commented on Sep 28, 2026

    @github-actions

    This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

  8. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions