Skip to content

GH-47422: [Python][C++][Flight] Expose ipc::ReadStats in Flight MetadataRecordBatchReader - #47432

Merged
raulcd merged 7 commits into
apache:mainfrom
raulcd:GH-47422
Sep 4, 2025
Merged

raulcd merged 7 commits into
apache:mainfrom
raulcd:GH-47422

Conversation

@raulcd

@raulcd raulcd commented Aug 26, 2025 •

Copy link
Copy Markdown
Member

Rationale for this change

Now that we have implemented dictionary replacement / dictionary deltas on DoGet/DoExchange, exposing ReadStatistics can help us easily understand / debug / diagnose much better what are the different messages being sent.

What changes are included in this PR?

Add API to expose arrow::ipc::ReadStats stats() on MetadataRecordBatchReader. As we are mainly using RecordBatchStreamReader everywhere now, the stats are already populated so we just have to expose them.
Add also Python bindings.

Are these changes tested?

Yes via several Python tests we are validating the stats are retrieved and the results are the expected ones.

Are there any user-facing changes?

No breaking changes, new API to retrieve Statistics from the readers.

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's one nit and one question about the accounting of metadata-only messages, otherwise LGTM.

Comment thread python/pyarrow/tests/test_flight.py Resolved
Comment thread python/pyarrow/tests/test_flight.py Resolved

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except one small suggestion

Comment thread python/pyarrow/tests/test_flight.py Resolved
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit cdea48e.

There weren't enough matching historic benchmark results to make a call on whether there were regressions.

The full Conbench report has more details.

Mottl pushed a commit to Mottl/arrow that referenced this pull request May 26, 2026
… MetadataRecordBatchReader (apache#47432)

### Rationale for this change

Now that we have implemented dictionary replacement / dictionary deltas on DoGet/DoExchange, exposing ReadStatistics can help us easily understand / debug / diagnose much better what are the different messages being sent.

### What changes are included in this PR?

Add API to expose `arrow::ipc::ReadStats stats()` on `MetadataRecordBatchReader`. As we are mainly using `RecordBatchStreamReader` everywhere now, the stats are already populated so we just have to expose them.
Add also Python bindings.

### Are these changes tested?

Yes via several Python tests we are validating the stats are retrieved and the results are the expected ones.

### Are there any user-facing changes?

No breaking changes, new API to retrieve Statistics from the readers.

* GitHub Issue: apache#47422

Authored-by: Raúl Cumplido <raulcumplido@gmail.com>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants