Skip to content

feat(query): add query_arrow for Arrow results in one call - #238

Merged
eddietejeda merged 2 commits into
mainfrom
feat/query-arrow
Nov 19, 2026
Merged

eddietejeda merged 2 commits into
mainfrom
feat/query-arrow

Conversation

@eddietejeda

@eddietejeda eddietejeda commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What

Adds QueryApi.query_arrow, which runs a query and returns the rows as a pyarrow.Table in one request.

POST /v1/query already answers with an Arrow IPC stream when asked, and sends the values the JSON body would carry as response headers. The generated client cannot decode that body or surface the headers, so until now Arrow meant fetching the stored result in a second call.

Behaviour

  • Returns ArrowQueryResult: table, query_run_id, result_id, total_row_count, execution_time_ms, warning, rollup (RollupUsage), followed.
  • dialect and the other QueryRequest fields work as in query.
  • HTTP 429 is retried with the same RetryPolicy as query.
  • A result too large to send inline (202 with X-Result-Id) is read from the Results API, under the max_auto_rows guard. With auto_follow=False it raises the new ResultNotInlineError, which carries the result_id.
  • Async submission is refused with ValueError.
  • Other errors (including the 408 query timeout) raise ApiException with the status, as in query.

Tests

  • Unit tests in tests/test_query_arrow.py (stubbed transport, real Arrow bytes).
  • Integration test in tests/integration/test_query_arrow.py. Passes against a live workspace.

POST /v1/query can answer with an Arrow IPC stream and carries the run id,
result id, row count, timing, warning and rollup outcome as response
headers. The generated client cannot decode that body or surface the
headers, so add QueryApi.query_arrow, which returns an ArrowQueryResult.

429 is retried as in query(). A result too large to send inline (202 with
X-Result-Id) is read from the Results API, under the max_auto_rows guard.
Comment thread hotdata/query.py
total if total is not None else "unknown",
query_run_id,
)
if max_auto_rows is not None and total is not None and total > max_auto_rows:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Fall back to the X-Total-Row-Count header, or fail closed, when get_query_run fails. (not blocking)

The except ApiException at the top of _follow_arrow sets run = None. total is then None, and this guard is skipped. get_result_arrow then pulls the full result into memory with no max_auto_rows check. In _materialize_full, _authoritative_total falls back to the preview's total_row_count, so query keeps the guard in this case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2929b46: the follow now reads the result batch by batch and counts rows as it goes, so max_auto_rows holds even when the run lookup fails. Added a test for that case.

Comment thread hotdata/query.py Outdated
# names the saved result in X-Result-Id instead of sending partial rows.
ticket, result_id = outcome
if result_id is None:
raise ResultUnavailableError(warning=ticket.reason)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Raise an error with a message that fits query_arrow here. (not blocking)

ResultUnavailableError says "Query result is truncated … Re-run with auto_follow=False to use the preview." query_arrow has no preview. This branch also runs before the follow check, so auto_follow=False raises the same error. A caller who follows the advice gets the same exception again.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2929b46: this case now gets its own message, which names the query run and no longer mentions a preview or auto_follow.

claude[bot]
claude Bot previously approved these changes Nov 19, 2026
Count rows while reading the followed result, so max_auto_rows holds when
the query-run lookup fails. Give the no-result-id case a message that fits
query_arrow.
@eddietejeda
eddietejeda marked this pull request as ready for review November 19, 2026 19:10
@eddietejeda
eddietejeda requested a review from a team as a code owner November 19, 2026 19:10
@eddietejeda
eddietejeda requested review from rohan-hotdata and removed request for a team November 19, 2026 19:10
Comment thread hotdata/query.py

batches = []
fetched = 0
with _ArrowResultsApi(self.api_client).stream_result_arrow(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Pass limit=max_auto_rows + 1 to stream_result_arrow when max_auto_rows is not None. (not blocking)

The row guard caps memory but not transfer. When the guard raises inside the with block, stream_result_arrow exits through _release_conn, and _release_conn calls drain_conn (hotdata/arrow.py:64). drain_conn reads and discards the rest of the body, so the client still downloads the full oversized result. With the limit, the server sends at most one row past the guard, and the existing fetched > max_auto_rows check still detects the overflow.

@eddietejeda
eddietejeda merged commit 48a68e3 into main Nov 19, 2026
5 checks passed
@eddietejeda
eddietejeda deleted the feat/query-arrow branch November 19, 2026 19:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant