Skip to content

Support filters with limit in ScanBuilder - #8766

Merged
robert3005 merged 11 commits into
developfrom
adamg/remove-map-fn-limit
Oct 11, 2026
Merged

robert3005 merged 11 commits into
developfrom
adamg/remove-map-fn-limit

Conversation

@AdamGS

@AdamGS AdamGS commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Reviving #7565. Scans previously rejected combining a filter with a limit. This change lets callers push the limit into Vortex and stop after enough matching rows have been produced.

What changes are included in this PR?

  • Apply limits after filtering and row selection, preserving split order for ordered scans.
  • Share a row budget across unordered partitions. Ordered partitions apply local limits, with the caller enforcing the final total.
  • Push filtered limits through the Python and DataFusion integrations, including exact fetch enforcement in the v2 DataFusion source.
  • Keep ScanBuilder::map and the generic result types, applying the mapper to trimmed arrays.

What APIs are changed? Are there any user-facing changes?

  • ScanBuilder::with_limit now supports scans with filters.
  • ScanBuilder::map and ScanBuilder::build remain available.
  • Limited scans must use a stream or iterator; build and RepeatedScan::execute reject limits because independently scheduled split futures cannot coordinate them.
  • A prepared scan receives a fresh row budget on each execution unless it uses an internal shared partition budget.

Validation

Regression coverage includes filtered limits, ordering, shared budgets, sparse selections, error termination, and task construction. Tests and formatting were not run during this review revision, following repository guidance.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants