Skip to content

fix(python): VortexDataset.take accepts indices in any order - #11410

Merged
robert3005 merged 2 commits into
vortex-data:developfrom
jackylee-ch:fix/python-dataset-take-order
Nov 16, 2026
Merged

robert3005 merged 2 commits into
vortex-data:developfrom
jackylee-ch:fix/python-dataset-take-order

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

VortexDataset.take failed with "buffer values must be strictly increasing" for take([7, 2]) and take([7, 2, 7]), while pyarrow's Dataset.take returns the rows in the requested order.

Changes

  • When the indices are not strictly increasing and no row filter applies, scan the sorted distinct rows once, then gather them back into the requested order with Vortex Array.take. Sorted input keeps the single-scan path.
  • With a filter (the filter argument or Dataset.filter) the surviving rows cannot be mapped back to positions, so strictly increasing indices are still required. The docstring says so.
  • Indices outside a fragment's row range are still dropped.
  • Tests cover unordered, repeated, fragment and filtered-dataset takes.

AI assistance

Written with agentic AI assistance.

The scan layer requires strictly increasing row indices, so take([7, 2])
and take([7, 2, 7]) failed with "buffer values must be strictly
increasing" while pyarrow's Dataset.take returns the rows in the order
requested.

When the indices are not strictly increasing and no row filter applies,
scan the sorted distinct rows once and gather them back into the
requested order. Sorted input keeps the existing single-scan path. With
a filter, from the argument or from Dataset.filter, the surviving rows
cannot be mapped back to positions, so the indices must still be
strictly increasing; the docstring says so.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 12, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 12, 2026 12:36
Annotate the take helpers with pyarrow.Array[Any] and suppress two pyarrow-stubs overload mismatches so ty 0.0.98 passes.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
auto-merge was automatically disabled November 12, 2026 13:12

Head branch was pushed to by a user without write access

@robert3005
robert3005 enabled auto-merge (squash) November 12, 2026 13:35
@robert3005
robert3005 merged commit c9076aa into vortex-data:develop Nov 16, 2026
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants