Skip to content

fix(python): let VortexDataset.head return fewer rows than requested - #11384

Merged
robert3005 merged 3 commits into
vortex-data:developfrom
jackylee-ch:fix/python-dataset-head
Nov 16, 2026
Merged

robert3005 merged 3 commits into
vortex-data:developfrom
jackylee-ch:fix/python-dataset-head

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Summary

VortexDataset.head(n) raised RuntimeError: OutOfBounds: stop 10 > length 5 whenever fewer than n rows matched, where a pyarrow Parquet dataset returns the rows it has: head(10) on a 5-row file, or head(10, filter=pc.field("x") > 3) matching one row. ds.scanner(batch_size=N).head(k) always raised ValueError: batch_size is not supported, although to_batches takes batch_size.

Changes

head now reads through the record batch reader that to_batches uses, with batch_size as the split size and num_rows as a new scan limit. A short result is returned as is, and the scan stops once it has enough rows: on a 20M-row file, head(10, filter=pc.field("i") > 5) grows peak RSS by 20 MiB instead of 109 MiB.

SplitBy::RowCount(0), now reachable from head, panicked on step_by(0); it returns an error, as LayoutSubSplitting does for a zero max_rows. to_table still rejects batch_size, left for a follow-up.

The new tests fail on develop.

AI assistance

Written with agentic AI assistance.

head sliced the filtered array to num_rows, which fails when fewer rows
match, and rejected batch_size, so scanner(batch_size=N).head(k) always
raised. Read through the record batch reader that to_batches uses and
pass num_rows to the scan as its limit. Fail a zero row count in
SplitBy::RowCount, as LayoutSubSplitting does, instead of panicking.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Comment thread vortex-python/python/vortex/dataset.py Resolved
split_by is a low-level io/compute control, not a batch size, so head no longer maps batch_size to it and keeps rejecting batch_size. Drops the SplitBy::RowCount(0) guard, which head can no longer reach.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005 robert3005 added the changelog/fix A bug fix label Nov 16, 2026
@robert3005
robert3005 enabled auto-merge (squash) November 16, 2026 10:24
@robert3005
robert3005 merged commit 122be0e into vortex-data:develop Nov 16, 2026
116 of 118 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants