Skip to content

fix(python): read pyarrow files positionally in open_readable - #11074

Merged
robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:fix/py-arrow-readable
Nov 2, 2026
Merged

robert3005 merged 1 commit into
vortex-data:developfrom
jackylee-ch:fix/py-arrow-readable

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

vortex.io.ReadAt's docs say pyarrow files work with vx.open_readable, but none do. A pyarrow file has size() and read_at, so it is read as a ReadBytesAt, but pyarrow's read_at takes (nbytes, offset). For the new test's file, a read of 65535 bytes at offset 420089 read 420089 bytes at offset 65535, and vx.open_readable(pa.OSFile(path)) failed with RuntimeError: Python read of 420089..485624 failed: BufferError: read_at(420089, 65535) returned 420089 bytes.

open_readable now wraps a pyarrow file in a ReadBytesAt that passes the arguments in pyarrow's order. It is the only caller of the native open_readable, so readable.rs needs no pyarrow-specific mode. Reads through the wrapper are positional: an OS file or in-memory buffer serves several reads at once, with the GIL released while pyarrow reads, while a pa.PythonFile, which pyarrow returns for fsspec filesystems, still reads one at a time. Any pyarrow filesystem's open_input_file returns such a file, so this also covers HadoopFileSystem, for which Vortex has no store; it was not tested against HDFS. pyarrow-stubs types read_at without parameters, so the call carries a ty: ignore.

Tests

The new test reads a file through pa.OSFile, pa.memory_map and LocalFileSystem().open_input_file; it fails on develop. The vortex-python suite, ruff, ty and the docs build and doctests pass.

AI assistance

Written with agentic AI assistance.

@robert3005 robert3005 added the changelog/feature A new feature label Nov 2, 2026
A pyarrow file has size() and read_at(), so open_readable read it as a
ReadBytesAt, but pyarrow's read_at takes (nbytes, offset): every read
asked for the wrong range, and opening any pyarrow file failed, although
the vortex.io.ReadAt docs say pyarrow files are accepted. Wrap pyarrow
files in a ReadBytesAt that passes the arguments in pyarrow's order.

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
@robert3005
robert3005 force-pushed the fix/py-arrow-readable branch from bef9cc4 to 33fb4fc Compare November 2, 2026 11:34
@robert3005
robert3005 enabled auto-merge (squash) November 2, 2026 11:34
@codspeed

codspeed Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 11.52%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

❌ 1 regressed benchmark
✅ 2317 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime encode_chunked_nullable_avx2[i64, 524288] 134.4 µs 151.9 µs -11.52%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jackylee-ch:fix/py-arrow-readable (33fb4fc) with develop (4b1dcc3)

Open in CodSpeed

@robert3005
robert3005 merged commit 7c9bb58 into vortex-data:develop Nov 2, 2026
72 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants