Repository navigation
Conversation
a919c91 to
a11813a
Compare
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.975x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.998x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.010x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.011x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.012x ➖, 0↑ 1↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.033x ➖, 2↑ 3↓)
datafusion / parquet / ns (1.033x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.021x ➖, 0↑ 2↓)
duckdb / parquet / ns (0.928x ➖, 1↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.015x ➖, 0↑ 1↓)
duckdb / parquet / ns (1.011x ➖, 0↑ 0↓)
No file size changes detected. |
Moves the segment types and readers into a new `vortex_io::segments` module: `SegmentId`, `SegmentSource` and `SegmentLocation` from vortex-scan, the coalescing `FileSegmentSource` with its read driver, `BufferSegmentSource` and `RequestMetrics` from vortex-file, and `SharedSegmentSource` from vortex-layout. The I/O layer can then read a file's segments itself, which #11623 needs for its session-wide scheduler, without depending on vortex-file or vortex-layout. Every item stays re-exported from its old path, and behaviour is unchanged. The only edits beyond imports: `FileSegmentSource` and `BufferSegmentSource` take `SegmentLocation` instead of vortex-file's `SegmentSpec` (same fields), so `Footer::segment_locations()` now does the conversion that `VortexFile` did; `SegmentLocation::target()` becomes a private free function in vortex-scan because the type now lives in another crate; and two casts and a test `cfg` change to satisfy vortex-io's stricter lints. Written with Claude Code. The vortex-io, vortex-scan, vortex-layout and vortex-file tests and clippy pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Scan I/O requests carry a SourceId and a segment target instead of a byte range, and one IoScheduler per session serves them. Placed sources read through the coalescing FileSegmentSource, serve the segments the footer read already fetched, share in-flight reads through SharedSegmentSource, and reuse a live registration across concurrent scans of one file; opaque sources read segments themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
a11813a to
2a14cbb
Compare
Merging this PR will improve performance by 14.56%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Every request names a segment of a registered source and which bytes of it, and every result is those bytes, so IoTarget is a plain struct and completions carry a BufferHandle. A source's length is read when opening it, not through the scheduler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Nicholas Gates <nick@nickgates.com>
Pull Request is not mergeable
Scan reads now name a source and a segment instead of a byte range, and one
IoSchedulerperVortexSessionserves them. Byte-range requests forced every I/O service to know file offsets and alignment, which shuts out stores whose segments have no offsets, such as a database table or a key/value store, and gave each scan its own reader, read sharing and retention. AnIoRequestnow carries aSourceIdand anIoTarget::Segment { segment, bytes }, wherebytesis the whole segment or a list of sorted, disjoint ranges delivered packed into one buffer at the segment's alignment. This reverses the earlier decision to keepIoTargeta closed byte-range type, and overlaps the scheduler work onrk/scan-scheduler.A source registers once with the session's scheduler as either placed (a
VortexReadAtplus its segment locations, which the scheduler reads through the coalescingFileSegmentSource) or opaque (anySegmentSource). A placed file starts from the segments its footer read already fetched, and concurrent scans of one file share its registration, so they share one driver, its in-flight reads (throughSharedSegmentSource) and its retained segments under the existing 64 MiB default;ScanCacheholds the registration, so repeated scans reuse oneSourceId.ScanInputandScanFilereplacesegments: Arc<dyn SegmentSource>withreader: SegmentReader, which is the public API change;ReadAtIoSourceis gone.Left for follow-ups: the scan adaptor stamps one source per plan, so plans over several sources are expressible in the protocol but not yet delivered;
Rangesreads the whole segment and slices it; a file opened with a userSegmentCacheorwith_usageregisters as opaque so those wrappers still see its reads; the concurrency cap and the 64 MiB retention budget stay per registered source, as per file before, with a session-wide budget and prioritisation tracked in #11624; andio_scheduler()returns the concreteIoScheduler, so a replacementIoServicecannot yet resolve aSourceId.Stacked on #11625, which moves those readers and the segment types into
vortex_io::segmentswithout changing behaviour; review that first.On TPC-H SF1 with DataFusion's V2 scan (
--scan-v1 false) over local NVMe, three alternating rounds against the base commit give a geometric-mean time ratio of 0.996, within the 1% that base runs differ from each other, with similar read counts and bytes. Remote object stores and DuckDB were not benchmarked. Placed files no longer report thevortex.file.segments.cache.*metrics, since their V2 reads skip the segment-cache adapter.Written with Claude Code. The vortex-io, vortex-scan, vortex-layout and vortex-file tests pass locally, including one that scans a placed file end to end and checks it reads nothing the footer read already fetched; DataFusion was compiled and benchmarked but its tests were not run, and DuckDB, CUDA and the bindings were not built locally.
🤖 Generated with Claude Code