Skip to content

feat(io): a segment-keyed IO protocol for planner-driven scans - #10568

Draft
joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol
Draft

joseph-isaacs wants to merge 1 commit into
developfrom
ji/segment-io-protocol

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds vortex_io::segments, an IO protocol for planner-driven scans that is keyed by segment source and source request. It sits next to the byte-range vortex_io::request protocol, which the V2 executor on develop still uses. This PR is additive: it changes nothing else, so it compiles on develop by itself.

#10527 is stacked on this PR. It switches the planning traits and the V2 executor to this protocol, adds FileSegmentIo to serve it, and removes request.

Changes

  • A request names IoKey { source: SegmentSourceId, request: SourceRequest }, where SourceRequest ::= Size | Segment(id). It also carries an IoIntent: Fetch, Prefetch or Announce. One service can serve many segment sources, and a source can answer more than segment reads.
  • Reads resolve to a SourceResult: either Size(u64) or Bytes(BufferHandle). The checked accessors are size() and into_bytes().
  • SegmentIoSession::submit registers a whole batch before any read starts, and returns one IoHandle per request.
    • The session holds fetches and prefetches until they finish.
    • An announcement is withdrawn by dropping its handles.
  • SegmentIoSession::poll_ready reports each fetch's owner once the fetch finishes, including fetches that finished at submit, and drives the session's prefetches. FetchTracker implements this for sessions.
  • SegmentIoService::session() opens one session per scope. RegisteredSource is a source's registration with a service.

Tests cover handles, results checked against what was asked, and FetchTracker.

Checks run:

  • cargo clippy -p vortex-io --all-targets --all-features -- -D warnings
  • cargo +nightly-2026-09-10 fmt -p vortex-io
  • RUSTDOCFLAGS="-D warnings --cfg docsrs" cargo doc --no-deps -p vortex-io --all-features
  • cargo test -p vortex-io --lib segments (4 tests)
  • typos

API Changes

New: vortex_io::segments.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4PmceDHg8rFe5bgkhGFU9

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants