Skip to content

Expose the existing footer segment end offset - #11196

Merged
joseph-isaacs merged 1 commit into
vortex-data:developfrom
polarsignals:brancz/footer-segment-offsets
Nov 4, 2026
Merged

joseph-isaacs merged 1 commit into
vortex-data:developfrom
polarsignals:brancz/footer-segment-offsets

Conversation

@brancz

@brancz brancz commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Summary

We want to keep an offset to prefetch/cache and segments_end already fits the bill really nicely. The goal is essentially that we have all important metadata eg. zone maps, indexes, etc. in low latency access available and only segments are fetched from object storage.

Changes

Expose Footer::segments_end without tracking another write-time offset. Document how layout segment IDs resolve zone-map locations through the footer segment map.

Verify that the tail after the segments contains a readable footer with embedded or external dtype and custom metadata, and that zone-map offsets survive serialization.

API Changes

One additive API.

Expose Footer::segments_end without tracking another write-time offset. Document how layout segment IDs resolve zone-map locations through the footer segment map.

Verify that the tail after the segments contains a readable footer with embedded or external dtype and custom metadata, and that zone-map offsets survive serialization.

Signed-off-by: Frederic Branczyk <frederic.branczyk@dash0.com>
Amp-Thread-ID: https://ampcode.com/threads/T-4fff387e-d227-4456-b82b-f880791e1a5e
Co-authored-by: Amp <amp@ampcode.com>
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Nov 4, 2026
Comment on lines +190 to +191
/// Returns `Some(0)` when there are no segments, or `None` if any segment's end overflows
/// `u64`. For files produced by the Vortex writer, reading from this offset through the end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this happen?

Comment on lines +190 to +191
/// Returns `Some(0)` when there are no segments, or `None` if any segment's end overflows
/// `u64`. For files produced by the Vortex writer, reading from this offset through the end

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure vortex can handle this in many other places

@joseph-isaacs
joseph-isaacs merged commit a10a0e0 into vortex-data:develop Nov 4, 2026
104 of 105 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants