Repository navigation
Expose the existing footer segment end offset - #11196
Merged
joseph-isaacs merged 1 commit intoNov 4, 2026
Merged
joseph-isaacs merged 1 commit into
joseph-isaacs merged 1 commit into
Conversation
Expose Footer::segments_end without tracking another write-time offset. Document how layout segment IDs resolve zone-map locations through the footer segment map. Verify that the tail after the segments contains a readable footer with embedded or external dtype and custom metadata, and that zone-map offsets survive serialization. Signed-off-by: Frederic Branczyk <frederic.branczyk@dash0.com> Amp-Thread-ID: https://ampcode.com/threads/T-4fff387e-d227-4456-b82b-f880791e1a5e Co-authored-by: Amp <amp@ampcode.com>
Comment on lines
+190
to
+191
| /// Returns `Some(0)` when there are no segments, or `None` if any segment's end overflows | ||
| /// `u64`. For files produced by the Vortex writer, reading from this offset through the end |
Comment on lines
+190
to
+191
| /// Returns `Some(0)` when there are no segments, or `None` if any segment's end overflows | ||
| /// `u64`. For files produced by the Vortex writer, reading from this offset through the end |
Contributor
There was a problem hiding this comment.
I am not sure vortex can handle this in many other places
joseph-isaacs
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We want to keep an offset to prefetch/cache and segments_end already fits the bill really nicely. The goal is essentially that we have all important metadata eg. zone maps, indexes, etc. in low latency access available and only segments are fetched from object storage.
Changes
Expose Footer::segments_end without tracking another write-time offset. Document how layout segment IDs resolve zone-map locations through the footer segment map.
Verify that the tail after the segments contains a readable footer with embedded or external dtype and custom metadata, and that zone-map offsets survive serialization.
API Changes
One additive API.