Skip to content

Write patch chunk offsets for ALP-RD arrays - #11644

Merged
0ax1 merged 1 commit into
vortex-data:developfrom
vishalkotak:vishalkotak/alp-rd-chunk-offsets
Nov 20, 2026
Merged

0ax1 merged 1 commit into
vortex-data:developfrom
vishalkotak:vishalkotak/alp-rd-chunk-offsets

Conversation

@vishalkotak

@vishalkotak vishalkotak commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Part of #11516

This is the first step toward an ALP-RD GPU kernel, as @0ax1 suggested on the issue. The CUDA patch loader needs chunk_offsets to find a chunk's patches in constant time, but the ALP-RD encoder never wrote them, and the deserializer dropped them on read. Both spots had a TODO(0ax1).

New ALP-RD files now carry one extra child holding the offsets. The compressor narrows it, so it's 16 bytes for 16k rows in the btrblocks golden snapshot. Older files without it read the same as before, and older readers simply ignore the extra child.

Could a maintainer add the changelog/feature label? I can't set labels from a fork.

I used Claude Code for analysis, debugging, and ideas.

The ALP-RD encoder now records the first patch of each 1024-row chunk,
and the deserializer reads those offsets back, so readers such as the
CUDA patch loader can find a chunk's patches without scanning the rest.
Files written before this have no offsets and still read as before.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
@vishalkotak
vishalkotak force-pushed the vishalkotak/alp-rd-chunk-offsets branch from 8fed597 to fc17253 Compare November 19, 2026 17:04
@0ax1 0ax1 added the changelog/feature A new feature label Nov 20, 2026
@0ax1
0ax1 self-requested a review November 20, 2026 10:36
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review November 20, 2026 11:54

@0ax1 0ax1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thank you for the contribution!

@0ax1
0ax1 merged commit 2f6136b into vortex-data:develop Nov 20, 2026
186 checks passed
@vishalkotak

Copy link
Copy Markdown
Contributor Author

Thanks for the quick turnaround, appreciate it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants