Repository navigation
Decode decimal byte parts on the GPU at the canonical width - #11460
Merged
Merged
Conversation
This was referenced Nov 13, 2026
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 13, 2026 18:12
cad5c8f to
ddb0198
Compare
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
6 times, most recently
from
November 16, 2026 19:18
fb675af to
1ea9b28
Compare
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 16, 2026 20:45
1ea9b28 to
7c949a2
Compare
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 16, 2026 21:57
7c949a2 to
2f54fc6
Compare
connortsui20
marked this pull request as ready for review
November 16, 2026 22:26
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 17, 2026 15:04
2f54fc6 to
2ad83bf
Compare
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
0ax1
approved these changes
Nov 17, 2026
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 17, 2026 16:28
2ad83bf to
beda529
Compare
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 17, 2026 18:32
beda529 to
5a54104
Compare
33 of 36 tasks
This was referenced Nov 17, 2026
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
added a commit
that referenced
this pull request
Nov 17, 2026
## Summary - Tracking Issue: #10701 - Replaces #11354 #10701 needs one buffer width for each decimal precision, but DecimalByteParts decoding and `narrowed_decimal` in the compressor still make narrower decimal arrays on `develop`. The MSP's logical dtype now depends on the precision alone: the signed integer at the canonical width, capped at `i64`. The canonical width is the smallest signed width that holds `10^p - 1`, so every value of a narrower signed integer is a valid `Decimal(p)` value, and a narrow MSP is an integer `Widen` that needs no precision label and no scan. The closed #10905 decoded a narrow MSP into a decimal `Widen` instead, which needed both, and CodSpeed reported decoding 4.7x to 8.4x slower with that design. ## Changes The wire stores the child of the `Widen`, not a new form, so old readers read new files and the integer `Widen` wire form can wait for #10733. Only files from earlier releases hold an MSP that is wider than its logical dtype. The reader gives it a checked copy on the host instead of a cast, because a cast keeps a null value that fits the width and skips its check when the child has exact min and max statistics. That MSP fails to read from device memory. `WidenArray::from_stored_integer` and `WidenArray::to_stored_integer` live in `vortex-array` instead of private helpers, because any encoding that fixes the dtype of an integer child needs the same two steps. Null slots no longer choose the lower-part count or the MSP type, which is why `p38_large_null_slots` goes from one lower part to none. The GPU decoder still returns `i64` storage above precision 18 until #11460. For review, read the commits one at a time, because the second one is a pure `git mv`. ### Known costs - `decimal_wide_p40` doubles in size, because the canonical width of precision 40 is `i256`. - Decoding an `i64` MSP at precision 38 gains one copy, to `i128`. - With lower parts, compare loses its whole-answer shortcut when the constant's MSP word is outside the range of the narrow child. A `TODO(connor)` in `compare.rs` names the fix. - At precision 18 or below, the Arrow `Decimal128` export of a narrow MSP makes two copies instead of one. The follow-up is an Arrow export hook keyed by encoding. ## API Changes - `DecimalByteParts::try_new`, `try_new_with_lower_parts`, and `DecimalBytePartsData::validate` require the logical MSP dtype of the precision and at most the number of lower parts that the precision needs. - `split_decimal` takes the validity mask and an explicit lower-part count. - `narrowed_decimal` is removed. - A file whose wider MSP holds a valid value outside the precision no longer reads. <details> <summary>Validation</summary> On the rebased stack (`develop` at `26a12a670b`): - `cargo +nightly-2026-11-13 fmt --all --check` passes. - `cargo clippy --all-targets --all-features -- -D warnings` passes for `vortex-decimal-byte-parts`, `vortex-btrblocks`, `vortex-array`, `vortex-file`, `vortex-compat`, `vortex-arrow`, and `vortex-cuda`. - `cargo nextest run --all-features` passes for `vortex-decimal-byte-parts`, `vortex-btrblocks`, `vortex-compat`, and `vortex-file`, and both golden tests pass with `INSTA_UPDATE=no`. Before the rebase (`develop` at `d81bd4ce25`), the same checks also passed for `vortex-duckdb` and `vortex`, with `cargo bench -- --test` for the DBP, `to_arrow`, `decimal_export`, and `common_encoding_tree_throughput` benches. `compat.py check --mode all` read every fixture of the 28 published versions, including the wide legacy MSP files of 0.91.0 to 0.93.0. Of the generated fixtures, only `decimal_byte_parts_narrow_msp.vortex` changes. The debug assert in the v1 scheme, the `WidenArray` stored integer functions, and the rebase onto `develop` at `c58d6237e5` came after these runs. They only passed the format check, and no local clippy or test run covers them yet. Not run: the CUDA tests (no GPU), CodSpeed, and the arm64 and Windows jobs. `cargo clippy -p vortex-btrblocks --all-targets` with default features fails on an unused import in `vortex-btrblocks/src/tests.rs` that is already on `develop`. </details> --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/dbp-gpu-canonical-width
branch
from
November 17, 2026 20:15
5a54104 to
63fa0c7
Compare
15 of 28 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After #11459, the GPU decoder still executes a
WidenMSP to its logical width and uses that buffer as the decimal storage. Above precision 18 it then returnsi64values for a decimal whose canonical width isi128ori256, which breaks the rule that #10701 needs before everyDecimalArraycan have the canonical width. The decoder now reads the child of theWidenand widens it once, straight to the canonical width, which also removes the copy to the logical width.Changes
This is its own PR because the decoder test has not run yet: it needs a GPU, so it runs for the first time in GPU CI, and a decoder bug found there then does not block #11459. The test now asserts the canonical width. The four
vortex-arraydoc sentences that name DecimalByteParts as a source of noncanonical storage go in this PR, because they stay true until it lands.