Repository navigation
Keep the DecimalByteParts MSP at its logical integer width - #11459
Merged
Merged
Conversation
This was referenced Nov 13, 2026
This was referenced Nov 13, 2026
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
2 times, most recently
from
November 13, 2026 18:13
75fd400 to
7d3bd0b
Compare
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 13, 2026 18:14
931e0f6 to
5577bd8
Compare
connortsui20
added a commit
that referenced
this pull request
Nov 16, 2026
…#11457) ## Summary - Tracking Issue: #10701 #11459 changes what the decimal scheme writes, but the golden corpus has only one decimal entry. Without a baseline, most of those changes would show up as new snapshot files instead of changed lines. This PR records the current output first, so that each later change to the scheme is a visible diff. ## Changes The pins record what `develop` writes. They do not approve it. For example, every slot, null or not, takes part in choosing the lower-part count today, so `p38_large_null_slots` pins one lower part for values that fit in `i64`. #11459 changes that pin. The golden corpus gains three entries: nullable prices with large values in the null slots, signed values, and values above 64 bits at precision 40. `decimal_scheme_part_shapes` pins the lower-part count and the MSP type for each storage width and value range, for v1 and v2. No existing snapshot changes. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 16, 2026 17:28
5577bd8 to
6f729da
Compare
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
2 times, most recently
from
November 16, 2026 19:18
97a8de6 to
ff5c667
Compare
connortsui20
marked this pull request as ready for review
November 16, 2026 19:20
Merging this PR will degrade performance by 15.85%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 16, 2026 20:45
ff5c667 to
0cccf6d
Compare
connortsui20
changed the base branch from
develop
to
ct/dbp-bench-canonical-width
November 16, 2026 20:45
connortsui20
added a commit
that referenced
this pull request
Nov 17, 2026
## Summary - Tracking Issue: #10701 `decode` and `filter` currently stop at whatever storage width DecimalByteParts returns. That makes a decoder that returns narrow storage look faster than one that returns the canonical decimal width, even though the results have different physical layouts. ## Changes Both benchmarks now end at the canonical width, with a copy only when the decoded width differs. Names and cases stay the same so CodSpeed can compare the same workloads across the decoder change. CodSpeed is expected to report slower `decode` and `filter` results on this PR because these benchmarks now include the widening copy that callers need. #11459 can then be compared against the same endpoint. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 17, 2026 15:01
7f3f379 to
13e6aef
Compare
robert3005
approved these changes
Nov 17, 2026
connortsui20
enabled stacked auto-merge (squash)
November 17, 2026 16:22
connortsui20
disabled stacked auto-merge
November 17, 2026 16:23
connortsui20
enabled stacked auto-merge (squash)
November 17, 2026 16:23
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 17, 2026 16:28
13e6aef to
666b473
Compare
An encoding that fixes the dtype of an integer child can still store the child narrower, and hold it in memory as an integer Widen. Its reader must then wrap the stored child, and its writer must take the child out again. The two functions give these steps one owner in `vortex-array`. An encoding with a fixed integer child makes two calls and does not repeat the requirements of a Widen. Most encodings accept an integer child of any width and need neither function. The write step finds a Widen only at the root of the child. It is the one place that must change when optimizer rules move a Widen below structural encodings. DecimalByteParts is the first caller, in a later commit. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The next commit adds a device test module next to it. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The integer MSP sum can overflow even when the decimal state holds the batch total. Retry over the stored integer values in i128 instead of decoding and widening the batch. Materialize validity once and keep the state precision check and the overflow result of an i64 decimal state. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/dbp-logical-msp-width
branch
from
November 17, 2026 18:32
666b473 to
a3dc353
Compare
This was referenced Nov 17, 2026
connortsui20
added a commit
that referenced
this pull request
Nov 17, 2026
## Summary - Tracking Issue: #10701 After #11459, the GPU decoder still executes a `Widen` MSP to its logical width and uses that buffer as the decimal storage. Above precision 18 it then returns `i64` values for a decimal whose canonical width is `i128` or `i256`, which breaks the rule that #10701 needs before every `DecimalArray` can have the canonical width. The decoder now reads the child of the `Widen` and widens it once, straight to the canonical width, which also removes the copy to the logical width. ## Changes This is its own PR because the decoder test has not run yet: it needs a GPU, so it runs for the first time in GPU CI, and a decoder bug found there then does not block #11459. The test now asserts the canonical width. The four `vortex-array` doc sentences that name DecimalByteParts as a source of noncanonical storage go in this PR, because they stay true until it lands. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#10701 needs one buffer width for each decimal precision, but DecimalByteParts decoding and
narrowed_decimalin the compressor still make narrower decimal arrays ondevelop.The MSP's logical dtype now depends on the precision alone: the signed integer at the canonical width, capped at
i64. The canonical width is the smallest signed width that holds10^p - 1, so every value of a narrower signed integer is a validDecimal(p)value, and a narrow MSP is an integerWidenthat needs no precision label and no scan. The closed #10905 decoded a narrow MSP into a decimalWideninstead, which needed both, and CodSpeed reported decoding 4.7x to 8.4x slower with that design.Changes
The wire stores the child of the
Widen, not a new form, so old readers read new files and the integerWidenwire form can wait for #10733.Only files from earlier releases hold an MSP that is wider than its logical dtype. The reader gives it a checked copy on the host instead of a cast, because a cast keeps a null value that fits the width and skips its check when the child has exact min and max statistics. That MSP fails to read from device memory.
WidenArray::from_stored_integerandWidenArray::to_stored_integerlive invortex-arrayinstead of private helpers, because any encoding that fixes the dtype of an integer child needs the same two steps.Null slots no longer choose the lower-part count or the MSP type, which is why
p38_large_null_slotsgoes from one lower part to none. The GPU decoder still returnsi64storage above precision 18 until #11460. For review, read the commits one at a time, because the second one is a puregit mv.Known costs
decimal_wide_p40doubles in size, because the canonical width of precision 40 isi256.i64MSP at precision 38 gains one copy, toi128.TODO(connor)incompare.rsnames the fix.Decimal128export of a narrow MSP makes two copies instead of one. The follow-up is an Arrow export hook keyed by encoding.API Changes
DecimalByteParts::try_new,try_new_with_lower_parts, andDecimalBytePartsData::validaterequire the logical MSP dtype of the precision and at most the number of lower parts that the precision needs.split_decimaltakes the validity mask and an explicit lower-part count.narrowed_decimalis removed.Validation
On the rebased stack (
developat26a12a670b):cargo +nightly-2026-11-13 fmt --all --checkpasses.cargo clippy --all-targets --all-features -- -D warningspasses forvortex-decimal-byte-parts,vortex-btrblocks,vortex-array,vortex-file,vortex-compat,vortex-arrow, andvortex-cuda.cargo nextest run --all-featurespasses forvortex-decimal-byte-parts,vortex-btrblocks,vortex-compat, andvortex-file, and both golden tests pass withINSTA_UPDATE=no.Before the rebase (
developatd81bd4ce25), the same checks also passed forvortex-duckdbandvortex, withcargo bench -- --testfor the DBP,to_arrow,decimal_export, andcommon_encoding_tree_throughputbenches.compat.py check --mode allread every fixture of the 28 published versions, including the wide legacy MSP files of 0.91.0 to 0.93.0. Of the generated fixtures, onlydecimal_byte_parts_narrow_msp.vortexchanges.The debug assert in the v1 scheme, the
WidenArraystored integer functions, and the rebase ontodevelopatc58d6237e5came after these runs. They only passed the format check, and no local clippy or test run covers them yet.Not run: the CUDA tests (no GPU), CodSpeed, and the arm64 and Windows jobs.
cargo clippy -p vortex-btrblocks --all-targetswith default features fails on an unused import invortex-btrblocks/src/tests.rsthat is already ondevelop.