Skip to content

Keep the DecimalByteParts MSP at its logical integer width - #11459

Merged
connortsui20 merged 4 commits into
developfrom
ct/dbp-logical-msp-width
Nov 17, 2026
Merged

connortsui20 merged 4 commits into
developfrom
ct/dbp-logical-msp-width

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

#10701 needs one buffer width for each decimal precision, but DecimalByteParts decoding and narrowed_decimal in the compressor still make narrower decimal arrays on develop.

The MSP's logical dtype now depends on the precision alone: the signed integer at the canonical width, capped at i64. The canonical width is the smallest signed width that holds 10^p - 1, so every value of a narrower signed integer is a valid Decimal(p) value, and a narrow MSP is an integer Widen that needs no precision label and no scan. The closed #10905 decoded a narrow MSP into a decimal Widen instead, which needed both, and CodSpeed reported decoding 4.7x to 8.4x slower with that design.

Changes

The wire stores the child of the Widen, not a new form, so old readers read new files and the integer Widen wire form can wait for #10733.

Only files from earlier releases hold an MSP that is wider than its logical dtype. The reader gives it a checked copy on the host instead of a cast, because a cast keeps a null value that fits the width and skips its check when the child has exact min and max statistics. That MSP fails to read from device memory.

WidenArray::from_stored_integer and WidenArray::to_stored_integer live in vortex-array instead of private helpers, because any encoding that fixes the dtype of an integer child needs the same two steps.

Null slots no longer choose the lower-part count or the MSP type, which is why p38_large_null_slots goes from one lower part to none. The GPU decoder still returns i64 storage above precision 18 until #11460. For review, read the commits one at a time, because the second one is a pure git mv.

Known costs

  • decimal_wide_p40 doubles in size, because the canonical width of precision 40 is i256.
  • Decoding an i64 MSP at precision 38 gains one copy, to i128.
  • With lower parts, compare loses its whole-answer shortcut when the constant's MSP word is outside the range of the narrow child. A TODO(connor) in compare.rs names the fix.
  • At precision 18 or below, the Arrow Decimal128 export of a narrow MSP makes two copies instead of one. The follow-up is an Arrow export hook keyed by encoding.

API Changes

  • DecimalByteParts::try_new, try_new_with_lower_parts, and DecimalBytePartsData::validate require the logical MSP dtype of the precision and at most the number of lower parts that the precision needs.
  • split_decimal takes the validity mask and an explicit lower-part count.
  • narrowed_decimal is removed.
  • A file whose wider MSP holds a valid value outside the precision no longer reads.
Validation

On the rebased stack (develop at 26a12a670b):

  • cargo +nightly-2026-11-13 fmt --all --check passes.
  • cargo clippy --all-targets --all-features -- -D warnings passes for vortex-decimal-byte-parts, vortex-btrblocks, vortex-array, vortex-file, vortex-compat, vortex-arrow, and vortex-cuda.
  • cargo nextest run --all-features passes for vortex-decimal-byte-parts, vortex-btrblocks, vortex-compat, and vortex-file, and both golden tests pass with INSTA_UPDATE=no.

Before the rebase (develop at d81bd4ce25), the same checks also passed for vortex-duckdb and vortex, with cargo bench -- --test for the DBP, to_arrow, decimal_export, and common_encoding_tree_throughput benches. compat.py check --mode all read every fixture of the 28 published versions, including the wide legacy MSP files of 0.91.0 to 0.93.0. Of the generated fixtures, only decimal_byte_parts_narrow_msp.vortex changes.

The debug assert in the v1 scheme, the WidenArray stored integer functions, and the rebase onto develop at c58d6237e5 came after these runs. They only passed the format check, and no local clippy or test run covers them yet.

Not run: the CUDA tests (no GPU), CodSpeed, and the arm64 and Windows jobs. cargo clippy -p vortex-btrblocks --all-targets with default features fails on an unused import in vortex-btrblocks/src/tests.rs that is already on develop.

@connortsui20 connortsui20 changed the title ct/dbp logical msp width Keep the DecimalByteParts MSP at its logical integer width Nov 13, 2026
@connortsui20 connortsui20 added the changelog/break A breaking API change label Nov 13, 2026
connortsui20 added a commit that referenced this pull request Nov 13, 2026
## Summary

- Tracking Issue: #10701

#11458 and #11459 change both the code and the tests of `array.rs`,
`assemble.rs`, and `split.rs`. This PR is a purely cosmetic change.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch 2 times, most recently from 75fd400 to 7d3bd0b Compare November 13, 2026 18:13
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from 931e0f6 to 5577bd8 Compare November 13, 2026 18:14
connortsui20 added a commit that referenced this pull request Nov 16, 2026
…#11457)

## Summary

- Tracking Issue: #10701

#11459 changes what the decimal scheme writes, but the golden corpus has
only one decimal entry. Without a baseline, most of those changes would
show up as new snapshot files instead of changed lines. This PR records
the current output first, so that each later change to the scheme is a
visible diff.

## Changes

The pins record what `develop` writes. They do not approve it. For
example, every slot, null or not, takes part in choosing the lower-part
count today, so `p38_large_null_slots` pins one lower part for values
that fit in `i64`. #11459 changes that pin.

The golden corpus gains three entries: nullable prices with large values
in the null slots, signed values, and values above 64 bits at precision
40. `decimal_scheme_part_shapes` pins the lower-part count and the MSP
type for each storage width and value range, for v1 and v2. No existing
snapshot changes.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from 5577bd8 to 6f729da Compare November 16, 2026 17:28
Base automatically changed from ct/dbp-u64-lower-parts to develop November 16, 2026 17:44
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch 2 times, most recently from 97a8de6 to ff5c667 Compare November 16, 2026 19:18
@connortsui20
connortsui20 marked this pull request as ready for review November 16, 2026 19:20
@codspeed

codspeed Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 15.85%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 2 improved benchmarks
❌ 12 regressed benchmarks
✅ 1961 untouched benchmarks
⏩ 53 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation decimal_export[gap_msp_i16_p10] 2.2 ms 3.2 ms -30.21%
❌ CPU Simulation decimal_export[msp_i32_p18] 2.4 ms 3.3 ms -28.68%
❌ CPU Simulation decimal_export[msp_i32_p18_nullable] 2.4 ms 3.4 ms -28.46%
❌ CPU Simulation compare_constant_out_of_range[gap_msp_i16_p10] 39.4 µs 51.1 µs -22.78%
❌ CPU Simulation decompress[dbp_bp_i32] 61.1 µs 76.6 µs -20.15%
❌ CPU Simulation min_max[gap_msp_i16_p10] 85 µs 102.9 µs -17.43%
❌ CPU Simulation min_max[msp_i32_p18] 101.9 µs 122.3 µs -16.68%
❌ CPU Simulation sum[gap_msp_i16_p10] 58 µs 69 µs -15.98%
❌ CPU Simulation compare_constant_out_of_range[msp_i32_p18] 61.5 µs 72.7 µs -15.37%
❌ CPU Simulation sum[msp_i32_p18] 72.5 µs 83.8 µs -13.48%
❌ CPU Simulation null_count_run_end[(32000, 256, 0.01)] 7.7 µs 8.7 µs -11.72%
❌ CPU Simulation compare_constant[gap_msp_i16_p10] 75.2 µs 85.1 µs -11.65%
⚡ Walltime add_constant_shapes_neon[(32768, ConstantPerRow)] 15.9 µs 14.2 µs +11.9%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowConstant)] 15.7 µs 14.3 µs +10.09%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/dbp-logical-msp-width (a3dc353) with develop (5e2e8b6)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from ff5c667 to 0cccf6d Compare November 16, 2026 20:45
@connortsui20
connortsui20 removed this pull request from stack #11461 November 16, 2026 20:45
@connortsui20
connortsui20 changed the base branch from develop to ct/dbp-bench-canonical-width November 16, 2026 20:45
Base automatically changed from ct/dbp-bench-canonical-width to develop November 17, 2026 14:52
connortsui20 added a commit that referenced this pull request Nov 17, 2026
## Summary

- Tracking Issue: #10701

`decode` and `filter` currently stop at whatever storage width
DecimalByteParts returns. That makes a decoder that returns narrow
storage look faster than one that returns the canonical decimal width,
even though the results have different physical layouts.

## Changes

Both benchmarks now end at the canonical width, with a copy only when
the decoded width differs. Names and cases stay the same so CodSpeed can
compare the same workloads across the decoder change.

CodSpeed is expected to report slower `decode` and `filter` results on
this PR because these benchmarks now include the widening copy that
callers need. #11459 can then be compared against the same endpoint.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from 7f3f379 to 13e6aef Compare November 17, 2026 15:01
@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 16:22
@connortsui20
connortsui20 disabled stacked auto-merge November 17, 2026 16:23
@connortsui20
connortsui20 enabled stacked auto-merge (squash) November 17, 2026 16:23
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from 13e6aef to 666b473 Compare November 17, 2026 16:28
An encoding that fixes the dtype of an integer child can still store
the child narrower, and hold it in memory as an integer Widen. Its
reader must then wrap the stored child, and its writer must take the
child out again.

The two functions give these steps one owner in `vortex-array`. An
encoding with a fixed integer child makes two calls and does not repeat
the requirements of a Widen. Most encodings accept an integer child of
any width and need neither function.

The write step finds a Widen only at the root of the child. It is the
one place that must change when optimizer rules move a Widen below
structural encodings.

DecimalByteParts is the first caller, in a later commit.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The next commit adds a device test module next to it.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The integer MSP sum can overflow even when the decimal state holds the batch total. Retry over the stored integer values in i128 instead of decoding and widening the batch. Materialize validity once and keep the state precision check and the overflow result of an i64 decimal state.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/dbp-logical-msp-width branch from 666b473 to a3dc353 Compare November 17, 2026 18:32
@connortsui20
connortsui20 merged commit 51cd7d0 into develop Nov 17, 2026
81 of 84 checks passed
@connortsui20
connortsui20 deleted the ct/dbp-logical-msp-width branch November 17, 2026 20:15
connortsui20 added a commit that referenced this pull request Nov 17, 2026
## Summary

- Tracking Issue: #10701

After #11459, the GPU decoder still executes a `Widen` MSP to its
logical width and uses that buffer as the decimal storage. Above
precision 18 it then returns `i64` values for a decimal whose canonical
width is `i128` or `i256`, which breaks the rule that #10701 needs
before every `DecimalArray` can have the canonical width. The decoder
now reads the child of the `Widen` and widens it once, straight to the
canonical width, which also removes the copy to the logical width.

## Changes

This is its own PR because the decoder test has not run yet: it needs a
GPU, so it runs for the first time in GPU CI, and a decoder bug found
there then does not block #11459. The test now asserts the canonical
width. The four `vortex-array` doc sentences that name DecimalByteParts
as a source of noncanonical storage go in this PR, because they stay
true until it lands.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants