Skip to content

feat(array): migrate structural encodings to scalar probes - #10860

Open
jiengup wants to merge 5 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/structural
Open

jiengup wants to merge 5 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/structural

Conversation

@jiengup

@jiengup jiengup commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

OperationsVTable::scalar_at cannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads.

This PR migrates 10 implementations: Filter, Interleave, ScalarFnValidity, Union, Variant, FixedSizeList, List, ListView, VarBin, DecimalByteParts.

  • These implementations use ProbeState = (); validity and child probes are retained through stable slots.
  • List, ListView, and FixedSizeList gather only the selected row's elements.
  • VarBin reads host-resident Primitive offsets directly and uses child probes for other offset encodings.
  • DecimalByteParts takes validity from the high part and reconstructs lower parts in a bounded stack buffer.
  • Each migrated scalar_at remains a one-off probe compatibility shim.

Verification

The following checks passed locally before rebasing onto 20cc6e2aef.

  • cargo nextest run --cargo-profile ci --locked -p vortex-array -p vortex-decimal-byte-parts --all-features --no-fail-fast — 5,001 passed, 1 skipped.
  • cargo test --profile ci --locked -p vortex-array -p vortex-decimal-byte-parts --all-features --doc --no-fail-fast — 84 passed, 21 ignored.
  • cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-decimal-byte-parts --check — passed.
  • cargo clippy --profile ci --locked -p vortex-array -p vortex-decimal-byte-parts --all-targets --all-features -- -D warnings — passed.
  • Focused cargo bench runs — all 2 benchmark targets completed successfully.

Comparison to baseline

Measured on a development machine (macOS with Apple Silicon); not representative of production performance.

Historical comparison of d270200d65 with the cumulative migration tree cf5a9da2aa, using Rust 1.99.1 and the same benchmark sources, manifests, and lockfile. These are cumulative-branch measurements, not a comparison against this PR's current base.

cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15. Values are Divan medians in microseconds per iteration, with one retained probe per iteration.

Array Fixture Reads Baseline (µs) Migrated (µs) Change
Filter nullable input, random 256 65.040 55.490 -14.7%
Interleave two nullable inputs, random 256 34.240 19.450 -43.2%
ScalarFnValidity nullable binary function, random 256 31.830 24.700 -22.4%
Union two variants, random 256 23.490 20.040 -14.7%
Variant constant core storage, random 256 7.916 8.458 +6.8%
FixedSizeList nullable three-element lists, random 256 105.100 35.160 -66.5%
List nullable three-element lists, random 256 108.800 45.580 -58.1%
ListView nullable three-element views, random 256 115.100 48.490 -57.9%
VarBin nullable Primitive offsets, random 256 14.290 9.499 -33.5%
DecimalByteParts three lower parts, random 256 26.700 26.200 -1.9%

The Variant retained-read fixture regressed by 6.8%; its one-off reads improved from 7.332 to 6.541 µs. The VarBin result includes the direct Primitive-offset path.

AI Usage

GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex.

@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

3 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

⚡ 9 improved benchmarks
❌ 1 regressed benchmark
✅ 2419 untouched benchmarks
🆕 24 new benchmarks
⏩ 126 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation missing_stats[16] 70.4 µs 88 µs -19.92%
⚡ Walltime filter_neon[8] 1,801 ns 882 ns ×2
⚡ Walltime threshold_neon[8] 2.7 µs 1.3 µs ×2
⚡ Walltime mul_u64_nonnull_neon 39.5 µs 20.2 µs +95.94%
⚡ Walltime threshold_neon[64] 3.7 µs 2.6 µs +45.37%
⚡ Walltime filter_neon[64] 2.5 µs 1.7 µs +44.53%
⚡ Walltime threshold_neon[80] 4.5 µs 3.3 µs +35.53%
⚡ Walltime filter_neon[80] 3.2 µs 2.4 µs +30.64%
⚡ CPU Simulation take_filter_primitive_small_uncached_random_mask_random_indices[4096, 10] 23.4 µs 20.6 µs +13.49%
⚡ CPU Simulation random_i8[0.01] 114.8 µs 102.1 µs +12.52%
🆕 CPU Simulation one_off[0] N/A 307.3 µs N/A
🆕 CPU Simulation one_off[1] N/A 404.8 µs N/A
🆕 CPU Simulation one_off[3] N/A 600.2 µs N/A
🆕 CPU Simulation repeated[0] N/A 298.2 µs N/A
🆕 CPU Simulation repeated[1] N/A 407.9 µs N/A
🆕 CPU Simulation repeated[3] N/A 627.9 µs N/A
🆕 CPU Simulation once[interleave] N/A 490 µs N/A
🆕 CPU Simulation once[list] N/A 1.2 ms N/A
🆕 CPU Simulation repeated[interleave] N/A 504.5 µs N/A
🆕 CPU Simulation repeated[list] N/A 1.1 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jiengup:jiengup/scalar-probes/structural (6cacf75) with develop (b9150dd)

Open in CodSpeed

Footnotes

  1. 126 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@jiengup
jiengup force-pushed the jiengup/scalar-probes/structural branch from 6cacf75 to c4c5786 Compare October 28, 2026 04:49
Extract the reusable test-only helper from c91e73493b so independent
encoding migrations can carry the same prerequisite commit.

Source-commit: c91e73493b592c0436b08fa775cb594dc8afd955
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
(cherry picked from commit c91e73493b592c0436b08fa775cb594dc8afd955)
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Keep only this branch's portions of the original lint, session, and
benchmark follow-ups, including the matching manifest/lockfile changes.

Source-commits: 4c49e477e2, 8b1811ff84, 4405aa8187
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
@jiengup
jiengup force-pushed the jiengup/scalar-probes/structural branch from c4c5786 to 2b5bda2 Compare November 3, 2026 06:37
@jiengup

jiengup commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

@joseph-isaacs

Could you check this when you are free?
And also #11121 #11122

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant