Skip to content

feat(array): migrate Map, ScalarFn, and ParquetVariant to scalar probes - #11121

Open
jiengup wants to merge 6 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/delegating
Open

jiengup wants to merge 6 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/delegating

Conversation

@jiengup

@jiengup jiengup commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

OperationsVTable::scalar_at cannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads.

This PR migrates 3 implementations: Map, ScalarFn, ParquetVariant.

  • These implementations use ProbeState = () and reuse child probes without an encoding-owned cache.
  • Map wraps the entries scalar in the Map dtype, preserving null and empty maps without requiring a physical Struct child.
  • ScalarFn reads arguments through child probes and evaluates length-one Constant arrays with the existing VecExecutionArgs adapter. The result supplies nullness, avoiding recursion through ScalarFnValidity.
  • ParquetVariant checks outer validity, prefers typed values, and falls back to raw values. Typed structs still merge with raw object fields.
  • Each migrated scalar_at remains a one-off probe compatibility shim.

Verification

The following checks passed locally before rebasing onto 20cc6e2aef.

  • cargo nextest run --cargo-profile ci --locked -p vortex-array -p vortex-parquet-variant --all-features --no-fail-fast — 4,779 passed, 1 skipped.
  • cargo test --profile ci --locked -p vortex-array -p vortex-parquet-variant --all-features --doc --no-fail-fast — 84 passed, 21 ignored.
  • cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-parquet-variant --check — passed.
  • cargo clippy --profile ci --locked -p vortex-array -p vortex-parquet-variant --all-targets --all-features -- -D warnings — passed.
  • Focused cargo bench runs — all 2 benchmark targets completed successfully.

Comparison to baseline

Measured on a development machine (macOS with Apple Silicon); not representative of production performance.

cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15. Values are Divan medians in microseconds per iteration, with one retained probe per iteration.

Array Fixture Reads Baseline (µs) Migrated (µs) Change
Map nullable entries, random 256 160.900 62.580 -61.1%
ParquetVariant mixed typed/raw, random 256 36.540 30.450 -16.7%

The Map result includes the migrated entries child from the cumulative branch; that child migration is outside this PR.

AI Usage

GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex.

Extract the reusable test-only helper from c91e73493b so independent
encoding migrations can carry the same prerequisite commit.

Source-commit: c91e73493b592c0436b08fa775cb594dc8afd955
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
(cherry picked from commit 63d0c40f8f34a9bea3baa04c26d3a0c1b9f0c5a0)
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
(cherry picked from commit 6d94a37c33791a3b0fafd58b01e7b72b84f31c73)
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
(cherry picked from commit 8b1811ff8450b7a0bb21d0df407971ddeb42b23d)
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Keep only this branch's portions of the original lint, session, and
benchmark follow-ups, including the matching manifest/lockfile changes.

Source-commits: 4c49e477e2, 8b1811ff84, 4405aa8187
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant