Skip to content

feat(array): migrate simple encodings to scalar probes - #10859

Merged
connortsui20 merged 4 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/simple
Nov 4, 2026
Merged

connortsui20 merged 4 commits into
vortex-data:developfrom
jiengup:jiengup/scalar-probes/simple

Conversation

@jiengup

@jiengup jiengup commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

OperationsVTable::scalar_at cannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads.

This PR migrates 8 implementations: Bool, Constant, Decimal, Null, VarBinView, ByteBool, Sequence, PythonVTable.

  • These implementations use ProbeState = (); generic probe validity is reused where needed.
  • Constant and Null scalars supply their own nullness
  • Sequence computes values directly
  • PythonVTable retains its unsupported scalar-access behavior.
  • Each migrated scalar_at remains a one-off probe compatibility shim.

Verification

  • cargo nextest run --cargo-profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence --all-features --no-fail-fast — 4,832 passed, 1 skipped; the Python-binding Rust suite also passed (33 tests).
  • uv run --locked --all-packages --reinstall-package vortex-data pytest --benchmark-disable vortex-python/test/test_pyarray.py vortex-python/test/test_scalar.py vortex-python/test/test_array.py vortex-python/test/test_encodings.py — 86 passed, 1 skipped.
  • cargo test --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-features --doc --no-fail-fast — 85 passed, 21 ignored.
  • cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --check — passed.
  • cargo clippy --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-targets --all-features -- -D warnings — passed.
  • Focused cargo bench runs — all 3 benchmark targets completed successfully.

Comparison to baseline

Measured on a development machine (macOS with Apple Silicon); not representative of production performance

cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15. Values are Divan medians in microseconds per iteration, with one retained probe per iteration.

Array Fixture Reads Baseline (µs) Migrated (µs) Change
Bool nullable random 256 4.957 4.624 -6.7%
Constant fixed scalar, random 256 2.666 2.103 -21.1%
Decimal decimal128, random 256 4.541 3.957 -12.9%
Null all null, random 256 1.958 1.260 -35.6%
VarBinView nullable strings, random 256 6.290 5.707 -9.3%
ByteBool nullable random 256 14.080 4.957 -64.8%
Sequence i64 arithmetic, random 256 8.582 2.333 -72.8%
PythonVTable Scalar probing is unsupported by this vtable — N/A N/A N/A

AI Usage

GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex.

@connortsui20

Copy link
Copy Markdown
Member

this seems to be a duplicate of #10860 but there are more changes in the next one?

@jiengup

jiengup commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

this seems to be a duplicate of #10860 but there are more changes in the next one?

No these are different ones. Both separated from #10460.

@connortsui20

Copy link
Copy Markdown
Member

Ah I see. Can you clean up the PR descriptions of both to be easier for a human to read?

@codspeed

codspeed Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 16.6%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 4 improved benchmarks
✅ 1761 untouched benchmarks
🆕 2 new benchmarks
⏩ 130 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation missing_stats[16] 92.7 µs 75.3 µs +23.1%
⚡ CPU Simulation bench_compare_varbin[(50000, 2)] 173.1 µs 144.5 µs +19.8%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(9999, 10000)] 244.3 µs 217.1 µs +12.51%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 104.3 µs 93.6 µs +11.41%
🆕 CPU Simulation once[bool] N/A 183.5 µs N/A
🆕 CPU Simulation repeated[bool] N/A 149.1 µs N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing jiengup:jiengup/scalar-probes/simple (75ebc13) with develop (bf3cfd0)

Open in CodSpeed

Footnotes

  1. 130 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@jiengup
jiengup force-pushed the jiengup/scalar-probes/simple branch from 99a8b0f to c079274 Compare October 28, 2026 04:49
@jiengup

jiengup commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Ah I see. Can you clean up the PR descriptions of both to be easier for a human to read?

Sure, I've cleaned up the redundant and duplicated content in these two PRs.
More PRs to cover #9971 are on the way.
Please let me know if you're still confused about the message.

@jiengup
jiengup force-pushed the jiengup/scalar-probes/simple branch from c079274 to 278551f Compare November 2, 2026 02:29
@jiengup

jiengup commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

@connortsui20 @joseph-isaacs

Could you please take a look at this and #10860 ?
I'd like to push more PRs if these two look good to you.

@joseph-isaacs

Copy link
Copy Markdown
Contributor

I don't think we nee 14 new benchmarks.

Comment thread vortex-python/src/arrays/py/vtable.rs Resolved

@joseph-isaacs joseph-isaacs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, but I am not sure we need these benchmarks. Can you acutally remove the constant, prim and ones from probe in benches

@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Nov 2, 2026
@jiengup

jiengup commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

Fixing:

  • @joseph-isaacs: review summary — Overall looks good, but I am not sure we need these benchmarks. Can you…
  • @joseph-isaacs: comment — I don't think we nee 14 new benchmarks.

Will be in the next commit.

@jiengup
jiengup force-pushed the jiengup/scalar-probes/simple branch 2 times, most recently from eb0f131 to 738a528 Compare November 2, 2026 14:09
@jiengup

jiengup commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@joseph-isaacs

I've dropped some benchs showing little performance change.
And the other two PRs #11121 #11122 are created.

@robert3005

Copy link
Copy Markdown
Contributor

If you rebase this we can merge this pr, I just tried doing it and allow edit by maintainers is off

Signed-off-by: Gunther Xing <jiengup@gmail.com>
(cherry picked from commit 85ce9182220c667038a881ab2daa74d6f1c63d3d)
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Keep only this branch's portions of the original lint, session, and
benchmark follow-ups, including the matching manifest/lockfile changes.

Source-commits: 4c49e477e2, 8b1811ff84, 4405aa8187
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
@jiengup
jiengup force-pushed the jiengup/scalar-probes/simple branch from 738a528 to 75ebc13 Compare November 4, 2026 16:53
@jiengup

jiengup commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

If you rebase this we can merge this pr, I just tried doing it and allow edit by maintainers is off

I've rebased and allowed all the PRs' edit that I've opened.

@connortsui20
connortsui20 enabled auto-merge (squash) November 4, 2026 17:04
@connortsui20
connortsui20 merged commit 8d9e027 into vortex-data:develop Nov 4, 2026
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants