Repository navigation
feat(array): migrate simple encodings to scalar probes - #10859
Conversation
|
this seems to be a duplicate of #10860 but there are more changes in the next one? |
|
Ah I see. Can you clean up the PR descriptions of both to be easier for a human to read? |
Merging this PR will improve performance by 16.6%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
99a8b0f to
c079274
Compare
Sure, I've cleaned up the redundant and duplicated content in these two PRs. |
c079274 to
278551f
Compare
|
Could you please take a look at this and #10860 ? |
|
I don't think we nee 14 new benchmarks. |
joseph-isaacs
left a comment
There was a problem hiding this comment.
Overall looks good, but I am not sure we need these benchmarks. Can you acutally remove the constant, prim and ones from probe in benches
|
Fixing:
Will be in the next commit. |
eb0f131 to
738a528
Compare
|
If you rebase this we can merge this pr, I just tried doing it and allow edit by maintainers is off |
Signed-off-by: Gunther Xing <jiengup@gmail.com> (cherry picked from commit 85ce9182220c667038a881ab2daa74d6f1c63d3d) Signed-off-by: Gunther Xing <jiengup@gmail.com>
Keep only this branch's portions of the original lint, session, and benchmark follow-ups, including the matching manifest/lockfile changes. Source-commits: 4c49e477e2, 8b1811ff84, 4405aa8187 Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
Signed-off-by: Gunther Xing <jiengup@gmail.com>
738a528 to
75ebc13
Compare
I've rebased and allowed all the PRs' edit that I've opened. |
Summary
OperationsVTable::scalar_attoprobe_scalar#9971OperationsVTable::scalar_atcannot retain preparation between row reads. The probe API introduced in #9843 retains encoding state, validity, and child probes across repeated reads.This PR migrates 8 implementations: Bool, Constant, Decimal, Null, VarBinView, ByteBool, Sequence, PythonVTable.
ProbeState = (); generic probe validity is reused where needed.ConstantandNullscalars supply their own nullnessSequencecomputes values directlyPythonVTableretains its unsupported scalar-access behavior.scalar_atremains a one-off probe compatibility shim.Verification
cargo nextest run --cargo-profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence --all-features --no-fail-fast— 4,832 passed, 1 skipped; the Python-binding Rust suite also passed (33 tests).uv run --locked --all-packages --reinstall-package vortex-data pytest --benchmark-disable vortex-python/test/test_pyarray.py vortex-python/test/test_scalar.py vortex-python/test/test_array.py vortex-python/test/test_encodings.py— 86 passed, 1 skipped.cargo test --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-features --doc --no-fail-fast— 85 passed, 21 ignored.cargo +nightly-2026-09-10 fmt -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --check— passed.cargo clippy --profile ci --locked -p vortex-array -p vortex-bytebool -p vortex-sequence -p vortex-python --all-targets --all-features -- -D warnings— passed.cargo benchruns — all 3 benchmark targets completed successfully.Comparison to baseline
Measured on a development machine (macOS with Apple Silicon); not representative of production performance
cargo bench -p <crate> --bench <name> -- --min-time 0.1 --sample-count 15. Values are Divan medians in microseconds per iteration, with one retained probe per iteration.AI Usage
GPT-6-Sol implemented the changes with Codex. The human reviewed all the code and can explain every line of it. Benchmark results and the origin/develop comparisons are done by Codex.