Repository navigation
Sync develop with vortex-data/vortex develop - #15
Closed
ChristopherWirt wants to merge 41 commits into
Closed
ChristopherWirt wants to merge 41 commits into
ChristopherWirt wants to merge 41 commits into
Conversation
This is needed for adding a dynamic registry for partitioning rules Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…vortex-data#10796) ## Summary `codspeed` 6.0.0 (vortex-data#10784, vortex-data#10785) adds process isolation, which is on by default. `codspeed-divan-compat` now runs each benchmark in a new copy of the benchmark binary. Simulation mode measures one iteration without warmup, so each benchmark also pays for allocator setup (mimalloc) and for simulated cache misses on memory it touches for the first time. Before, an earlier benchmark in the same process had already paid these costs. The CodSpeed report on vortex-data#10785 shows a near-constant ~15 µs added to short benchmarks.
Instead of error being an enum we convert an error to be a struct with message and error kind. I had followed the structure of Python errors which have couple error categories https://docs.python.org/3/builtins/exceptions.html#exception-hierarchy majority of which don't apply to this project Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary - Tracking Issue: vortex-data#10701 Adds compat fixtures for the old decimal forms that current writers still produce: decimals stored narrower than the canonical width of their precision (including values that need the next precision, and high scales), decimals stored wider, nonzero values in null slots, and DecimalByteParts with narrow and wide most significant parts. A release must publish these files before vortex-data#10701 changes how old decimal arrays read, so that the new read path is checked against real old files. ## Changes A flat fixture can now override `build_written()` to write a physical form that `build()` must not return. `generate` and `publish` write `build_written()`, and `check` compares each published file with `build()`, which returns canonical arrays only. So the published file keeps the old form while the expected array never changes. The overrides work only while `DecimalArray::try_new` accepts a non-canonical width. Once it rejects one, the overrides must be deleted, and the files published before that keep the old forms. `DESIGN.md` documents the hook. The new tests check that every decimal in `build()` has the canonical width and that every written column keeps its stored width through a file round trip. That round-trip check reads with the current reader, so the change in vortex-data#10701 that converts old decimal arrays on read must update it. Until a release publishes the new fixtures, `compat.py check --mode last` skips them. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary - Tracking Issue: vortex-data#9128 - Independent validity candidate: vortex-data#10742 Routes float32 comparisons through the existing word-packing collectors. This recovers the measured float32 gains from the RowFn comparison investigation with two private dispatch changes. It is independent of the comparison migration and fusion work in vortex-data#10676 and vortex-data#10675. ## Changes Adds `F32` to the existing collector selections for array pairs and array/constant comparisons. Integer collector policies remain unchanged. The floating-point documentation now accounts for NaN signs and payloads. **Draft pending fresh isolated benchmarks on this head.** Saved measurements show a consistent float32 gain, but also some unrelated ARM regressions. The later combined packing/validity candidate had an AVX2 regression that has not been attributed to either change individually. <details> <summary>Saved benchmark evidence and validation</summary> The isolated packing snapshot `4c98df49e379853f8e990d9764791b5330511771` was compared with frozen `develop` at `a1693389301e446eec41373cb274d98a303d3ca0`. Measurements used Rust 1.99.1 / LLVM 23.1.1, 16 CGUs, no LTO, CPU 2, and 21 balanced paired blocks on Graviton3 C7g and Sapphire Rapids C7i. This was native wall time, not CodSpeed simulation. Across 48 float32 cases at 8,192 and 65,536 rows, covering array pairs, both constant orientations, equality/ordering, and four validity shapes, the median block ratios of equally weighted geometric means were: | Target | Candidate / baseline time | 95% paired bootstrap interval | | --- | ---: | ---: | | NEON | 0.611 | 0.610–0.611 | | AVX2 | 0.632 | 0.630–0.633 | | AVX-512 | 0.679 | 0.678–0.681 | Among all 375 probe cases, the isolated snapshot had no flagged regressions on AVX2 or AVX-512. ARM had four, including roughly 4.7% slower u64 multiply and 4.6% slower large f64 equality. Those paths do not use the changed float32 collector, so their movement is not evidence of extra float32 work, but it still counts against a no-regression claim. A separate combined packing/validity candidate at `8d8ef8151454bbc73e1cd0b236567542a1401cbe`, based on `d834a11faa79370b4d2e0bbcc73195f451a8a212`, retained the float32 gain but regressed large nullable u64 add by about 18% on AVX2. Profiling found a byte-identical retry hot loop with essentially unchanged instructions per operation and more cycles. The precise placement-sensitive mechanism remains unresolved. That combined result does not isolate this PR. The combined candidate passed 225 RowFn tests and 232 binary-function tests on that newer base. Separate native probes checked scalar-reference results and float/encoding/tail edge cases. The changed file passed the pinned formatter. These are saved checks, not fresh validation of this individual PR. This PR is rebased onto `c750fbd80882f78efd6a2703bc0f3e17f1aca9f9`. The patch is unchanged by the rebase. Tests and benchmarks were not rerun after it. </details> Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`ArrayRef::as_constant`, `Array::as_constant` and `ReduceNode::as_constant` now return `Option<&Scalar>` instead of cloning the constant's scalar. Let the callers decide if they need an owned scalar
…data#10707) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary - Closes vortex-data#10750 - Replaces vortex-data#10746 The docs on `Validity::cast_nullability` read as if every `CastReduce` rule that removes nullability must call `Validity::trivially_cast_nullability` and pair with a `CastKernel`. Most push-down rules (FoR, ALP, ALP-RD, ZigZag, RLE, RunEnd, DateTimeParts, DecimalByteParts, Chunked, Sparse, Extension) instead cast the child that holds their validity, and the cast of that child already fails on nulls, so no result is wrong. Guarding each rule, as vortex-data#10746 does for DecimalByteParts, adds a kernel per encoding and delays the rewrite until execution without changing any result. The guard in vortex-data#10746 comes from the closed vortex-data#10230, where a canonical decimal held a lazy child that a cast to canonical did not execute, by way of the FX2 notes in vortex-data#10633. `develop` keeps one buffer per decimal, so that reason no longer applies. ## Changes `CastReduce` now owns the contract and documents both forms of rule. A rule that builds a new `Validity` keeps the guard and the kernel, and a rule that pushes the cast into its children needs neither when the nulls of those children are exactly the null rows of the array. `Dict` is the documented exception, because its values can hold nulls that no code references. No new tests are needed: every push-down encoding listed above already runs `test_cast_conformance` on a partially valid array, and that suite asserts that the cast to non-nullable fails. --- _Generated by [Claude Code](https://claude.ai/code/session_01V1YHcosAAKufkyKPorUC2n)_
## Summary Split out of vortex-data#10761. This keeps only the `vortex-fastlanes` cast change; the bit-packed Arrow dictionary export from that PR is dropped. `BitPacked`'s `CastKernel` only took its direct unpack path for widening integer casts. Bit-packing rejects negative integers, so a patch-free array packed at a global `bit_width` holds only values in `0..2^bit_width`. A narrowing cast whose target holds every such value cannot overflow. One place this applies is the Arrow export of Vortex dictionaries. `dict_to_dict` exports the codes through `to_arrow_primitive`, which casts them to the requested key type. Dictionary codes are often bit-packed, so a key type narrower than the codes' ptype hits this kernel. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary Fixes vortex-data#10623. `Dict::fill_null` converted the index of the matching dictionary value into the codes type with an unchecked `as` cast. A dictionary can hold more values than its codes type can index, so an unreferenced fill value could truncate to the wrong code (with `u8` codes, filling with value 300 returned value 44) or panic for signed codes. ## Changes The kernel now checks the index against the range of the codes type. When the index does not fit, the kernel declines and the generic `fill_null` path fills the decoded array, which is slower but correct. Non-nullable codes need no fill code, so they keep the fast path. The new tests cover `u8` and `i8` codes at and past their maximum, and non-nullable codes over nullable values. The three cases past the maximum fail without the fix. The `Narrow` work (vortex-data#10733) depends on this fix, because a rule there makes narrow dictionary codes common. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary - Tracking Issue: vortex-data#9128 - Supersedes vortex-data#10674 - Related discussion: vortex-data#9694 - Independent float32 candidate: vortex-data#10741 Attaches input validity directly to canonical RowFn output using the existing helper. Preserves kernel-output validation, relabeling, outer nullability, and the non-canonical mask fallback. `values.mask(valid)` builds and optimizes a `ScalarFnArray` to dispatch through parent-reduce rules. The benchmark-driven bypass stays inside RowFn to avoid adding canonical-specific dispatch to the generic builtin. It can be removed if `.mask()` gains equivalent direct dispatch. This revisits the abstraction concern in vortex-data#9694. **Draft pending fresh isolated benchmarks on this head.** Saved results include regressions and do not establish a general speedup. <details> <summary>Saved benchmark evidence and validation</summary> The isolated validity snapshot `b1961452ea3505d4e2588e34b14c20ea177ed175` was compared with frozen `develop` at `a1693389301e446eec41373cb274d98a303d3ca0`. Measurements used Rust 1.99.1 / LLVM 23.1.1, 16 CGUs, no LTO, CPU 2, and 21 balanced paired blocks on Graviton3 C7g and Sapphire Rapids C7i. This was native wall time, not CodSpeed simulation. For nullable u64 addition with 64 rows and a right-hand constant, median paired time ratios were: | Target | Candidate / baseline time | 95% paired bootstrap interval | | --- | ---: | ---: | | NEON | 0.798 | 0.796–0.802 | | AVX2 | 0.850 | 0.846–0.852 | | AVX-512 | 0.846 | 0.844–0.855 | The isolated snapshot also had flagged regressions outside that case: roughly 3.4% slower ARM u64 multiply and up to 4.2% slower AVX2 byte comparisons. No regressions crossed the 3% flag on AVX-512 in that run. These movements remain part of the review, even where unchanged code is sensitive to compiler context or binary layout. A separate combined packing/validity candidate at `8d8ef8151454bbc73e1cd0b236567542a1401cbe`, based on `d834a11faa79370b4d2e0bbcc73195f451a8a212`, retained the small-batch gain but regressed nullable u64 add by about 12% at 8,192 rows and 18% at 32,768 rows on AVX2. Profiling found a byte-identical retry hot loop with essentially unchanged instructions per operation and more cycles. The precise placement-sensitive mechanism remains unresolved. That combined result does not isolate this PR. The combined candidate passed 225 RowFn tests and 232 binary-function tests on that newer base, including output validation, allocator, and deferred-error coverage. Separate native probes checked null-row arithmetic errors and comparison edge cases. The changed file passed the pinned formatter. These are saved checks, not fresh validation of this individual PR. This PR is rebased onto `c750fbd80882f78efd6a2703bc0f3e17f1aca9f9`. The patch is unchanged by the rebase. Tests and benchmarks were not rerun after it. </details> --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…0792) ## Summary - Tracking Issue: vortex-data#10791 On Spark 4.2 every read through `VortexTable` fails with a `NoSuchMethodError`: Spark 4.2 changed `CatalogV2Util.structTypeToV2Columns` to take an extra `boolean`. This is a stopgap. vortex-data#9658 rewrites the connector as a `FileFormat`, deletes `VortexTable` and adds Spark 4.2 to its compat suite. Once it merges, this change goes with it. Until then, it keeps the current connector usable on Spark 4.2. ## Changes - `VortexTable` builds its columns with `Column.create(name, type, nullable, comment, metadataJson)` instead of `CatalogV2Util.structTypeToV2Columns`. The method has the same signature in Spark 3.5, 4.1 and 4.2, so one jar still serves all three. - `-PsparkTestVersion` runs the `vortex-spark` tests against a newer Spark than the jar compiles against. The jar still compiles against 4.1.2. - New `VortexTableTest` case: the scan's read schema keeps nullability, comments and metadata. Tested with: - `./gradlew :vortex-spark_2.13:test`: 211 passed on Spark 4.1.2 - `./gradlew :vortex-spark_2.13:test -PsparkTestVersion=4.2.0`: 211 passed on Spark 4.2.0 ## AI assistance Disclosed per the [AI policy](https://docs.vortex.dev/project/contributing#ai-assistance): the investigation, code and tests were drafted with Claude (Claude Code) and reviewed by me. Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
…a#10808) ## Summary CodSpeed reported a regression on 49 of 68 recent PRs that changed no benchmarked Rust code. On the walltime legs it compares the fastest sample, and divan runs most benchmarks there with one iteration per sample, so one fast iteration out of 1000 can decide the result. Expect a one-time change in the two pinned series on this PR's own report. ## Changes A fixed `sample_size` makes each sample the average of several iterations and stops divan from picking a different sample size per run. Those were the two causes behind `bitpack_blocked_compress` (flagged on 148 PRs) and `words_gather_dispatch` (58). `bitpacking_filter.rs` gets back the `mimalloc` allocator that a rewrite dropped, and the guide now makes `mimalloc` mandatory and lists what not to do, so the same flakes stop coming back. The NEON `threshold`/`filter`, `mul_u64_nonnull` and `starts_with[VarBin]` benchmarks also switch between two states for a whole run, but the cause is not known and a larger iteration does not help. The guide lists them as known unstable instead of changing them. <details> <summary>Evidence and checks</summary> - Sources: the CodSpeed comments on the last 300 PRs and the divan tables from 100 CodSpeed runs. - The values in CodSpeed's comments match divan's `fastest` column in 108 of 122 checkable cases, and the median in 9. - `bitpack_blocked_compress` (AVX2): the median stayed within 1% over 100 runs. The fastest iteration was near 6.7 µs in 29 runs and near 7.55 µs in the rest. - `words_gather_dispatch` (NEON): divan picked one iteration per sample in 42 runs (1.98 to 2.01 µs) and two in 57 runs (1.80 to 1.82 µs). Checks run: - `cargo clippy --locked -p vortex-buffer -p vortex-fastlanes --benches --features vortex-fastlanes/_test-harness -- -D warnings` - `RUSTFLAGS="--cfg codspeed -D warnings" cargo check` on the same targets. The touched files are clean. `canonicalize_bench.rs` (untouched) has unused-import errors under this configuration. - `cargo +nightly-2026-09-10 fmt --check`, `typos`, `git diff --check` - Built both benchmarks and confirmed that divan runs 8 and 16 iterations per sample. Not run: the CodSpeed jobs (they run on this PR) and the Sphinx docs build. </details> https://claude.ai/code/session_01KF3pMFx7U5ngk8BzQQFS2W --- _Generated by [Claude Code](https://claude.ai/code)_ --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>
…x-data#10752) ## Summary - Tracking Issue: vortex-data#10701 Adds benchmarks for the decimal paths that vortex-data#10701 changes, so CodSpeed has a base before those changes land. They cover compute on DecimalByteParts arrays whose most significant part is narrower than the canonical width (decode, compare, filter, sum and min-max), a compressed DecimalByteParts tree in the decompress throughput bench, casts that grow the canonical width, and decimal export to Arrow and DuckDB. ## Changes The DecimalByteParts compute, decompress and cast benches run in CodSpeed. The Arrow and DuckDB export benches run locally only, because those crates have no CodSpeed shard, and the decimal compress bench stays inside the existing `cfg(not(codspeed))` module. The compress bench enables the default `core` edition, so the decimal scheme uses its v1 variant as in a default write. `vortex-duckdb` gains a `#[doc(hidden)]` `_benchmarking` module that re-exports `ArrayExporter` and `ConversionCache` for its bench, and the fields of `ConversionCache` become `pub(crate)` so that the module exposes only the type. No library behavior changes. Every input is a canonical decimal or a DecimalByteParts array, so the benches stay valid once every decimal array must have the canonical width. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…10735) The scaled-down integer to decimal cast selects its carrier as `max(carrier(S), T)`. The source type `S` and the target type `T` fix the carrier, but `match_each_decimal_value_type!` compiled the kernel for all six decimal widths in each of the 48 `(S, T)` pairs. An `if const` condition now removes the five arms that cannot match before code generation. The arm that runs is the same as before. `primitive_decimal_carrier_type` becomes a `const fn` so that the condition can use it. | | before | after | |---|---|---| | `vortex-array` release `__text` (macOS arm64) | 14,012,988 B | 13,697,852 B (-315 KB) | | `vortex-array` LLVM IR lines | 5,405,422 | 5,200,767 | Signed-off-by: Robert Kruszewski <github@robertk.io>
…#10736) The filter take kernel for fixed-width values copies values and does not inspect them, but it was compiled for each of the 17 primitive and decimal value types, for each of the 8 index types. `take_primitive` and `take_decimal` now reinterpret the values buffer by its byte width, so six kernels (1, 2, 4, 8, 16 and 32 bytes) serve all value types. The reinterpretation does not copy data, and the per-element loop is the same. | | before | after | |---|---|---| | `vortex-array` release `__text` (macOS arm64) | 14,012,988 B | 13,808,764 B (-204 KB) | Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The generic constructors (`ConstantArray::new<S>`,
`PrimitiveArray::{new, try_new, new_unchecked, empty}<T>`,
`DecimalArray::{try_new, new_unchecked}<T>`, and the `impl IntoIterator`
constructors of `StructArray` and `ChunkedArray`) called the generic
`Array::from_parts_unchecked` and `Array::try_from_parts` directly. Each
crate that called them monomorphized its own copy of those functions and
of the full `DynArrayData` vtable for the encoding.
The generic constructors now call a private, non-generic builder.
`vortex-array` compiles each vtable once, and other crates call it.
`ScalarFn` had the same problem because `new_from_validated_slots` was
`#[inline]`, and it is not inlined now. The cost is one function call
when an array is constructed.
| `vortex-ffi` release cdylib (macOS arm64) | before | after |
|---|---|---|
| `__text` | 37,223,740 B | 36,613,308 B (-610 KB) |
| Copies of the vtable: Constant / Primitive / Struct / Chunked /
Decimal / ScalarFn | 22 / 17 / 7 / 5 / 5 / 4 | 1 / 1 / 1 / 1 / 1 / 1 |
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tex-data#10739) **Negated operators share kernels.** `NotEq`, `Gte` and `Lte` are the inverse of `Eq`, `Lt` and `Gt`, so each pair now uses one set of kernels. When the operator is a negation, `stream_compare_fused` inverts each 1024-bit output block right after the transpose, while the block is still in cache. This adds 16 word NOTs for each 1024 values. `between` passes `negate = false`. **Signed types use the unsigned kernels.** Bit-packing rejects negative integers (`ensure_non_negative_integers`), so packed values of a signed type are never negative. - A negative constant compares the same against every packed value as against zero, so the result is constant. It is handled before the fused kernel. - Otherwise both sides are non-negative, and the unsigned physical values compare the same as the signed values. The fused kernel now takes function items of the physical type (`<T as PhysicalPType>::Physical::is_lt` and so on), so signed and unsigned arrays of one width share one set of kernels. Patches and width 0 still use the exact predicate on the logical type. | `vortex-fastlanes` release `__text` (macOS arm64) | bytes | |---|---| | develop | 4,442,156 | | negated operators share kernels | 3,279,596 | | + signed types use unsigned kernels | 2,846,252 (-1.60 MB, -36%) | --------- Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The list take by `PiecewiseSequence` compiled its gather kernels for every combination of piece start, piece length, list offset and output offset type (4 x 4 x 4 x 4). The kernels read each piece start and length only once per piece. The inner loop over list offsets depends only on the offset types. The take now widens the starts and lengths to `u64` and dispatches only on the offset types. The widening does not copy when the pieces are already `u64`, which is the type that the list take itself produces. For other widths it copies the piece arrays once. That copy is proportional to the number of pieces, not to the number of elements. | | before | after | |---|---|---| | `vortex-array` release `__text` (macOS arm64) | 14,012,988 B | 12,798,204 B (-1.21 MB) | Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary - Tracking Issue: vortex-data#10804 Follow-up to vortex-data#10745, as suggested in review there. When the fill value already exists in the dictionary values but its index does not fit the codes type (e.g. `u8` codes and the fill value at index 300), the `Dict` `fill_null` kernel returned `None`, so the generic path decoded the whole array before filling. The kernel only rewrites the null codes, so it can widen them instead and keep the result dictionary-encoded. ## Changes - For nullable codes whose type cannot hold the fill index, cast the codes to `PType::min_unsigned_ptype_for_value(fill_index)` (kept nullable) and then fill the nulls with the index. Codes are never negative, so the cast keeps every code. Codes that already fit and non-nullable codes take the same path as before. - New test `fill_unreferenced_value_widens_codes` calls the kernel directly and checks that it returns a `Dict` with widened, non-nullable codes and the right values, for `u8 -> u16` (256, 300), `i8 -> u8` (128), `i8 -> u16` (256) and `u16 -> u32` (65536). All five cases fail on `develop`, where the kernel returns `None`. Tested with `cargo test -p vortex-array --lib` (4437 passed), `cargo clippy -p vortex-array --all-targets --all-features -- -D warnings`, and `cargo +nightly-2026-09-10 fmt -p vortex-array`. AI assistance: I used an AI coding assistant to help write this change and the description; I reviewed the code and ran the tests above myself. Signed-off-by: Vincent <1729065730@qq.com>
vortex-data#10810 updated `cc` from 1.8.0 to 1.10.0. After that, `Rust tests (windows-x64)` fails in the `libmimalloc-sys` build script: ```text c1xx: fatal error C1083: Cannot open source file: '/WX-': No such file or directory error: failed to run custom build command for `libmimalloc-sys v0.1.49` ``` Cause: 1. Starting with `cc` 1.9.0, `cc` runs `cl.exe` through the `RUSTC_WRAPPER` fallback. CI sets `RUSTC_WRAPPER=sccache`. `cc` 1.8.0 runs `cl.exe` directly. 2. For `.cc` sources, `cc` passes `-Tp <file>` as two separate arguments. sccache 0.18.0 does not know `-Tp`, so it treats it as an unknown flag. 3. The sccache preprocess command adds `/WX-` before the input file. `cl.exe` then reads `/WX-` as the `-Tp` source file. --------- Signed-off-by: Robert Kruszewski <github@robertk.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rtex-data#10744) ## Summary In DataFusion, `a NOT IN (1, 2, 3, 4, 5)` counted null rows that sit in an all-null chunk. Data: `id` Int32 0..999. `a` Int32 nullable, null for rows 0..499 and `id % 20` for rows 500..999. Written as ten 100-row chunks (`WriteStrategyBuilder` with `with_row_block_size(100)` and `with_data_block_target_bytes(None)`), so rows 0..499 are five all-null chunks. ```sql SELECT count(*) FROM t WHERE a NOT IN (1, 2, 3, 4, 5) ``` A DataFusion `MemTable` with the same data returns 375. On develop Vortex returns 875, so it counts all 500 null rows. With this change it returns 375. vortex-datafusion turns a NOT IN of consecutive integers into `not(between(...))` (`in_list_of` in `vortex-array/src/expr/exprs.rs`) and reports the filter as pushed, so DataFusion does not check it again. Each all-null chunk is a `ConstantArray`. The `Constant` `BetweenReduce` rule compared it with the constant bounds using `Scalar`'s ordering, which puts null below every value. So `between` returned `false` instead of null, and `not` turned that into `true`. In normal dispatch null bounds are handled earlier by `short_circuit`, so a null array value is the case that got through. ## Changes `short_circuit`, which runs before every encoding's `between`, now returns a null `Bool` constant when the array is a null constant, with the combined nullability of the array and both bounds. The new test `scalar_fn::fns::between::tests::constant_null_array` checks that `between` and `not(between)` are all null for `f64` and `i32`. Checked locally: `cargo test -p vortex-array --lib between` (46 passed, and the new test fails on develop), the DataFusion query above (875 on develop, 375 with this change), clippy with `-D warnings`, and nightly fmt. I used AI tools to help write and review this change. --------- Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Third of a stack of three that lands the planner-driven V2 scan for DataFusion behind a switch, cut from vortex-data#10694. Stacked on vortex-data#10707 (the planner chain), itself on vortex-data#10706. With this merged, `scan_v2` runs TPC-H end to end; the perf and diagnostics commits of vortex-data#10694 follow as separate PRs on top. ## Measured on its own Against the V1 scan, locally (14 cores, interleaved A/B/B/A, 10 and 5 iterations a run) at the same code before the stack was cut (da8a7c8): | query | SF1 | SF10 | |---|---|---| | q1 | 1.074 | 1.024 | | q6 | 0.966 | 0.832 | | q10 | 1.025 | 0.882 | | q12 | 0.939 | 0.811 | | q13 | 1.640 | 1.125 | | q19 | 0.996 | 0.909 | | geomean | 1.085 | 0.924 | SF10 improves over vortex-data#10707 (0.977 to 0.924) as the per-step task hops go; SF1 does not yet, and q13 at SF1 is far behind because every partition still plans the filter and projection itself (the plan-once and interning commits of vortex-data#10694 fix that, taking the full branch to 0.963 at SF1). This PR's base is a branch, so CI's benchmark dispatch does not run on it. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…a#10809) Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…odings (vortex-data#10558) UNSTABLE exists to aggregate all core encodings that are in the works and should be tested and benchmarked but not written by default writer
## Summary The writer’s unused copy of `DictLayoutMetadata` recommends prost’s lenient enum getter. ## Changes Remove the duplicate and its constructor. The active dictionary layout metadata is unchanged. ## API Changes Removes `vortex_layout::layouts::dict::writer::DictLayoutMetadata`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This is still transitively picked up through arrow_cast and parquet variant but the logic is simpler without it
A const condition removes the unreachable arms before codegen in three places: - `check_decimal_identical` dispatched on the lhs storage type, the rhs storage type and their widest type. The two distinct inputs fix the widest type, so only 30 of 216 arms can run. - `widened_buffer::<W>` is only called for storage types narrower than `W` (all callers pass the wider of two inputs), and the narrowing branch of `converted_buffer::<W>` only runs for storage types wider than `W`. Only 15 of 36 arms of each can run. - The bit-packed widening cast compiled all 64 integer pairs, but only widening pairs reach the kernel. Bit-packing rejects negative values, so sign extension of a packed value is the same as zero extension. The bit-packed cast now views a signed source as unsigned (the packed buffer is shared and the patch values are reinterpreted) and reinterprets the result to the target type. Signed and unsigned casts of the same widths share one kernel, so the cast compiles 6 kernels instead of 18 reachable ones. | release `__text` (macOS arm64) | before | after | |---|---|---| | `vortex-array` | 14,023,548 B | 13,933,616 B (-90 KB) | | `vortex-fastlanes` | 4,442,156 B | 4,331,764 B (-110 KB) | Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary - Tracking Issue: vortex-data#10701 Prost decodes an enum value it does not know as the first variant, so a stored `DecimalType` or `PType` outside its enum silently reads as `I8` or `U8`. The reader then decodes the buffer or child at the wrong width, which either fails later with an unrelated error or returns wrong values. Readers need to reject such values before a later release adds a variant, such as a wider integer `PType`. The read path for old decimal arrays in vortex-data#10701 also depends on the stored `values_type`, because it compares that width with the canonical width. ## Changes Each metadata struct with one of these fields gets a `get_<field>()` accessor that returns an error naming the field and the raw value, and `deserialize` calls it before reading any buffer or child. The protobuf dtype decoder does the same through `pb::PType::try_from`. Files from current writers read as before, and `compat.py check --mode all` passes against every published fixture. The handwritten enum fields use `prost(int32)`, so prost no longer generates lenient enum getters or setters. The protobuf tags and encoded bytes stay the same. For review, read the commits one at a time. ## API Changes Removes the generated enum getters and setters from the affected handwritten metadata structs. Decoding uses the strict `get_<field>()` methods. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Reject raw serialization of sliced PCO arrays because the format does not encode slice offsets. This prevents writing data that cannot reconstruct the original slice; callers can materialize the slice before serialization. Signed-off-by: Daniel King <dan@spiraldb.com>
Depends on vortex-data#10755. ## Summary Public metadata fields expose the stored layout as Rust API and let callers bypass typed accessors. Restrict them to the owning crate while preserving the encoded metadata. ## Changes Add getters for CUDA’s Zstd metadata and a constructor and CRS getter for spatial callers. The TUI uses the existing flat-layout accessor. Internal struct literals and metadata golden files stay unchanged. Formatting and clippy passed. Tests passed apart from `chunked_input_with_mixed_dimensions_returns_error`, which fails identically on the base. GPU tests were skipped without `nvcc`. ## API Changes Metadata fields are now `pub(crate)`. Direct field access and struct literals from other crates no longer compile. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The v2 VortexDataSource reports incorrect statistics when there are filter expressions or limits. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
Allow passing a custom expression converter into the V2 DataFusion VortexDataSource --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
…rtex-data#10062) Ordered scans must emit partitions in source order even when later partitions are ready first. Concatenate ordered partition streams while preserving bounded unordered concurrency and the existing global fetch trimming. Closes vortex-data#10056. Deterministic regressions cover complete output, fetch zero, the first row, and a fetch crossing partition/batch boundaries. Three cases fail with unordered flattening; all four pass with the fix. The combined v2 module passes 17 tests; scoped Clippy and pinned formatting pass. AI assistance: Claude Code and Codex, under @gatesn's direction. --------- Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Nicholas Gates <nick@nickgates.com>
Returns arbitrary aggregate function partials for a given field path. This allows callers to probe for any statistics. To consider: * Exposing this as a batch endpoint? (Vec<AggregateFnRef> -> Vec<Precision<Scalar>>) * Deprecate the field_statistics function --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ta#10743) ## Summary Adds `ListArray::compact` and a list view rebuild mode, `ListViewRebuildMode::MakeExactWithEmptyNulls`. Both return the same lists over only the elements that valid lists reference: nothing before the first list, after the last, between lists or under a null list. A list column can hold elements that no valid list shows. Slicing keeps the parent's elements on both sides, and a null list may still cover elements. A function that only reads values does not care, but a function that can fail on an element, such as a checked `x + 1`, must not see them: an overflow in an element under a null list would fail the whole batch. `list_transform` (vortex-data#10503, stacked on this PR) is the first user; it applies its lambda to the rebuilt elements whenever the lambda can fail. For example: ```text elements [9, 1, 2, 9, 9, 3] offsets [1, 3, 5, 6] validity [valid, null, valid] lists [1, 2], null over [9, 9], [3] rebuilt elements [1, 2, 3] offsets [0, 2, 2, 3] ``` ## How it works **`ListArray`** - Non-nullable or all-valid lists with nothing at the ends are returned as they are, after reading two offsets. - Otherwise it looks for null lists that hold elements: one pass marks the lists whose next offset is larger, and one bit operation clears the valid ones. Only when bits are left does it collect them into runs. - With no such run, only the ends go: the elements are sliced to `[first offset, last offset)` and the offsets are shifted down, without copying an element. - With some, the elements between those runs are kept, by a slice if one range is left and otherwise by one filter whose mask carries the ranges, so it copies range by range. The offsets are copied once and each one moves down by the elements dropped before it, so the null lists in those runs become empty. - The offsets keep their type and the validity is unchanged. **`ListViewRebuildMode::MakeExactWithEmptyNulls`** - `MakeExact` only trims the ends of a view that is already zero-copy to a list, so its null lists keep their elements. The new mode drops those too. - Such a view has its lists back to back, so it goes through the same two steps on its offsets, with the last list's end as the final boundary. It keeps its layout and its offsets' and sizes' types. Only the sizes of the null lists that held elements are set to 0; otherwise the sizes are kept as they are. - Any other view is rebuilt with `MakeExact`, which already copies only the valid lists' elements and leaves null lists empty. `MakeExact`'s documentation now says both. - The result is always zero-copy to a list. ## Benchmark `list_transform` with a checked `x + 1` over 100k lists (~800k `i32` elements, ~1/8 of the lists null), through vortex-data#10503, exported to Arrow, median µs over three alternating runs of a local divan benchmark. "Before" is vortex-data#10503 without this PR, which used the existing list view rebuild instead. | Lists | Before | This PR | DataFusion 54.1 `array_transform` | |---|---|---|---| | Null lists holding elements | 1,272 | 253 | 224 | | Null lists already empty | 97 | 73 | 60 | | Sliced, with elements on both sides | 78 | 81 | 55 | ## Testing - `rstest` cases for both types: ends to drop, null lists holding elements or not, every list null, no lists, an empty valid list between null lists, overlapping and out-of-order views, views with and without the zero-copy flag. - Exact arrays come back as the same arrays (`ArrayRef::ptr_eq`), and a sliced `ListArray` whose validity starts mid-byte is rebuilt correctly. - A seeded random test compares `ListArray` and both kinds of view against a reference that gathers each valid list's elements, over five offset types, with non-nullable validity and validity arrays that start mid-byte. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
## Summary Moves the `vortex-runend` crate into `vortex-array` as `vortex_array::arrays::runend`, the same way `vortex-dict` moved in vortex-data#5289. Once RunEnd is built in, vortex-array code can build run-end arrays itself. One example is spreading a captured outer value across each list's elements in `list_transform` (vortex-data#10702). ## Changes - `RunEnd` is now a built-in, like `Dict`. It is registered in `ArraySession::default`, in the default kernel session, and in `AggregateFnSession::default` (min_max, is_constant, is_sorted, sum). - The `vortex-runend` crate is removed. `vortex::encodings::runend` still re-exports the module. - The benches move to `vortex-array/benches/run_end_*`. CodSpeed picks them up in the array builders shard. --------- Signed-off-by: Matt Katz <mhkatz97@gmail.com>
## Summary
`list_transform(list, lambda)` applies a one-parameter lambda to every
element of a `List` (in any encoding), keeping each list's length and
validity.
```rust
list_transform(col("l"), Lambda::new(["x"], checked_add(var("x"), lit(1_i32)))?)
// vortex.list.transform($.l, (x) -> (@x + 1i32))
```
## Changes
- `ListTransform`'s options are `ListTransformOptions`: the lambda as
written (`Lambda`) in an expression, or bound to the element's dtype
(`BoundLambda`) in a bound expression or an array. Its only child is the
list.
- `bind_options` binds the lambda to the element's dtype when the node
is bound, so `execute` only runs it. A `FixedSizeList`, or a lambda with
anything but exactly one parameter, is rejected there and in
`return_dtype`. A lambda already bound to another element dtype is an
error.
- The result keeps the input's shape: a `ListArray` stays a `ListArray`
and anything else runs as a list view, with the same offsets (and sizes)
and validity around the new elements.
- A lambda that can fail runs only on the elements valid lists
reference. One that can't runs on the whole elements buffer, cut to the
lists' range for a `ListArray`.
- The function is strict (a null list gives a null result), it can fail
only if the lambda can, and it is not serializable yet.
## Benchmark
100k lists, ~800k `i32` elements, ~1/8 of the lists null where nullable,
result exported to Arrow, median µs, local scratch divan benchmark
against DataFusion 55.2's `array_transform` on the same lists:
| Lists | Lambda | DataFusion | This PR |
|---|---|---|---|
| Exact | `x + 1` | 49 | 71 |
| Null lists already empty | `x + 1` | 60 | 73 |
| Null lists holding elements | `x + 1` | 224 | 274 |
| Null lists holding elements | `x = 3` | 249 | 81 |
| Sliced | `x = 3` | 85 | 99 |
| List view, back to back | `x + 1` | 903 | 75 |
Most of the `x + 1` gap is Vortex's checked add against DataFusion's
wrapping add; with `x = 3` the two kernels are even. DataFusion casts a
list view to a list before transforming it.
---------
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | --- > [!WARNING] > Some dependencies could not be looked up. Check the [Dependency Dashboard](..vortex-data/issues/357) for more information. 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODAuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE4NC4xIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Brings in list_transform (vortex-data#10503) and 39 other upstream commits. Signed-off-by: Christopher Wirt <cwirt@quantcast.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges the 40 commits on upstream
developthat we're missing, includinglist_transform(vortex-data#10503).develophas 5 fork-only commits, and the ruleset blocks pushing a merge todevelopdirectly.VortexTable.java, and git auto-merged it.🤖 Generated with Claude Code