Skip to content

Sync develop with vortex-data/vortex develop - #15

Closed
ChristopherWirt wants to merge 41 commits into
developfrom
cwirt/sync-upstream-develop
Closed

ChristopherWirt wants to merge 41 commits into
developfrom
cwirt/sync-upstream-develop

Conversation

@ChristopherWirt

Copy link
Copy Markdown
Collaborator

Merges the 40 commits on upstream develop that we're missing, including list_transform (vortex-data#10503).

  • GitHub's Sync fork button can't do this. Our develop has 5 fork-only commits, and the ruleset blocks pushing a merge to develop directly.
  • The merge applied cleanly. The only file both sides changed is VortexTable.java, and git auto-merged it.
  • Merge with a merge commit, not squash, so later syncs keep their upstream history.

🤖 Generated with Claude Code

myrrc and others added 30 commits October 26, 2026 12:25
This is needed for adding a dynamic registry for partitioning rules

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
…vortex-data#10796)

## Summary

`codspeed` 6.0.0 (vortex-data#10784, vortex-data#10785) adds process isolation, which is on by
default. `codspeed-divan-compat` now runs each benchmark in a new copy
of the benchmark binary. Simulation mode measures one iteration without
warmup, so each benchmark also pays for allocator setup (mimalloc) and
for simulated cache misses on memory it touches for the first time.
Before, an earlier benchmark in the same process had already paid these
costs.

The CodSpeed report on vortex-data#10785 shows a near-constant ~15 µs added to
short benchmarks.
Instead of error being an enum we convert an error to be a struct with
message and error kind. I had followed the structure of Python errors
which have couple error categories
https://docs.python.org/3/builtins/exceptions.html#exception-hierarchy
majority of which don't apply to this project

Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

- Tracking Issue: vortex-data#10701

Adds compat fixtures for the old decimal forms that current writers
still produce: decimals stored narrower than the canonical width of
their precision (including values that need the next precision, and high
scales), decimals stored wider, nonzero values in null slots, and
DecimalByteParts with narrow and wide most significant parts. A release
must publish these files before vortex-data#10701 changes how old decimal arrays
read, so that the new read path is checked against real old files.

## Changes

A flat fixture can now override `build_written()` to write a physical
form that `build()` must not return. `generate` and `publish` write
`build_written()`, and `check` compares each published file with
`build()`, which returns canonical arrays only. So the published file
keeps the old form while the expected array never changes. The overrides
work only while `DecimalArray::try_new` accepts a non-canonical width.
Once it rejects one, the overrides must be deleted, and the files
published before that keep the old forms. `DESIGN.md` documents the
hook.

The new tests check that every decimal in `build()` has the canonical
width and that every written column keeps its stored width through a
file round trip. That round-trip check reads with the current reader, so
the change in vortex-data#10701 that converts old decimal arrays on read must
update it. Until a release publishes the new fixtures, `compat.py check
--mode last` skips them.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary

- Tracking Issue: vortex-data#9128
- Independent validity candidate: vortex-data#10742

Routes float32 comparisons through the existing word-packing collectors.
This recovers the measured float32 gains from the RowFn comparison
investigation with two private dispatch changes. It is independent of
the comparison migration and fusion work in vortex-data#10676 and vortex-data#10675.

## Changes

Adds `F32` to the existing collector selections for array pairs and
array/constant comparisons. Integer collector policies remain unchanged.
The floating-point documentation now accounts for NaN signs and
payloads.

**Draft pending fresh isolated benchmarks on this head.** Saved
measurements show a consistent float32 gain, but also some unrelated ARM
regressions. The later combined packing/validity candidate had an AVX2
regression that has not been attributed to either change individually.

<details>
<summary>Saved benchmark evidence and validation</summary>

The isolated packing snapshot `4c98df49e379853f8e990d9764791b5330511771`
was compared with frozen `develop` at
`a1693389301e446eec41373cb274d98a303d3ca0`. Measurements used Rust
1.99.1 / LLVM 23.1.1, 16 CGUs, no LTO, CPU 2, and 21 balanced paired
blocks on Graviton3 C7g and Sapphire Rapids C7i. This was native wall
time, not CodSpeed simulation.

Across 48 float32 cases at 8,192 and 65,536 rows, covering array pairs,
both constant orientations, equality/ordering, and four validity shapes,
the median block ratios of equally weighted geometric means were:

| Target | Candidate / baseline time | 95% paired bootstrap interval |
| --- | ---: | ---: |
| NEON | 0.611 | 0.610–0.611 |
| AVX2 | 0.632 | 0.630–0.633 |
| AVX-512 | 0.679 | 0.678–0.681 |

Among all 375 probe cases, the isolated snapshot had no flagged
regressions on AVX2 or AVX-512. ARM had four, including roughly 4.7%
slower u64 multiply and 4.6% slower large f64 equality. Those paths do
not use the changed float32 collector, so their movement is not evidence
of extra float32 work, but it still counts against a no-regression
claim.

A separate combined packing/validity candidate at
`8d8ef8151454bbc73e1cd0b236567542a1401cbe`, based on
`d834a11faa79370b4d2e0bbcc73195f451a8a212`, retained the float32 gain
but regressed large nullable u64 add by about 18% on AVX2. Profiling
found a byte-identical retry hot loop with essentially unchanged
instructions per operation and more cycles. The precise
placement-sensitive mechanism remains unresolved. That combined result
does not isolate this PR.

The combined candidate passed 225 RowFn tests and 232 binary-function
tests on that newer base. Separate native probes checked
scalar-reference results and float/encoding/tail edge cases. The changed
file passed the pinned formatter. These are saved checks, not fresh
validation of this individual PR.

This PR is rebased onto `c750fbd80882f78efd6a2703bc0f3e17f1aca9f9`. The
patch is unchanged by the rebase. Tests and benchmarks were not rerun
after it.

</details>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`ArrayRef::as_constant`, `Array::as_constant` and
`ReduceNode::as_constant` now return `Option<&Scalar>` instead of
cloning the constant's scalar. Let the callers decide if they need an
owned scalar
## Summary

- Closes vortex-data#10750
- Replaces vortex-data#10746

The docs on `Validity::cast_nullability` read as if every `CastReduce`
rule that removes nullability must call
`Validity::trivially_cast_nullability` and pair with a `CastKernel`.
Most push-down rules (FoR, ALP, ALP-RD, ZigZag, RLE, RunEnd,
DateTimeParts, DecimalByteParts, Chunked, Sparse, Extension) instead
cast the child that holds their validity, and the cast of that child
already fails on nulls, so no result is wrong. Guarding each rule, as
vortex-data#10746 does for DecimalByteParts, adds a kernel per encoding and delays
the rewrite until execution without changing any result.

The guard in vortex-data#10746 comes from the closed vortex-data#10230, where a canonical
decimal held a lazy child that a cast to canonical did not execute, by
way of the FX2 notes in vortex-data#10633. `develop` keeps one buffer per decimal,
so that reason no longer applies.

## Changes

`CastReduce` now owns the contract and documents both forms of rule. A
rule that builds a new `Validity` keeps the guard and the kernel, and a
rule that pushes the cast into its children needs neither when the nulls
of those children are exactly the null rows of the array. `Dict` is the
documented exception, because its values can hold nulls that no code
references. No new tests are needed: every push-down encoding listed
above already runs `test_cast_conformance` on a partially valid array,
and that suite asserts that the cast to non-nullable fails.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01V1YHcosAAKufkyKPorUC2n)_
## Summary

Split out of vortex-data#10761. This keeps only the
`vortex-fastlanes` cast change; the bit-packed Arrow dictionary export
from that PR is dropped.

`BitPacked`'s `CastKernel` only took its direct unpack path for widening
integer casts. Bit-packing rejects negative integers, so a patch-free
array packed at a global `bit_width` holds only values in
`0..2^bit_width`. A narrowing cast whose target holds every such value
cannot overflow.

One place this applies is the Arrow export of Vortex dictionaries.
`dict_to_dict` exports the codes through `to_arrow_primitive`, which
casts them to the requested key type. Dictionary codes are often
bit-packed, so a key type narrower than the codes' ptype hits this
kernel.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary

Fixes vortex-data#10623.

`Dict::fill_null` converted the index of the matching dictionary value
into the codes type with an unchecked `as` cast. A dictionary can hold
more values than its codes type can index, so an unreferenced fill value
could truncate to the wrong code (with `u8` codes, filling with value
300 returned value 44) or panic for signed codes.

## Changes

The kernel now checks the index against the range of the codes type.
When the index does not fit, the kernel declines and the generic
`fill_null` path fills the decoded array, which is slower but correct.
Non-nullable codes need no fill code, so they keep the fast path. The
new tests cover `u8` and `i8` codes at and past their maximum, and
non-nullable codes over nullable values. The three cases past the
maximum fail without the fix.

The `Narrow` work (vortex-data#10733) depends on this fix, because a rule there
makes narrow dictionary codes common.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary

- Tracking Issue: vortex-data#9128
- Supersedes vortex-data#10674
- Related discussion: vortex-data#9694
- Independent float32 candidate: vortex-data#10741

Attaches input validity directly to canonical RowFn output using the
existing helper. Preserves kernel-output validation, relabeling, outer
nullability, and the non-canonical mask fallback.

`values.mask(valid)` builds and optimizes a `ScalarFnArray` to dispatch
through parent-reduce rules. The benchmark-driven bypass stays inside
RowFn to avoid adding canonical-specific dispatch to the generic
builtin. It can be removed if `.mask()` gains equivalent direct
dispatch. This revisits the abstraction concern in vortex-data#9694.

**Draft pending fresh isolated benchmarks on this head.** Saved results
include regressions and do not establish a general speedup.

<details>
<summary>Saved benchmark evidence and validation</summary>

The isolated validity snapshot
`b1961452ea3505d4e2588e34b14c20ea177ed175` was compared with frozen
`develop` at `a1693389301e446eec41373cb274d98a303d3ca0`. Measurements
used Rust 1.99.1 / LLVM 23.1.1, 16 CGUs, no LTO, CPU 2, and 21 balanced
paired blocks on Graviton3 C7g and Sapphire Rapids C7i. This was native
wall time, not CodSpeed simulation.

For nullable u64 addition with 64 rows and a right-hand constant, median
paired time ratios were:

| Target | Candidate / baseline time | 95% paired bootstrap interval |
| --- | ---: | ---: |
| NEON | 0.798 | 0.796–0.802 |
| AVX2 | 0.850 | 0.846–0.852 |
| AVX-512 | 0.846 | 0.844–0.855 |

The isolated snapshot also had flagged regressions outside that case:
roughly 3.4% slower ARM u64 multiply and up to 4.2% slower AVX2 byte
comparisons. No regressions crossed the 3% flag on AVX-512 in that run.
These movements remain part of the review, even where unchanged code is
sensitive to compiler context or binary layout.

A separate combined packing/validity candidate at
`8d8ef8151454bbc73e1cd0b236567542a1401cbe`, based on
`d834a11faa79370b4d2e0bbcc73195f451a8a212`, retained the small-batch
gain but regressed nullable u64 add by about 12% at 8,192 rows and 18%
at 32,768 rows on AVX2. Profiling found a byte-identical retry hot loop
with essentially unchanged instructions per operation and more cycles.
The precise placement-sensitive mechanism remains unresolved. That
combined result does not isolate this PR.

The combined candidate passed 225 RowFn tests and 232 binary-function
tests on that newer base, including output validation, allocator, and
deferred-error coverage. Separate native probes checked null-row
arithmetic errors and comparison edge cases. The changed file passed the
pinned formatter. These are saved checks, not fresh validation of this
individual PR.

This PR is rebased onto `c750fbd80882f78efd6a2703bc0f3e17f1aca9f9`. The
patch is unchanged by the rebase. Tests and benchmarks were not rerun
after it.

</details>

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…0792)

## Summary

- Tracking Issue: vortex-data#10791

On Spark 4.2 every read through `VortexTable` fails with a
`NoSuchMethodError`: Spark 4.2 changed
`CatalogV2Util.structTypeToV2Columns` to take an extra `boolean`.

This is a stopgap. vortex-data#9658 rewrites the connector as a `FileFormat`,
deletes `VortexTable` and adds Spark 4.2 to its compat suite. Once it
merges, this change goes with it. Until then, it keeps the current
connector usable on Spark 4.2.

## Changes

- `VortexTable` builds its columns with `Column.create(name, type,
nullable, comment, metadataJson)` instead of
`CatalogV2Util.structTypeToV2Columns`. The method has the same signature
in Spark 3.5, 4.1 and 4.2, so one jar still serves all three.
- `-PsparkTestVersion` runs the `vortex-spark` tests against a newer
Spark than the jar compiles against. The jar still compiles against
4.1.2.
- New `VortexTableTest` case: the scan's read schema keeps nullability,
comments and metadata.

Tested with:

- `./gradlew :vortex-spark_2.13:test`: 211 passed on Spark 4.1.2
- `./gradlew :vortex-spark_2.13:test -PsparkTestVersion=4.2.0`: 211
passed on Spark 4.2.0

## AI assistance

Disclosed per the [AI
policy](https://docs.vortex.dev/project/contributing#ai-assistance): the
investigation, code and tests were drafted with Claude (Claude Code) and
reviewed by me.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
…a#10808)

## Summary

CodSpeed reported a regression on 49 of 68 recent PRs that changed no
benchmarked Rust code. On the walltime legs it compares the fastest
sample, and divan runs most benchmarks there with one iteration per
sample, so one fast iteration out of 1000 can decide the result. Expect
a one-time change in the two pinned series on this PR's own report.

## Changes

A fixed `sample_size` makes each sample the average of several
iterations and stops divan from picking a different sample size per run.
Those were the two causes behind `bitpack_blocked_compress` (flagged on
148 PRs) and `words_gather_dispatch` (58). `bitpacking_filter.rs` gets
back the `mimalloc` allocator that a rewrite dropped, and the guide now
makes `mimalloc` mandatory and lists what not to do, so the same flakes
stop coming back.

The NEON `threshold`/`filter`, `mul_u64_nonnull` and
`starts_with[VarBin]` benchmarks also switch between two states for a
whole run, but the cause is not known and a larger iteration does not
help. The guide lists them as known unstable instead of changing them.

<details>
<summary>Evidence and checks</summary>

- Sources: the CodSpeed comments on the last 300 PRs and the divan
tables from 100 CodSpeed runs.
- The values in CodSpeed's comments match divan's `fastest` column in
108 of 122 checkable cases, and the median in 9.
- `bitpack_blocked_compress` (AVX2): the median stayed within 1% over
100 runs. The fastest iteration was near 6.7 µs in 29 runs and near 7.55
µs in the rest.
- `words_gather_dispatch` (NEON): divan picked one iteration per sample
in 42 runs (1.98 to 2.01 µs) and two in 57 runs (1.80 to 1.82 µs).

Checks run:

- `cargo clippy --locked -p vortex-buffer -p vortex-fastlanes --benches
--features vortex-fastlanes/_test-harness -- -D warnings`
- `RUSTFLAGS="--cfg codspeed -D warnings" cargo check` on the same
targets. The touched files are clean. `canonicalize_bench.rs`
(untouched) has unused-import errors under this configuration.
- `cargo +nightly-2026-09-10 fmt --check`, `typos`, `git diff --check`
- Built both benchmarks and confirmed that divan runs 8 and 16
iterations per sample.

Not run: the CodSpeed jobs (they run on this PR) and the Sphinx docs
build.

</details>

https://claude.ai/code/session_01KF3pMFx7U5ngk8BzQQFS2W

---
_Generated by [Claude Code](https://claude.ai/code)_

---------

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
…x-data#10752)

## Summary

- Tracking Issue: vortex-data#10701

Adds benchmarks for the decimal paths that vortex-data#10701 changes, so CodSpeed
has a base before those changes land. They cover compute on
DecimalByteParts arrays whose most significant part is narrower than the
canonical width (decode, compare, filter, sum and min-max), a compressed
DecimalByteParts tree in the decompress throughput bench, casts that
grow the canonical width, and decimal export to Arrow and DuckDB.

## Changes

The DecimalByteParts compute, decompress and cast benches run in
CodSpeed. The Arrow and DuckDB export benches run locally only, because
those crates have no CodSpeed shard, and the decimal compress bench
stays inside the existing `cfg(not(codspeed))` module. The compress
bench enables the default `core` edition, so the decimal scheme uses its
v1 variant as in a default write. `vortex-duckdb` gains a
`#[doc(hidden)]` `_benchmarking` module that re-exports `ArrayExporter`
and `ConversionCache` for its bench, and the fields of `ConversionCache`
become `pub(crate)` so that the module exposes only the type. No library
behavior changes. Every input is a canonical decimal or a
DecimalByteParts array, so the benches stay valid once every decimal
array must have the canonical width.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…10735)

The scaled-down integer to decimal cast selects its carrier as
`max(carrier(S), T)`. The source type `S` and the target type `T` fix
the carrier, but `match_each_decimal_value_type!` compiled the kernel
for all six decimal widths in each of the 48 `(S, T)` pairs. An `if
const` condition now removes the five arms that cannot match before code
generation. The arm that runs is the same as before.

`primitive_decimal_carrier_type` becomes a `const fn` so that the
condition can use it.

| | before | after |
|---|---|---|
| `vortex-array` release `__text` (macOS arm64) | 14,012,988 B |
13,697,852 B (-315 KB) |
| `vortex-array` LLVM IR lines | 5,405,422 | 5,200,767 |

Signed-off-by: Robert Kruszewski <github@robertk.io>
…#10736)

The filter take kernel for fixed-width values copies values and does not
inspect them, but it was compiled for each of the 17 primitive and
decimal value types, for each of the 8 index types. `take_primitive` and
`take_decimal` now reinterpret the values buffer by its byte width, so
six kernels (1, 2, 4, 8, 16 and 32 bytes) serve all value types. The
reinterpretation does not copy data, and the per-element loop is the
same.

| | before | after |
|---|---|---|
| `vortex-array` release `__text` (macOS arm64) | 14,012,988 B |
13,808,764 B (-204 KB) |

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The generic constructors (`ConstantArray::new<S>`,
`PrimitiveArray::{new, try_new, new_unchecked, empty}<T>`,
`DecimalArray::{try_new, new_unchecked}<T>`, and the `impl IntoIterator`
constructors of `StructArray` and `ChunkedArray`) called the generic
`Array::from_parts_unchecked` and `Array::try_from_parts` directly. Each
crate that called them monomorphized its own copy of those functions and
of the full `DynArrayData` vtable for the encoding.

The generic constructors now call a private, non-generic builder.
`vortex-array` compiles each vtable once, and other crates call it.
`ScalarFn` had the same problem because `new_from_validated_slots` was
`#[inline]`, and it is not inlined now. The cost is one function call
when an array is constructed.

| `vortex-ffi` release cdylib (macOS arm64) | before | after |
|---|---|---|
| `__text` | 37,223,740 B | 36,613,308 B (-610 KB) |
| Copies of the vtable: Constant / Primitive / Struct / Chunked /
Decimal / ScalarFn | 22 / 17 / 7 / 5 / 5 / 4 | 1 / 1 / 1 / 1 / 1 / 1 |

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tex-data#10739)

**Negated operators share kernels.** `NotEq`, `Gte` and `Lte` are the
inverse of `Eq`, `Lt` and `Gt`, so each pair now uses one set of
kernels. When the operator is a negation, `stream_compare_fused` inverts
each 1024-bit output block right after the transpose, while the block is
still in cache. This adds 16 word NOTs for each 1024 values. `between`
passes `negate = false`.

**Signed types use the unsigned kernels.** Bit-packing rejects negative
integers (`ensure_non_negative_integers`), so packed values of a signed
type are never negative.
- A negative constant compares the same against every packed value as
against zero, so the result is constant. It is handled before the fused
kernel.
- Otherwise both sides are non-negative, and the unsigned physical
values compare the same as the signed values. The fused kernel now takes
function items of the physical type (`<T as
PhysicalPType>::Physical::is_lt` and so on), so signed and unsigned
arrays of one width share one set of kernels.

Patches and width 0 still use the exact predicate on the logical type.

| `vortex-fastlanes` release `__text` (macOS arm64) | bytes |
|---|---|
| develop | 4,442,156 |
| negated operators share kernels | 3,279,596 |
| + signed types use unsigned kernels | 2,846,252 (-1.60 MB, -36%) |

---------

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The list take by `PiecewiseSequence` compiled its gather kernels for
every combination of piece start, piece length, list offset and output
offset type (4 x 4 x 4 x 4). The kernels read each piece start and
length only once per piece. The inner loop over list offsets depends
only on the offset types. The take now widens the starts and lengths to
`u64` and dispatches only on the offset types.

The widening does not copy when the pieces are already `u64`, which is
the type that the list take itself produces. For other widths it copies
the piece arrays once. That copy is proportional to the number of
pieces, not to the number of elements.

| | before | after |
|---|---|---|
| `vortex-array` release `__text` (macOS arm64) | 14,012,988 B |
12,798,204 B (-1.21 MB) |

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary

- Tracking Issue: vortex-data#10804

Follow-up to vortex-data#10745, as suggested in review there. When the fill value
already exists in the dictionary values but its index does not fit the
codes type (e.g. `u8` codes and the fill value at index 300), the `Dict`
`fill_null` kernel returned `None`, so the generic path decoded the
whole array before filling. The kernel only rewrites the null codes, so
it can widen them instead and keep the result dictionary-encoded.

## Changes

- For nullable codes whose type cannot hold the fill index, cast the
codes to `PType::min_unsigned_ptype_for_value(fill_index)` (kept
nullable) and then fill the nulls with the index. Codes are never
negative, so the cast keeps every code. Codes that already fit and
non-nullable codes take the same path as before.
- New test `fill_unreferenced_value_widens_codes` calls the kernel
directly and checks that it returns a `Dict` with widened, non-nullable
codes and the right values, for `u8 -> u16` (256, 300), `i8 -> u8`
(128), `i8 -> u16` (256) and `u16 -> u32` (65536). All five cases fail
on `develop`, where the kernel returns `None`.

Tested with `cargo test -p vortex-array --lib` (4437 passed), `cargo
clippy -p vortex-array --all-targets --all-features -- -D warnings`, and
`cargo +nightly-2026-09-10 fmt -p vortex-array`.

AI assistance: I used an AI coding assistant to help write this change
and the description; I reviewed the code and ran the tests above myself.

Signed-off-by: Vincent <1729065730@qq.com>
vortex-data#10810 updated `cc` from 1.8.0 to 1.10.0. After that, `Rust tests
(windows-x64)` fails in the `libmimalloc-sys` build script:

```text
c1xx: fatal error C1083: Cannot open source file: '/WX-': No such file or directory
error: failed to run custom build command for `libmimalloc-sys v0.1.49`
```

Cause:

1. Starting with `cc` 1.9.0, `cc` runs `cl.exe` through the
`RUSTC_WRAPPER` fallback. CI sets `RUSTC_WRAPPER=sccache`. `cc` 1.8.0
runs `cl.exe` directly.
2. For `.cc` sources, `cc` passes `-Tp <file>` as two separate
arguments. sccache 0.18.0 does not know `-Tp`, so it treats it as an
unknown flag.
3. The sccache preprocess command adds `/WX-` before the input file.
`cl.exe` then reads `/WX-` as the `-Tp` source file.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rtex-data#10744)

## Summary

In DataFusion, `a NOT IN (1, 2, 3, 4, 5)` counted null rows that sit in
an all-null chunk.

Data: `id` Int32 0..999. `a` Int32 nullable, null for rows 0..499 and
`id % 20` for rows 500..999. Written as ten 100-row chunks
(`WriteStrategyBuilder` with `with_row_block_size(100)` and
`with_data_block_target_bytes(None)`), so rows 0..499 are five all-null
chunks.

```sql
SELECT count(*) FROM t WHERE a NOT IN (1, 2, 3, 4, 5)
```

A DataFusion `MemTable` with the same data returns 375. On develop
Vortex returns 875, so it counts all 500 null rows. With this change it
returns 375.

vortex-datafusion turns a NOT IN of consecutive integers into
`not(between(...))` (`in_list_of` in `vortex-array/src/expr/exprs.rs`)
and reports the filter as pushed, so DataFusion does not check it again.
Each all-null chunk is a `ConstantArray`. The `Constant` `BetweenReduce`
rule compared it with the constant bounds using `Scalar`'s ordering,
which puts null below every value. So `between` returned `false` instead
of null, and `not` turned that into `true`. In normal dispatch null
bounds are handled earlier by `short_circuit`, so a null array value is
the case that got through.

## Changes

`short_circuit`, which runs before every encoding's `between`, now
returns a null `Bool` constant when the array is a null constant, with
the combined nullability of the array and both bounds.

The new test `scalar_fn::fns::between::tests::constant_null_array`
checks that `between` and `not(between)` are all null for `f64` and
`i32`.

Checked locally: `cargo test -p vortex-array --lib between` (46 passed,
and the new test fails on develop), the DataFusion query above (875 on
develop, 375 with this change), clippy with `-D warnings`, and nightly
fmt.

I used AI tools to help write and review this change.

---------

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Third of a stack of three that lands the planner-driven V2 scan for
DataFusion behind a switch, cut from vortex-data#10694. Stacked on vortex-data#10707 (the
planner chain), itself on vortex-data#10706. With this merged, `scan_v2` runs TPC-H
end to end; the perf and diagnostics commits of vortex-data#10694 follow as
separate PRs on top.

## Measured on its own

Against the V1 scan, locally (14 cores, interleaved A/B/B/A, 10 and 5
iterations a run) at the same code before the stack was cut
(da8a7c8):

| query | SF1 | SF10 |
|---|---|---|
| q1 | 1.074 | 1.024 |
| q6 | 0.966 | 0.832 |
| q10 | 1.025 | 0.882 |
| q12 | 0.939 | 0.811 |
| q13 | 1.640 | 1.125 |
| q19 | 0.996 | 0.909 |
| geomean | 1.085 | 0.924 |

SF10 improves over vortex-data#10707 (0.977 to 0.924) as the per-step task hops go;
SF1 does not yet, and q13 at SF1 is far behind because every partition
still plans the filter and projection itself (the plan-once and
interning commits of vortex-data#10694 fix that, taking the full branch to 0.963 at
SF1). This PR's base is a branch, so CI's benchmark dispatch does not
run on it.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…odings (vortex-data#10558)

UNSTABLE exists to aggregate all core encodings that are in the works
and should be tested and benchmarked but not written by default writer
## Summary

The writer’s unused copy of `DictLayoutMetadata` recommends prost’s
lenient enum getter.

## Changes

Remove the duplicate and its constructor. The active dictionary layout
metadata is unchanged.

## API Changes

Removes `vortex_layout::layouts::dict::writer::DictLayoutMetadata`.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This is still transitively picked up through arrow_cast and parquet
variant but the logic is simpler without it
A const condition removes the unreachable arms before codegen in three
places:

- `check_decimal_identical` dispatched on the lhs storage type, the rhs
storage type and their widest type. The two distinct inputs fix the
widest type, so only 30 of 216 arms can run.
- `widened_buffer::<W>` is only called for storage types narrower than
`W` (all callers pass the wider of two inputs), and the narrowing branch
of `converted_buffer::<W>` only runs for storage types wider than `W`.
Only 15 of 36 arms of each can run.
- The bit-packed widening cast compiled all 64 integer pairs, but only
widening pairs reach the kernel.

Bit-packing rejects negative values, so sign extension of a packed value
is the same as zero extension. The bit-packed cast now views a signed
source as unsigned (the packed buffer is shared and the patch values are
reinterpreted) and reinterprets the result to the target type. Signed
and unsigned casts of the same widths share one kernel, so the cast
compiles 6 kernels instead of 18 reachable ones.

| release `__text` (macOS arm64) | before | after |
|---|---|---|
| `vortex-array` | 14,023,548 B | 13,933,616 B (-90 KB) |
| `vortex-fastlanes` | 4,442,156 B | 4,331,764 B (-110 KB) |

Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

- Tracking Issue: vortex-data#10701

Prost decodes an enum value it does not know as the first variant, so a
stored `DecimalType` or `PType` outside its enum silently reads as `I8`
or `U8`. The reader then decodes the buffer or child at the wrong width,
which either fails later with an unrelated error or returns wrong
values. Readers need to reject such values before a later release adds a
variant, such as a wider integer `PType`. The read path for old decimal
arrays in vortex-data#10701 also depends on the stored `values_type`, because it
compares that width with the canonical width.

## Changes

Each metadata struct with one of these fields gets a `get_<field>()`
accessor that returns an error naming the field and the raw value, and
`deserialize` calls it before reading any buffer or child. The protobuf
dtype decoder does the same through `pb::PType::try_from`. Files from
current writers read as before, and `compat.py check --mode all` passes
against every published fixture.

The handwritten enum fields use `prost(int32)`, so prost no longer
generates lenient enum getters or setters. The protobuf tags and encoded
bytes stay the same.

For review, read the commits one at a time.

## API Changes

Removes the generated enum getters and setters from the affected
handwritten metadata structs. Decoding uses the strict `get_<field>()`
methods.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
danking and others added 11 commits October 26, 2026 19:20
Reject raw serialization of sliced PCO arrays because the format does
not encode slice offsets. This prevents writing data that cannot
reconstruct the original slice; callers can materialize the slice before
serialization.

Signed-off-by: Daniel King <dan@spiraldb.com>
Depends on vortex-data#10755.

## Summary

Public metadata fields expose the stored layout as Rust API and let
callers bypass typed accessors. Restrict them to the owning crate while
preserving the encoded metadata.

## Changes

Add getters for CUDA’s Zstd metadata and a constructor and CRS getter
for spatial callers. The TUI uses the existing flat-layout accessor.
Internal struct literals and metadata golden files stay unchanged.

Formatting and clippy passed. Tests passed apart from
`chunked_input_with_mixed_dimensions_returns_error`, which fails
identically on the base. GPU tests were skipped without `nvcc`.

## API Changes

Metadata fields are now `pub(crate)`. Direct field access and struct
literals from other crates no longer compile.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The v2 VortexDataSource reports incorrect statistics when there are
filter expressions or limits.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Allow passing a custom expression converter into the V2 DataFusion
VortexDataSource

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…rtex-data#10062)

Ordered scans must emit partitions in source order even when later
partitions are ready first. Concatenate ordered partition streams while
preserving bounded unordered concurrency and the existing global fetch
trimming.

Closes vortex-data#10056.

Deterministic regressions cover complete output, fetch zero, the first
row, and a fetch crossing partition/batch boundaries. Three cases fail
with unordered flattening; all four pass with the fix. The combined v2
module passes 17 tests; scoped Clippy and pinned formatting pass.

AI assistance: Claude Code and Codex, under @gatesn's direction.

---------

Signed-off-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Gates <gatesn@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nicholas Gates <nick@nickgates.com>
Returns arbitrary aggregate function partials for a given field path.
This allows callers to probe for any statistics.

To consider:
* Exposing this as a batch endpoint? (Vec<AggregateFnRef> ->
Vec<Precision<Scalar>>)
* Deprecate the field_statistics function

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
…ta#10743)

## Summary

Adds `ListArray::compact` and a list view rebuild mode,
`ListViewRebuildMode::MakeExactWithEmptyNulls`. Both return the same
lists over only the elements that valid lists reference: nothing before
the first list, after the last, between lists or under a null list.

A list column can hold elements that no valid list shows. Slicing keeps
the parent's elements on both sides, and a null list may still cover
elements. A function that only reads values does not care, but a
function that can fail on an element, such as a checked `x + 1`, must
not see them: an overflow in an element under a null list would fail the
whole batch. `list_transform` (vortex-data#10503, stacked on this PR) is the first
user; it applies its lambda to the rebuilt elements whenever the lambda
can fail.

For example:

```text
elements  [9, 1, 2, 9, 9, 3]
offsets   [1, 3, 5, 6]
validity  [valid, null, valid]
lists     [1, 2], null over [9, 9], [3]

rebuilt
elements  [1, 2, 3]
offsets   [0, 2, 2, 3]
```

## How it works

**`ListArray`**

- Non-nullable or all-valid lists with nothing at the ends are returned
as they are, after reading two offsets.
- Otherwise it looks for null lists that hold elements: one pass marks
the lists whose next offset is larger, and one bit operation clears the
valid ones. Only when bits are left does it collect them into runs.
- With no such run, only the ends go: the elements are sliced to `[first
offset, last offset)` and the offsets are shifted down, without copying
an element.
- With some, the elements between those runs are kept, by a slice if one
range is left and otherwise by one filter whose mask carries the ranges,
so it copies range by range. The offsets are copied once and each one
moves down by the elements dropped before it, so the null lists in those
runs become empty.
- The offsets keep their type and the validity is unchanged.

**`ListViewRebuildMode::MakeExactWithEmptyNulls`**

- `MakeExact` only trims the ends of a view that is already zero-copy to
a list, so its null lists keep their elements. The new mode drops those
too.
- Such a view has its lists back to back, so it goes through the same
two steps on its offsets, with the last list's end as the final
boundary. It keeps its layout and its offsets' and sizes' types. Only
the sizes of the null lists that held elements are set to 0; otherwise
the sizes are kept as they are.
- Any other view is rebuilt with `MakeExact`, which already copies only
the valid lists' elements and leaves null lists empty. `MakeExact`'s
documentation now says both.
- The result is always zero-copy to a list.

## Benchmark

`list_transform` with a checked `x + 1` over 100k lists (~800k `i32`
elements, ~1/8 of the lists null), through vortex-data#10503, exported to Arrow,
median µs over three alternating runs of a local divan benchmark.
"Before" is vortex-data#10503 without this PR, which used the existing list view
rebuild instead.

| Lists | Before | This PR | DataFusion 54.1 `array_transform` |
|---|---|---|---|
| Null lists holding elements | 1,272 | 253 | 224 |
| Null lists already empty | 97 | 73 | 60 |
| Sliced, with elements on both sides | 78 | 81 | 55 |

## Testing

- `rstest` cases for both types: ends to drop, null lists holding
elements or not, every list null, no lists, an empty valid list between
null lists, overlapping and out-of-order views, views with and without
the zero-copy flag.
- Exact arrays come back as the same arrays (`ArrayRef::ptr_eq`), and a
sliced `ListArray` whose validity starts mid-byte is rebuilt correctly.
- A seeded random test compares `ListArray` and both kinds of view
against a reference that gathers each valid list's elements, over five
offset types, with non-nullable validity and validity arrays that start
mid-byte.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
## Summary

Moves the `vortex-runend` crate into `vortex-array` as
`vortex_array::arrays::runend`, the same way `vortex-dict` moved in
vortex-data#5289. Once RunEnd is built in, vortex-array code can build run-end
arrays itself. One example is spreading a captured outer value across
each list's elements in `list_transform` (vortex-data#10702).

## Changes

- `RunEnd` is now a built-in, like `Dict`. It is registered in
`ArraySession::default`, in the default kernel session, and in
`AggregateFnSession::default` (min_max, is_constant, is_sorted, sum).
- The `vortex-runend` crate is removed. `vortex::encodings::runend`
still re-exports the module.
- The benches move to `vortex-array/benches/run_end_*`. CodSpeed picks
them up in the array builders shard.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
## Summary

`list_transform(list, lambda)` applies a one-parameter lambda to every
element of a `List` (in any encoding), keeping each list's length and
validity.

```rust
list_transform(col("l"), Lambda::new(["x"], checked_add(var("x"), lit(1_i32)))?)
// vortex.list.transform($.l, (x) -> (@x + 1i32))
```

## Changes

- `ListTransform`'s options are `ListTransformOptions`: the lambda as
written (`Lambda`) in an expression, or bound to the element's dtype
(`BoundLambda`) in a bound expression or an array. Its only child is the
list.
- `bind_options` binds the lambda to the element's dtype when the node
is bound, so `execute` only runs it. A `FixedSizeList`, or a lambda with
anything but exactly one parameter, is rejected there and in
`return_dtype`. A lambda already bound to another element dtype is an
error.
- The result keeps the input's shape: a `ListArray` stays a `ListArray`
and anything else runs as a list view, with the same offsets (and sizes)
and validity around the new elements.
- A lambda that can fail runs only on the elements valid lists
reference. One that can't runs on the whole elements buffer, cut to the
lists' range for a `ListArray`.
- The function is strict (a null list gives a null result), it can fail
only if the lambda can, and it is not serializable yet.
## Benchmark

100k lists, ~800k `i32` elements, ~1/8 of the lists null where nullable,
result exported to Arrow, median µs, local scratch divan benchmark
against DataFusion 55.2's `array_transform` on the same lists:

| Lists | Lambda | DataFusion | This PR |
|---|---|---|---|
| Exact | `x + 1` | 49 | 71 |
| Null lists already empty | `x + 1` | 60 | 73 |
| Null lists holding elements | `x + 1` | 224 | 274 |
| Null lists holding elements | `x = 3` | 249 | 81 |
| Sliced | `x = 3` | 85 | 99 |
| List view, back to back | `x + 1` | 903 | 75 |

Most of the `x + 1` gap is Vortex's checked add against DataFusion's
wrapping add; with `x = 3` the two kernels are even. DataFusion casts a
list view to a list before transforming it.

---------

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](..vortex-data/issues/357) for more information.

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xODAuMCIsInVwZGF0ZWRJblZlciI6IjQ0LjE4NC4xIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Brings in list_transform (vortex-data#10503) and 39 other upstream commits.

Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants