Skip to content

Merge upstream develop into QC develop (2026-11-20) - #122

Open
ChristopherWirt wants to merge 130 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-18
Open

ChristopherWirt wants to merge 130 commits into
developfrom
cwirt/merge-upstream-develop-2026-11-18

Conversation

@ChristopherWirt

@ChristopherWirt ChristopherWirt commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Merge vortex-data/vortex:develop at 79e75c1d0a080136254734366de4ee7d761717ab into QC develop, preserving upstream ancestry. This brings in 128 upstream commits, in two merges:

  • 2026-11-18: 86 commits up to 8434405de, no conflicts.
  • 2026-11-20: 42 more commits up to 79e75c1d0, with the conflicts resolved below.

Changes

Conflict resolutions (2026-11-20)

  • zone_map.rs: kept the fork's large IN-list pruning tests. Upstream moved the other zone map tests to vortex-scan.
  • vortex-scan/src/plan/tests.rs: took upstream's version. Ported the fork's fallible ListLayout::try_new to the moved vortex-layout/src/lowering/tests.rs.
  • VortexSparkSession: took upstream's fix(spark): reject unusable write options on the driver vortex-data/vortex#11151, which replaces the fork's earlier version of that change. A provider class the driver can't load is now left for the executors to load.
  • VortexOutputWriterFactory: kept the fork's public validate, which also checks batch.bytes, capacity reset and listLayout.
  • VortexWriteOptionValidationTest: uses upstream's V1/V2 test structure with the fork's option cases. Out-of-range batch sizes are still rejected, as on the fork; upstream falls back to the default instead.

Testing

  • Ran pinned nightly-2026-11-13 rustfmt on vortex-layout/vortex-scan and spotlessApply on the Spark module. Neither produced changes.
  • Tests, builds and lint were not run locally.
  • Both merge commits are signed off. Imported upstream history is preserved.

🤖 Generated with Claude Code

0ax1 and others added 30 commits November 13, 2026 11:03
…ortex-data#11421)

## Summary

First of two stacked drafts: expose dictionary-preserving GPU batches so
consumers can postpone string expansion until after combining batches.
This PR adds the export API only; the companion PR vortex-data#11422 adds an
optional cuDF materialization helper and reports the ~20% end-to-end
hot-read gain.

A fixed-schema Arrow device stream cannot describe batches whose
dictionary encodings or index widths differ. The new iterator returns an
independently owned schema and device array for each batch, using the
existing `DictionaryExport::Preserve` policy.

## Changes

- Add `vx_cuda_scan_path_device_batches`, `vx_cuda_device_batches_next`,
and `vx_cuda_device_batches_free`.
- Reuse the existing scan options, ordered top-level projection, direct
I/O, and batch splitting.
- Keep the existing fixed-schema Arrow stream API and its decoded output
unchanged.
- Document ownership, stream synchronization, thread affinity,
end-of-stream, and terminal-error behavior; regenerate the C header.
- Add 10 regression tests for varying dictionary/index types, plain
batches, nulls, empty batches, projection, independent output lifetimes,
invalid inputs, and terminal errors.

---------

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
This PR adds benchmarks for the Delta encoding. They are split out of
vortex-data#11428 so that its changes can be measured against `develop`, and vortex-data#11428
stacks on this PR.

## Changes

`encodings/fastlanes/benches/delta_decode.rs` adds 15 benchmarks.

**Walltime (9 benchmarks).** `decode_bitpacked_deltas` decodes a 512 KiB
Delta array whose deltas are bit-packed, for u16, u32 and u64. It
carries `#[cpu_features]`, so it runs on each walltime CPU-feature leg,
where the build selects the transpose and decode kernels.

**Simulation (6 benchmarks).** These run once each, on 32 KiB arrays so
they stay under 1 ms in CodSpeed's simulation, which runs roughly 13×
slower than native:
- `decode_primitive_deltas` decodes a u32 Delta array whose deltas are a
primitive array.
- `probe_once` and `probe_repeated` read 8 rows from 2 chunks of a
2048-row array. `probe_once` uses a one-off probe per row;
`probe_repeated` reuses one probe for all 8.
- `filter` selects one row in every 64, or one in every 4096.
- `take` takes 128 scattered rows.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

Each change to an edition (adding, moving, or removing a member, or
freezing an edition) should go in its own PR and show up separately in
the release notes. This adds a dedicated `changelog/edition` label. The
label check already requires exactly one `changelog/*` label, so an
edition PR can't also be labelled as a feature or fix.

## Changes

- `.github/workflows/labels.yml`: accept `changelog/edition` as a
changelog label.
- `.github/release-drafter.yml`: add a "📜 Editions" release-notes
section after "Breaks", and make `changelog/edition` trigger a minor
version bump, the same as `changelog/feature` and `changelog/break`.
- `docs/specs/versioning/editions.md`: under "Maintaining edition
records", say that each edition change goes in its own PR labelled
`changelog/edition`.
- `.github/pull_request_template.md`: add a matching reminder next to
the `changelog/break` note.

The `changelog/edition` label still needs to be created in the
repository's label settings before it can be applied.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…vortex-data#11437)

## Summary

`IsSorted` orders nulls first. The non-strict branch of the bool kernel
(`check_bool_sorted`) broke that rule: it walked only the indices of
valid values, so it dropped nulls instead of ordering them first. The
strict branch already handled nulls correctly. It maps each element to
`Option`, where `None < Some(_)`, which is what the primitive kernel
does in both modes.

Two counterexamples showed the bug before this fix:

1. `BoolArray [false, null, true]` reported sorted, but the equivalent
`PrimitiveArray [0, null, 1]` reported not sorted.
2. `BoolArray [true, null]` reported sorted, but the same values split
into the `ChunkedArray` chunks `[true]` and `[null]` reported not
sorted. The accumulator's check at the chunk boundary already sees a
valid value followed by a null.
## Summary

The SQL benchmarks write Vortex files that we never look inside. This PR
adds one artifact per run containing `vx` file stats and layout trees
for a sample of each dataset's files, so you can see which encodings and
layouts a PR produced on real data.

The full trees are too large for logs: hundreds of thousands of lines
for ClickBench and the SF100 datasets. So every file is dumped on the
runner, but only a sample within a line budget is kept, and the kept
dumps from all jobs are merged into one zip retained for 5 days.
…#11441)

## Summary

`fastlanes.delta` is still only in the unstable edition. This PR fixes
its serialized contract before it is frozen. A review of Delta's serde
found two problems:

- **Validity is inferred from the child count.** A third child is always
read as validity, so the format can't add a child later without it being
mistaken for validity.
- **The reader is not robust to corrupt input.**
  - A float dtype panics in `lane_count` before `validate_parts` runs.
- `deltas_len` is not tied to `len`, so a tiny array with constant
children can claim an unbounded physical length and force a huge
allocation on decode.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…a#11443)

## Summary

The file-size section disappeared from SQL benchmark PR comments in
vortex-data#10987, which moved the baseline from S3 to the benchmarks Postgres.
File sizes were never stored there, so `compare-benchmark-jsons.py`
found no baseline rows and dropped the section without a message. This
PR stores the sizes on develop runs and reads them back on PR runs.
## Summary

vortex-data#11443 started storing SQL benchmark file sizes in the benchmarks
database, but only for `vortex-file-compressed` and `vortex-compact`.
This PR stores every on-disk format, so bench can chart Parquet, DuckDB,
Lance and the rest beside Vortex (vortex-data/vortex.dev#222). PR
comments still compare only the Vortex formats.

## Changes

- `scripts/capture-file-sizes.py` records every format directory that
`Format::name()` in `vortex-bench/src/lib.rs` names: `arrow-ipc`, `csv`,
`duckdb`, `lance`, `litdata`, `parquet`, `vortex-compact`,
`vortex-file-compressed` and `vortex-spatial-native`. Each develop
commit adds roughly one row per Parquet file; ClickBench has about 100.
- **DuckDB catalogs:** DuckDB keeps a catalog of views (`duckdb.db`, and
`duckdb.db.wal` while open) in every format directory it queries
(`duckdb-bench/src/lib.rs`). That isn't the format's data, so it's now
skipped everywhere except the `duckdb` directory, where the database is
the data. This also takes the catalog out of the Vortex totals. Rows
recorded today include it, so PR comments based on today's develop
commits list `duckdb.db` as a removed file, once.
- `scripts/compare-benchmark-jsons.py`: the "File Size Changes" section
compares only formats starting with `vortex`. The other formats' files
come from the data generators and don't change with a Vortex PR.

New tests check that capture records every format, skips the DuckDB
catalogs outside `duckdb`, and ignores unknown directories. Another
checks that the report leaves out non-Vortex formats.

### Checks

- `uv run --no-project --with pytest --with pandas --with numpy --with
orjson --with tabulate --with xxhash pytest
scripts/tests/test_measurement_id.py
scripts/tests/test_benchmark_reporting.py
scripts/tests/test_post_ingest.py`: 49 passed.
- `uvx ruff format --check` and `uvx ruff check` on the changed files
pass, as does `git diff --check`.
- Not run: a CI benchmark run. `action/bench-sql` on this PR would
exercise capture and the comment. The new formats only reach the
database from develop runs.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s)_

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary

- Set the stable toolchain in `rust-toolchain.toml` to `1.100.0`.
- Set `NIGHTLY_TOOLCHAIN` to `nightly-2026-11-13` in all 12 workflows
that pin it. The new rustfmt makes no changes to the code.

### Clippy 1.100 (`redundant_clone`)

- Remove redundant clones in `vortex-array`, `vortex-fastlanes`,
`vortex-fuzz`, `vortex-cuda`, `vortex-sqllogictest` and `xtask`.
- `vortex-duckdb`: `test_clone_logical_type` tests the FFI `Clone`. Keep
the clone and keep the original value alive, and check the type of both
values.

### Cargo 1.100 lints

- `unused_workspace_dependencies`:
  - Remove `arrow-arith`.
- Add `vortex-elias-fano` to `vortex`, re-export it as
`vortex::encodings::elias_fano` and register it in
`VortexSession::default()`. It is not in any file edition yet.
- `unused_dependencies`:
- `datafusion-bench/build.rs` checks `CARGO_CFG_TARGET_OS` and not
`#[cfg(target_os)]`, which reads the host OS. The link argument is for
the target.
- `vortex-web-wasm` puts all its dependencies under `cfg(target_arch =
"wasm32")`, because the library is empty on other targets.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
…ata#11184)

## Summary

- Tracking Issue: vortex-data#11114

Right now, an extension type validates whatever storage dtype the caller
hands it, so one type can have several storage forms, and its metadata
doesn't say which one a dtype uses. Each `ExtVTable` now declares its
storage from its metadata, so the metadata **deterministically fixes /
decides** the whole storage dtype except the top-level nullability.

Reads then require the stored storage to equal that declaration exactly,
**including nested nullability**. This has to be strict, since arrays
decode their storage child with the declared dtype, so a stored form
that differs would be misread instead of rejected. Every built-in type
accepts the same stored forms as before, so existing files still read.

_NB on nested nullability: this means that if we potentially need nulls
in the storage, an extension type MUST require its storage type to be
nullable, even if in some cases it might not be. I think this is a
worthwhile tradeoff for clarity of implementation._

## Changes

Some facts lived only in the storage dtype, like the element type of
`Vector` and `FixedShapeTensor` and the dimension of the native spatial
types, so they move into the in-memory `Metadata`.
`deserialize_metadata` decodes `StoredMetadata` (which is the old
`Metadata`, and `metadata_from_storage` combines it with storage to
rebuild complete metadata. The serialized bytes don't change.

Some other things:
- File reads and storage-based dtype, array, and scalar constructors
share the same checked path.
- `ForeignExtDType` can't derive its storage, since it doesn't know its
type, so it keeps the stored storage in its metadata and declares that.
- The nested-nullability checks that vortex-data#11183 added to the spatial helpers
go away, since the read check now covers them.

Follow-up: a read error in a struct field still panics on first access
(SF in vortex-data#11114).

## API Changes

- `ExtVTable::validate_dtype` becomes `storage_dtype` (plus an optional
`validate_metadata`), `deserialize_metadata` returns `StoredMetadata`,
and `metadata_from_storage` reconstructs complete metadata from stored
metadata and storage.
- `ExtDType::try_new` and `try_with_vtable` take a nullability in place
of the storage dtype.
- Reading an extension dtype whose stored storage differs from its
declaration fails, and so does building a scalar or array over the wrong
storage.
- `FixedShapeTensorMetadata::new` takes the element type, `Vector`'s
metadata is `VectorMetadata`, metadata-first construction of native
spatial dtypes needs `SpatialMetadata::with_dimension`, and
`SpatialMetadata` no longer implements `prost::Message`.
- `ExtDType` and `ExtensionArray` gain `try_from_storage` constructors,
and `Scalar` gains `try_extension_from_storage`, with explicit-vtable
variants. Existing metadata-first constructors remain available.
- `FixedShapeTensorStoredMetadata` supplies shape and layout, and
`SpatialStoredMetadata` supplies CRS when constructing from storage.
- Fallible constructors reject non-nullable null storage, including
extensions over null storage, with an error.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary

The benchmark jobs run every setup step one after another, even though
most of the steps don't depend on each other. This change groups the
independent steps into `parallel:` blocks, the same way `ci.yml` already
does, to shorten the time a benchmark machine spends before the timed
run. It also moves the Polar Signals debuginfo upload out of the build
jobs, so the bench jobs no longer wait for it.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ake and is_constant kernels (vortex-data#11428)

## Summary

FastLanes Delta decoded slowly and had almost no compute kernels: every
filter, take, `scalar_at` or `is_constant` decoded the whole array
first. Vortex now has the APIs this needs (the probe API and execute
kernels). This PR uses them, together with fastlanes 0.8.0's fused
unpack-and-undelta and SIMD transpose kernels, so Delta is cheap enough
to select by default.

## Changes

### Decode
- **Fused decode.** `Delta(BitPacked)` decodes each 1024-value chunk
with fastlanes' fused `unchecked_undelta_pack`. The deltas are no longer
unpacked into a full intermediate array. This covers patched and
block-width bit-packing.
- **Sliced arrays** decode only the chunks the slice covers, then trim
the result.
- **SIMD transposes.** fastlanes is updated to 0.8.0. Transposes are
dispatched at runtime through `fearless_simd`: AVX-512 for 8- and 16-bit
values; for 32- and 64-bit values, fastlanes' own dispatch, because
AVX-512 is slower there on Zen 4.
- `DeltaChunks` is a reusable chunk decoder with scratch space. The
kernels below share it.

### Kernels
- **Probe API.** `scalar_at` decodes only the chunk holding the row. A
repeated probe caches the last decoded chunk.
- **Filter and take** decode only the chunks that have selected rows.
Take falls back to a full decode for large index sets.
- **`is_constant`** reads only the stored parts where it can: all-null,
mixed-null, or all-zero interior deltas plus constant bases. Only the
partial chunks at either end of a slice are decoded.
- The chunked take logic in `take_chunks.rs` is shared between BitPacked
and Delta.

### Compression
- **Back-fill leading nulls per chunk.** `delta_compress` used to carry
the previous chunk's last value into a chunk that starts with nulls.
Values that restart in every chunk, such as RLE indices, then got
wrapping residuals, and the compressor rejected Delta for them. Those
leading nulls now take the chunk's first valid value.
- **Keep RunEnd off RLE indices.** RLE indices count up within each
chunk, so Delta packs them into a few bits. Allowing RunEnd there
produced `RLE(indices = RunEnd(ends = Delta))`, which every decode and
filter expands twice. This enables the exclusion that had been commented
out.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

C and C++ writers can now select a registered edition by name before
opening a writer. This lets them target older readers through the same
edition-aware writer used by Java and Python. The default file sink uses
the selected edition to build its write strategy and picks up the legacy
zone-map support from vortex-data#10805.

Codex assisted with the implementation, tests, and documentation.

## Changes

- Add `vx_session_enable_edition` to the C API. It returns a status code
and can report a `vx_error`.
- Add `vortex::Session::enable_edition(std::string_view)` to the C++
API. It reports failures through `vortex::VortexException`.
- Document how to select an edition before opening a C or C++ writer.
- `test_enable_edition` covers selecting an older core edition on a
shared session and reporting an unknown edition name.
- `test_sink_writes_legacy_zone_map_for_older_edition` checks that a
file written through the C sink with `core2025.10.0` has a
`vortex.stats` layout.
- The C++ session test covers a length-delimited edition name and the
exception for an unknown name.

---------

Signed-off-by: jinli.zjw <jinli.zjw@alibaba-inc.com>
## Summary

- Tracking Issue: vortex-data#10701

vortex-data#11458 and vortex-data#11459 change both the code and the tests of `array.rs`,
`assemble.rs`, and `split.rs`. This PR is a purely cosmetic change.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…data#11462)

## Summary

Every SQL benchmark that runs DuckDB fails with:

```
duckdb-bench: error while loading shared libraries: libduckdb.so: cannot open shared object file: No such file or directory
```

This started with the Rust 1.100 upgrade (vortex-data#11446). Rust 1.100's cargo
moved build script output from `build/<pkg>-<hash>/out` to
`build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB
cache two levels above `OUT_DIR`, so the library moved:

- before:
`target/release_debug/build/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so`
- now:
`target/release_debug/build/vortex-duckdb/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so`

The artifact path still names the old location. upload-artifact leaves
out an unmatched path without an error, so the bench jobs get
`duckdb-bench` without the library that its absolute rpath points to.

## Changes

`sql-bench-matrix.yml`, build job:
- The artifact path matches the library at any depth under
`target/release_debug/build`, so it works with both layouts. It also
matches a DuckDB built from source as well as a prebuilt one.
upload-artifact still roots the artifact at `target/release_debug`, so
the library unpacks at the path the rpath expects.
- A new step fails the build job if `libduckdb.so` is missing. A missing
library now stops the build, instead of letting every DuckDB benchmark
fail later.

A sturdier fix would be for `vortex-duckdb/build.rs` to stop deriving
its cache location from `OUT_DIR`, since cargo can change that layout
again. I kept this PR to the workflow.

Checks:
- `yamllint --strict -c .yamllint.yaml` passes.
- With cargo 1.100, a local build script prints
`OUT_DIR=.../target/debug/build/<pkg>/<hash>/out`.
- I ran the same fix, as a tar-based packaging step, on
vortex-data#11450. There FineWeb, TPC-H SF=1 and SF=10, TPC-DS
SF=1 and SF=10, ClickBench and TPC-H on S3 all finished with DuckDB
results. Before the fix they failed with the error above, on that PR and
on other branches.

---

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N)_

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary

A follow-up to vortex-data#11445. Step timings from recent runs showed CI jobs
outside the benchmarks still doing independent setup and checks one step
at a time. This groups those steps into `parallel:` blocks, splits the
Java job into a matrix, and speeds up the SQL benchmark binary handoff.
It also fixes the SQL benchmarks' DuckDB runs, which have been failing
since the Rust 1.100 upgrade.

## Changes

**SQL benchmarks (`sql-bench-matrix.yml`)**
- The build job packs the bench binaries into one tar with
multi-threaded zstd and uploads it with `compression-level: 0`, instead
of letting upload-artifact zip them with single-threaded zlib. Packing
and uploading took 12s, down from 64–90s. Each bench job then spends
about 4s unpacking, and every bench job starts 50–75s sooner. The tar
also keeps the executable bits, so the `chmod` step is gone.
- **Fix:** every DuckDB benchmark failed with `libduckdb.so: cannot open
shared object file`, on other branches too. Rust 1.100's cargo moved
build-script output from `build/<pkg>-<hash>/out` to
`build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB
cache two levels above `OUT_DIR`, so the library moved to
`build/vortex-duckdb/vortex-duckdb-cache/`, and the hard-coded upload
path matched nothing. upload-artifact skipped the missing file silently.
The build now searches `target/release_debug/build` for the library and
fails if it is missing, because `duckdb-bench` cannot run without it. It
loads the library through an absolute rpath, so the archive keeps its
relative path.

**Pre-merge CI (`ci.yml`)**
- Java: a matrix of four jobs. One leg runs `javadoc check` without the
Spark compat suites, and one leg runs each Spark compat suite (4.1, 4.2,
4.3). Gradle `--parallel` only splits work across projects, and all
three compat suites live in `:vortex-spark-4.0_2.13`, so they ran one
after another. A "Java" gate job keeps the required check name. It runs
with `always()` and fails if any leg failed or was cancelled, because a
skipped required check counts as passed.
- License check: `cargo deny check advisories` and `check bans licenses
sources` run at the same time. The advisory warning still runs after
them, and now has `!cancelled()` so it isn't skipped when both checks
fail.
- Python (test): after the main suite has built the extension and synced
the environment, the datasets-floor, Polars-floor and docs checks run at
the same time.
- Lightweight checks: the uv setup runs alongside the apt install.

**Other PR-path workflows**
- `docs-build.yml`:
  - Rust, JDK and uv install at the same time.
  - Javadoc (3m41s) builds alongside the Sphinx build (7m15s).
- The Javadoc is copied into `docs/_build/html/_static` after the Sphinx
build. The docs link to it with raw HTML, and Sphinx only copies
`_static` into the output, so the built site is the same.
- `publish-dry-runs.yml`, wheel build: setup-prebuild, Zig and uv
install at the same time. The Rust cache still waits for setup-prebuild,
because its key includes the rustc version.
- `web.yml`, `web-build.yml`: Rust, wasm-pack, Bun and Node install at
the same time.

**Fuzz, package and release workflows**
- `run-fuzzer.yml`, `minimize_fuzz_corpus_workflow.yml`:
`runs-on/action` and checkout run together, then building or downloading
the fuzzer runs alongside the corpus restore.
- `fuzz-coverage.yml`: the corpus download (python3 only) runs alongside
setup-prebuild.
- `close-fixed-fuzzer-issues.yml`: the LLVM setup runs with the
cargo-fuzz install. `setup-fuzz-linker` stays on its own, because both
it and the LLVM setup may run `apt-get`.
- `wasm-fuzz.yml`: the wasm32 build runs with the wabt install.
`setup-fuzz-linker` still comes after, because it sets mold `RUSTFLAGS`.
- `build-fuzzers.yml`, `package.yml`, `release-binaries.yml`,
`publish.yml`, `compat-gen-upload.yml`: `runs-on/action` and checkout
run together, and the JDK, Rust, Zig, AWS-credential and
artifact-download steps that don't depend on each other run at the same
time.

**Not changed**
- Windows: splitting the wheel build into its own job was tried and
reverted. The runner pool keeps one Windows instance warm and a cold one
takes 85–105s to boot, which costs more than the 67s it took off the
critical path.
- Rust tests (cpu-features): "Check CPU features" and the test step
compile into the same cargo target directory, so cargo would make them
wait for each other.
- The CUDA, sanitizer, musl and Rust/Clippy jobs: their time is almost
all compiling and running tests.
- No matrix caps `max-parallel`, so all matrix legs already run at once.

## Measured

| | Before | After |
|---|---|---|
| SQL bench: package + upload binaries | 64–90s | 12s |
| SQL bench: download + unpack, per bench job | 16–20s | 17–31s |
| Java, warm Gradle cache | 100s | 81s longest leg + 2s gate |
| Java, cold Gradle cache | 546–588s | the compat suites (about 1.5–2
min each) no longer run in series |
| License check | 92–110s | 67–84s |
| Docs build | 11m33s | 8m39s–10m36s |

The `ci.yml` wall time is still set by Windows (about 8.5–10.5 min), so
these changes shorten individual jobs, not the whole run. The SQL
benchmark changes shorten the bench run directly. On the latest commit
the SQL benchmarks all finished, each with a "No clear signal" verdict,
as expected for a CI-only change.

## Checks
- `yamllint --strict -c .yamllint.yaml .github/` passes.
- For every job in every workflow, I parsed the old and new YAML,
flattened the `parallel:` groups and compared the steps. They are
identical apart from the rewritten Javadoc copy step, the `!cancelled()`
on the advisory warning, the Java matrix, and the SQL bench packaging.
- I ran Cargo 1.100 locally to confirm the new `OUT_DIR` layout, and
checked that the `libduckdb.so` search matches it.
- A separate review pass checked each group for shared files,
environments, outputs, `PATH`/`GITHUB_ENV` needs and apt-lock conflicts.
- CI on the latest commit is green, including all four Java legs, the
Java gate, and the integration smoke tests. An earlier smoke-test
failure, where the RustFS bucket creation returned an HTTP error after
the health check passed, did not recur.

---
🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary

TurboQuant stores its Arrow metadata as base64 without an encoding
marker. Vortex's fallback for registered Vortex IDs decodes base64 only
when marked, so it reads TurboQuant's metadata as literal text and
rejects valid arrays, even after a plugin has imported them
successfully. An exporter that declines and delegates to storage
conversion can hit the same check.

## Changes

Successful plugin imports are final because the existing dtype and
length checks already validate their result. Export fallback asks the
registered importer to decode the field before comparing types, since
the same metadata can use different byte encodings. The complete dtype
check, apart from outer nullability, and the physical storage check
remain. Without a matching importer, the field must contain the source
type's serialized metadata.

Map entries import as a plain struct because the map's dtype comes from
its key and value fields. Running a plugin on the entries field could
produce a different dtype from schema inference. Key and value plugins
still run.

Tests cover native metadata imports and export round trips, invalid
metadata, and physical storage mismatches. A byte-collision case checks
that equal metadata bytes cannot hide different type parameters. Map
tests cover ignored entries plugins and length checks on value imports.

## API Changes

Import plugins no longer run on a map's entries field. Extension types
on the key and value fields still reach their plugins.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Mutation testing with cargo-mutants (see vortex-data#11471) found code in
`vortex-fastlanes` that the tests run but never check. This PR adds
tests for two such gaps, keeping only the cases that catch distinct
mutants. It changes tests only.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary

Adding a benchmark label (`action/bench-*`) to a stacked PR did nothing,
because `pr-bench-dispatch.yml` only triggered for PRs whose base branch
is `develop`.

## Changes

- Remove the `branches: ["develop"]` filter from the `pull_request`
trigger in `.github/workflows/pr-bench-dispatch.yml`.

The baseline needs no changes. `scripts/fetch-benchmark-baseline.py`
already picks the newest commit reachable from the PR head that has
recorded results, and only develop-mode runs write to the benchmarks
database. So a stacked PR is compared against `develop`, and the
comparison includes every commit in the stack below it.

GitHub reads the workflow from the PR's merge commit. An existing stack
picks this up only after its parent branch is rebased onto `develop`
with this change.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniBC1KY6sxmZBh4hoUwg)_

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
)

Several reads and slices binary search a sorted child, such as patch
indices or run ends. Some of them built a new searcher, and so a new
probe, for every search. RunEnd's probe lookup didn't use the
search-sorted API at all: it hand-rolled a binary search that read the
ends one scalar at a time. This PR changes only those call sites. Every
sorted search now goes through the search-sorted API with one shared
probe. That holds even for a single search inside a scalar read: the
search reuses the caller's probe instead of building its own.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Robert Kruszewski <github@robertk.io>
…ortex-data#11470)

## Summary

In the latest `develop` CodSpeed run
([43502927659](https://github.com/vortex-data/vortex/actions/runs/43502927659)),
these benchmarks spent most of their time either building fixtures under
the CPU simulation or measuring cases far larger than the rest of their
sweep. This PR removes that time without changing what the remaining
benchmarks measure.

## Changes

- **`vortex-fsst/fsst_take`** (8.5 s in CI)
- The fixture searched for a row length that reaches its compressed-size
target. It started from an uncompressed estimate and grew the rows 2% at
a time, retraining the FSST compressor at every step: 26 to 81 times per
case.
- It now scales the row length by the compression ratio it just
observed, which reaches the target in 2 to 4 compressions.
- Every fixture still reaches its target, which is what the
compressed-bytes gate under test depends on. The chosen row lengths stay
within about 2% of the previous ones, and the codes sizes within about
3%, so expect a small one-time shift on CodSpeed.
- **`vortex-file/split_collection`**
- The files were written with the BtrBlocks compressor, but split
collection walks only the layout tree. They are now written
uncompressed.
- For every config and the misaligned file, the layout trees and the
splits from both `SplitBy::Layout` and `SplitBy::default()` are
identical to before.
- **`vortex-onpair/decode`** (10.5 s in CI)
- The three `(UrlLog, 1_000_000)` cases cost 2.2–2.9 s each. They
measure the same kernels as the `(UrlLog, 100_000)` cases at ten times
the size.
- They now run only locally (`#[cfg(not(codspeed))]`), as
`split_collection` already does for its scaling cases.
- CodSpeed measures 9 of the 12 benchmarks, and
`scripts/codspeed_bench_counts.json` is updated to match. These three
benchmarks lose their CodSpeed history.

Left alone on purpose:
- **`vortex-spatial/binary_predicates`** (4.5 s): locally each benchmark
after the first takes about 0.01 s. Its CI cost is the measured
iteration under instrumentation, which the file sizes deliberately
against its per-iteration budget.
- **`vortex-arrow/to_arrow`** (29 s over 130 benchmarks): locally each
case takes 0.02–0.04 s including setup, so its CI cost is also mostly
measured work. Sharing fixtures between its benchmarks could let one
benchmark warm cached statistics for the next.

## Measurements

Local `valgrind --tool=callgrind` runs of the CodSpeed builds (`--cfg
codspeed`, `VORTEX_BENCH_VARIANT=simulation`). These include setup but
not CodSpeed's instrumentation, so they show the setup savings rather
than CI times.

| Binary | Before | After |
|---|---:|---:|
| `fsst_take` | 1.84 s | 0.65 s |
| `fsst_take`, the 18 `take` cases | 1.14 s | 0.27 s |
| `split_collection` `warm` (first to build the file) | 0.11 s | 0.02 s
|
| `onpair/decode` | 2.07 s | 0.78 s |

## Checks

- Built the three benches with `--cfg codspeed`: no warnings. CodSpeed
now runs 9 `decode`, 24 `fsst_take` and 8 `split_collection` benchmarks.
- Compared both fixture searches for every `fsst_take` case, and
compared trees and splits for every `split_collection` file with a
temporary debug `main`.
- `cargo +nightly-2026-09-10 fmt --check` on the changed bench files.
- `pytest scripts/tests/test_codspeed_shards.py` (22 passed).
- I did not run clippy or the crate test suites.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo)_

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
## Summary

- Tracking Issue: vortex-data#11359

`ArrayRef::slice` returns `Canonical::empty(dtype)` for an empty range,
and `Canonical::empty` panicked for `DType::Variant`, so slicing any
Variant array to an empty range panicked instead of returning an empty
array with the same dtype. This is the panic reported in vortex-data#11359, and it
also blocks the Arrow round-trip coverage for empty containers with
Variant children that vortex-data#11361 works around.

## Changes

- `Canonical::empty` now builds a Variant empty instead of panicking. A
Variant array needs Variant-typed core storage, and the core storage
must not be a `VariantArray`, because `VariantArray::try_new` would call
back into `Canonical::empty`. The empty is a zero-length `ConstantArray`
holding a null Variant scalar; a non-nullable Variant empty casts from
the nullable one, which is safe because there are no null values to
drop.

Signed-off-by: ump45nose <52391318+ump45nose@users.noreply.github.com>
This PR contains the following updates:

| Update | Change |
|---|---|
| lockFileMaintenance | All locks refreshed |

🔧 This Pull Request updates lock files to use the latest dependency
versions.

---

### Configuration

📅 **Schedule**: (UTC)

- Branch creation
  - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`)
- Automerge
  - At any time (no schedule defined)

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config
help](https://redirect.github.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/vortex-data/vortex).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…data#11410)

## Summary

`VortexDataset.take` failed with "buffer values must be strictly
increasing" for `take([7, 2])` and `take([7, 2, 7])`, while pyarrow's
`Dataset.take` returns the rows in the requested order.

## Changes

- When the indices are not strictly increasing and no row filter
applies, scan the sorted distinct rows once, then gather them back into
the requested order with Vortex `Array.take`. Sorted input keeps the
single-scan path.
- With a filter (the `filter` argument or `Dataset.filter`) the
surviving rows cannot be mapped back to positions, so strictly
increasing indices are still required. The docstring says so.

---------

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…ortex-data#11384)

## Summary

`VortexDataset.head(n)` raised `RuntimeError: OutOfBounds: stop 10 >
length 5` whenever fewer than `n` rows matched, where a pyarrow Parquet
dataset returns the rows it has: `head(10)` on a 5-row file, or
`head(10, filter=pc.field("x") > 3)` matching one row.
`ds.scanner(batch_size=N).head(k)` always raised `ValueError: batch_size
is not supported`, although `to_batches` takes `batch_size`.

## Changes

`head` now reads through the record batch reader that `to_batches` uses,
with `num_rows` as a new scan
`limit`. A short result is returned as is, and the scan stops once it
has enough rows: on a 20M-row file, `head(10, filter=pc.field("i") > 5)`
grows peak RSS by 20 MiB instead of 109 MiB.

---------

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Ray Data could read Vortex files but had no distributed write path. This
adds `VortexDatasink`, so `ds.write_datasink(VortexDatasink(path))`
streams each write task to one Vortex file instead of collecting data on
the driver or switching formats.

- Tracking Issue: vortex-data#11414

Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…1436)

Add a Pixi environment with the dependencies needed to build and test
CUDA/cuDF C++ code.

- Include cuDF, GTest, CMake/Ninja, and C++/CUDA compilers.
- Check in a lockfile for Linux x86-64 and AArch64, with CUDA pinned to
13.0 for CI driver compatibility.
- Document how to install Pixi and enter the environment.

This PR only adds the environment and documentation. AMI installation
and wiring integration tests into CI are separate changes.

---------

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
connortsui20 and others added 29 commits November 18, 2026 15:22
`cargo-mutants` skips macro bodies, which hides VarBin’s choice between
validating the whole byte range and checking non-null strings
individually. A generic helper exposes that decision and the per-string
loop to mutation testing.

Follow-up to vortex-data#11605.
Stacked on vortex-data#11606.

Prepared and opened with Codex.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`cargo-mutants` skips macro bodies, which hides the choice between
chunked and flat patch searches. Generic helpers expose that decision
and the flat search’s offset handling to mutation testing.

Follow-up to vortex-data#11605.
Stacked on vortex-data#11607.

Prepared and opened with Codex.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The file opener and the Scan API adapter both convert projected Vortex
chunks into Arrow record batches. Sharing that conversion keeps the two
paths consistent while each caller still chooses its target schema and
runs the conversion on its existing CPU task.

Stacked on vortex-data#11557.
Part of vortex-data#2254.

This change was prepared with assistance from OpenAI Codex.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
- Tracking Issue: vortex-data#10733

All DecimalByteParts v2 data lives in its children. The reader currently
accepts and ignores buffers attached to the node itself. Reject those
buffers before v2 enters preview so the reader enforces the same shape
the writer emits.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
)

## Summary

- Tracking Issue: vortex-data#11394

`IS [NOT] DISTINCT FROM` filters fail against a Vortex scan in DuckDB
with `Binder Error: cannot convert
DUCKDB_VX_EXPR_TYPE_COMPARE_DISTINCT_FROM`. So do `IS [NOT] TRUE` and
`IS [NOT] FALSE`, which DuckDB rewrites to them. `can_push_expression`
accepts every `BoundComparison`, but the converter only handles the six
operators in `Operator`.

This lowers the two operators in the converter so the filter is
evaluated by the scan.

## Changes

- `vortex-duckdb/src/convert/expr.rs`: lower `IS NOT DISTINCT FROM` as
`(l IS NULL AND r IS NULL) OR (l IS NOT NULL AND r IS NOT NULL AND l =
r)`, and `IS DISTINCT FROM` as its negation. Unlike `=`, the result is
never NULL, so it is correct under `NOT` and `OR`.
- `vortex-sqllogictest/slt/is_distinct_from.slt`: covers a constant,
`NULL`, column against column, `NOT (...)` and `IS [NOT] TRUE`. On
DuckDB it also asserts through `EXPLAIN` that no `FILTER` node remains.

---------

Signed-off-by: Ayush Mishra <ayushm1906@gmail.com>
…#11611)

Use the shared driver for the DataFusion V2 file opener and remove its
duplicate scheduler. The opener still waits for scan preparation so
failures are reported as file-open errors, and Arrow conversion still
runs on the task that produces each batch.

When several chunks run on the same scheduler, such as with
`scan_concurrency=0`, each chunk needs its own queue for pending reads.
The shared driver used to give these chunks the same I/O session, so
when one chunk finished and cleared the session, it could discard
another chunk’s pending reads. Give each planner root its own session
over the same file source, preserving the isolation DataFusion already
had for inline chunks.

Stacked on vortex-data#11610.
Part of vortex-data#9062.

This change was prepared with assistance from OpenAI Codex.

---------

Signed-off-by: Nicholas Gates <nick@nickgates.com>
- Tracking Issue: vortex-data#10733

The existing decimal wire format can preserve narrow storage only when
`Widen` wraps a decimal leaf. Integer arrays and decimals with encoded
children need a wire ID that carries the child dtype so readers can
interpret the narrow storage directly. No edition permits `vortex.widen`
yet.

A decimal leaf still writes `vortex.decimal`, even when the target
permits `vortex.widen`, because `ArrayPlugin::serialize` does not see
the permitted IDs and so cannot choose the ID by edition. Those arrays
stay readable under existing editions, but their reads keep the range
scan of the legacy decimal reader. When the target does not permit
`vortex.widen`, serialization returns an error instead of writing the
canonical array, as `develop` already does for these arrays.

This change was prepared with AI assistance.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…#11617)

- Tracking Issue: vortex-data#11114

Vortex declares the nullability of every nested level, and a
non-nullable Vortex array has no validity at all. Arrow is looser: a
level whose field is non-nullable can still carry null bits, for example
under a null parent row, and the declared storage of an extension type
can be stricter than the Arrow fields it imports from. So the import has
to put a real value in each of those slots:

```text
Arrow                                 Vortex
struct (nullable)  [ valid, null ]    struct (nullable)  [ valid, null ]
└── x (not null)   [ 1,     null ]    └── x (not null)   [ 1,     0    ]
```

vortex-data#11223 and vortex-data#11497 handled this by passing a mask of hidden slots down
the tree, dropping a null that a null ancestor hides and failing on any
other. The mask only went through structs, lists, and fixed-size lists,
so the same `x` below a list view, a map, a run-end array, or a
dictionary failed to import, and the `vortex_arrow::convert` functions
never used the mask at all. vortex-data#11602 extended it to the other levels,
which showed how much the approach costs: list views can overlap, a
run-end value covers many rows, and many dictionary keys share one
value, so each level needs its own rule for mapping rows to children.

This drops the mask, and each level now only looks at its own declared
nullability. A non-nullable leaf fills its nulls with the zero of its
storage type, and a non-nullable container drops its validity and keeps
its contents, so its children follow their own fields. Both import paths
share this rule.

Importers still get the raw Arrow array. Most go through
`from_arrow_storage`, which applies the rule with the nullability of the
storage dtype, and the UUID and Parquet Variant importers, which convert
validity themselves, now use the new public `level_validity`. A
non-nullable dictionary points its null keys at entry zero, and a
dictionary with keys but no values (as `new_null_array` builds) gets one
entry by importing a single null row of the value type under the same
rule.

Behavior changes:

- A null in a valid row of a non-nullable level now imports as zero
instead of failing, so a null element in a valid `Vector` row becomes
`0.0`.
- Inside extension storage, the fill uses the storage zero without
asking the extension, so a non-nullable JSON field under a null row
holds `""`.
- `vortex_arrow::convert` now imports run-end arrays, since it shares
the run-end import with the session.

Replaces vortex-data#11602.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
- Tracking Issue: vortex-data#11114

A cast between an extension dtype and any other dtype reinterprets the
storage instead of failing, so `Timestamp(ms)` casts to `u8`, and a
constant `Timestamp(ms)` of 1000 casts to `Timestamp(ns)` of 1000, which
turns one second into one microsecond. A conversion that changes what a
value means belongs to the type, so the generic cast now only changes
the top-level nullability or builds typed nulls from an untyped `Null`,
and code that wants the raw storage reads it with `ext_storage`.
`Cast::return_dtype` checks the rule, so a bad cast fails when the
expression is built instead of failing in one kernel but not another.

Breaking: a cast from or to an extension dtype, including one nested in
a struct, list, or map, fails unless it only changes nullability, so
serialized expressions that hold such a cast no longer load.

Breaking: `Cast::new` is now `Cast::try_new`.

Breaking: an Arrow export of an extension array fails when the target
neither names the type nor has an Arrow type that imports as it, such as
`Int64` for a timestamp, a nested field without metadata, or the target
of `Datum::try_new_with_target_datatype`. Apply `ext_storage` to export
the storage.

Breaking: Python no longer builds extension scalars or literals from raw
integers.

Prepared with AI assistance.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…a#11615)

vortex-data#11595 removed `vortex.decimal_byte_parts.v2` from the `unstable`
edition because `between` and aggregates decode to canonical form when
lower parts exist. That is a performance gap, not a wire problem, and
`unstable` has no compatibility guarantee, so the encoding goes back in
and the decompression kernels can follow. `vortex.widen` gets a wire ID
in vortex-data#11597. Adding it to `unstable` lets writers that enforce editions
emit integer `Widen` arrays and decimal `Widen` arrays over encoded
children. Without it, those writes return an error.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary

- Tracking Issue: vortex-data#11420

A buffer that grows by appending can have up to 2x its length in
capacity, and `freeze` keeps all of it. This adds
`BufferMut::shrink_to_fit`, which uses `realloc` to release the spare
capacity in place, so callers such as `FsstSink::finish` can trim a
buffer before they freeze it.

- `Allocation::shrink` forwards to `Allocator::shrink`. If the allocator
refuses, the allocation does not change.
- `shrink_to_fit` keeps the data at its offset, and moves it back to an
aligned offset if `realloc` misaligns it. A sliced buffer is copied once
into a new allocation. An empty buffer becomes unallocated. A second
call does nothing.
- Tests use an allocator that moves each shrunk block to a misaligned
address. New `shrink_to_fit` and `shrink_by_copy` benchmarks are in
`benches/allocation.rs`.

On glibc, `shrink_to_fit` is 30–550x faster than a copy (for example, 4
MiB: 663 ns vs 366 µs). mimalloc copies on `realloc`, so there it costs
the same as a copy.

## API Changes

Adds `BufferMut::shrink_to_fit`. After a shrink, the data has only the
buffer's own alignment, not the `DEFAULT_ALIGNMENT` over-alignment.
Growth already has the same guarantee.

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Robert Kruszewski <github@robertk.io>
…city (vortex-data#11434)

## Summary

- Tracking Issue: vortex-data#11420

`fsst_compress` builds the codes buffer by appending one batch at a
time, and `BufferMut` growth at least doubles the capacity. `freeze`
keeps that allocation, so a compressed FSST array could hold up to about
1.9x its `nbytes()`, depending on where the last batch landed relative
to the last doubling. OnPair has the same gap: `onpair::compress_rows`
grows the codes `Vec` from empty, and `Buffer::from(Vec)` keeps its
whole allocation.

## Changes

- FSST: at the end of `FsstSink::finish`, when the codes buffer's spare
capacity is more than a quarter of its length, copy the codes into an
exactly sized buffer (same alignment, the context's allocator) before
freezing. Otherwise the buffer is frozen as before.
- OnPair: in `onpair_compress`, call `shrink_to_fit` on the codes under
the same condition, before they become a `Buffer`. The dictionary bytes
were already copied into an exactly sized buffer.
- Add a `codes_buffer_has_little_spare_capacity` `rstest` regression to
each crate, over 8,192 random hex strings of several lengths. Each one
unwraps the uniquely owned codes buffer with `try_into_mut` and checks
that the spare capacity is at most a quarter of the length. Both fail on
`develop` and pass with this change.

---------

Signed-off-by: Shu Chen <338845574+shu-chen-dot-ca@users.noreply.github.com>
Co-authored-by: Robert Kruszewski <github@robertk.io>
…tex-data#11430)

## Summary

Follow-up to vortex-data#11418 and vortex-data#11423. The `#[array_slots]` accessors now
return references borrowed for the array, or for the `ArrayView`'s
lifetime `'a`. So there's no longer a reason to index `slots()` by
`XSlots` constants and unwrap by hand, or to keep hand-written accessors
that duplicate the generated ones.

## API Changes

- Removed `PrimitiveArrayExt::validity_child`,
`DecimalArrayExt::validity_child`, `InterleaveArrayExt::array_indices`,
`InterleaveArrayExt::row_indices` and
`ChunkedArrayExt::chunk_offsets_array`. The generated accessors replace
them, so callers need the encoding's `…ArraySlotsExt` trait in scope.
Use `chunk_offsets()` in place of `chunk_offsets_array()`.
- Newly public: `InterleaveSlots`, `InterleaveSlotsView` and
`InterleaveArraySlotsExt` from `arrays::interleave`, and
`ChunkedArraySlotsExt` from `arrays::chunked`.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
## Summary

Baseline benchmarks for taking `Patches` with `PiecewiseSequenceArray`
indices, ahead of the stacked follow-up
vortex-data#11547, which takes the runs
without expanding them. Landing the benchmarks first gives that change a
recorded baseline to compare against.

## Changes

- Adds `take_piecewise` to `vortex-array/benches/take_patches.rs`. It
takes patches at density 0.1 with ascending runs of contiguous indices,
covering half of a 4096-row array, for run lengths 1, 16 and 256.
- Inputs go through `mark_optimized`, like the existing benchmarks in
this file.

---------

Signed-off-by: Robert Kruszewski <github@robertk.io>
Add a `filter_bool` benchmark for validity bitmaps. Validity bitmaps are
mostly set, even when the selection is random. The existing sweeps use
only random source bitmaps.
 
- Add `density_sweep_validity` at 4 densities: 0.02, 0.1, 0.5, and 0.99.

Signed-off-by: Robert Kruszewski <github@robertk.io>
The dynamic dispatch kernel unpacks each 1024-element FastLanes chunk
with one thread per lane, but 32- and 64-bit chunks have only 32 and 16
lanes, so half or three quarters of each 64-thread block waited at the
barrier during the unpack. This splits those lanes into 16-row slices so
every thread decodes one, and spreads the work over all chunks of a
stage. The generator emits the per-row decode code once, as slices, and
builds the whole-lane decoders used by the standalone kernels from them.

Fused u32 and u64 kernels in `dynamic_dispatch_cuda` are 11–19% faster,
e.g. `alp_for_bp_6bw_f64` at 100M elements goes from 369 µs to 299 µs.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…x-data#11634)

`CudaExecutionCtx::copy_to_device` and `ensure_on_device` waited for a
stream callback after enqueueing each copy. That callback fires only
once *all* earlier work on the stream has finished, so every small
host-to-device copy during decode and Arrow export (e.g. FSST symbol
tables, binary-export pointer tables) stalled the host until the GPU
drained, leaving the GPU idle while the host queued the next kernels.

The wait isn't needed: later work on the same stream is already ordered
after the copy, and readers on other streams wait for the buffer's
recorded write. Both methods now return the buffer as soon as the copy
is enqueued. `DeviceReadAt`, which hands buffers across streams, keeps
the waiting copy.

Measured end to end with BenchPress (TPC-H SF10 `lineitem`, GPU read
through the CUDA FFI, hot page cache, 5 iterations, two runs each):

| | Before | After |
|---|---:|---:|
| Read time | 41.85 ms | 40.25 ms |
| Throughput | 196.0 GB/s | 203.9 GB/s |

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ortex-data#11631)

The fused kernel runs one block per 2048 output elements, and every
block first decodes the plan's input stages (dictionary values, run
ends) into its own shared memory. With large dictionaries or many runs,
most of the kernel's time went into decoding the same input stages over
and over.

This adds a `loop_ranges` kernel that launches only as many blocks as
fit on the GPU at once and loops each over several 2048-element ranges.
Each block decodes the input stages once and reuses them for all its
ranges, cutting redundant decodes by roughly 12× at 100M elements. Plans
with fewer than 64 input elements keep the original kernel, where the
decode is cheap and looping doesn't pay off.

`dynamic_dispatch_cuda` at 100M elements:

| Benchmark | Before | After |
|---|---:|---:|
| `dict_4096vals_bp12bw_codes/dispatch_u32` | 1142.8 µs | 673.9 µs |
| `dict_4096vals_bp12bw_codes/dispatch_u64` | 2348.7 µs | 1377.9 µs |
| `runend_1024runs/dispatch_u32` | 370.7 µs | 206.6 µs |
| `dict_1024vals_bp10bw_codes/dispatch_u32` | 341.9 µs | 312.2 µs |

---------

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Moves the segment types and readers into a new `vortex_io::segments`
module: `SegmentId`, `SegmentSource` and `SegmentLocation` from
vortex-scan, the coalescing `FileSegmentSource` with its read driver,
`BufferSegmentSource` and `RequestMetrics` from vortex-file, and
`SharedSegmentSource` from vortex-layout. The I/O layer can then read a
file's segments itself, which vortex-data#11623 needs for its session-wide
scheduler, without depending on vortex-file or vortex-layout. Every item
stays re-exported from its old path, and behaviour is unchanged.

The only edits beyond imports: `FileSegmentSource` and
`BufferSegmentSource` take `SegmentLocation` instead of vortex-file's
`SegmentSpec` (same fields), so `Footer::segment_locations()` now does
the conversion that `VortexFile` did; `SegmentLocation::target()`
becomes a private free function in vortex-scan because the type now
lives in another crate; and two casts and a test `cfg` change to satisfy
vortex-io's stricter lints.

Written with Claude Code. The vortex-io, vortex-scan, vortex-layout and
vortex-file tests and clippy pass locally.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On `develop`, a V2 scan that reads one field from a nullable struct can
still plan every child. The scan fails if a sibling uses a layout that
V2 cannot execute. This PR plans only the requested fields and the
validity needed to preserve their nulls.

For `address.zip`, the scan can skip `address.city` and an unused
sibling list. A null `address` still produces a null `zip`.

Related to vortex-data#10952, which handles deep projections in the classic reader.

This change was implemented and reviewed with OpenAI Codex.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Delta's serialized form was settled in vortex-data#11441, and with the fused decode
and kernels (vortex-data#11428) and the sampled estimator (vortex-data#11453) it is cheap
enough to offer for broad opt-in use. This moves `fastlanes.delta` from
`unstable` into the draft `preview2026.08.0` edition, so Delta's wire
format becomes fixed when that edition freezes. Writers that enable only
`core` still never write Delta.

Benchmark sessions now enable `preview` as well as `unstable` through
`enable_unstable_encodings`, so every benchmark keeps writing Delta.
This also enables `preview`'s other member, the
`vortex.uncompressed_size_in_bytes` aggregate, in benchmark files. PR
benchmarks against `develop`, which already writes Delta through
`unstable`, are flat.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…11151)

A malformed `batch.size` or `vortex.write.batch.size`, a negative
`vortex.workerThreads`, or a `vortex.session.provider` class that can't
be used fails only inside write tasks. On a V1 overwrite, Spark has
already deleted the old files by then. The connector also ignores
`vortex.workerThreads` once the JVM's worker pool is sized.

---------

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
Part of vortex-data#11516

This is the first step toward an ALP-RD GPU kernel, as @0ax1 suggested
on the issue. The CUDA patch loader needs `chunk_offsets` to find a
chunk's patches in constant time, but the ALP-RD encoder never wrote
them, and the deserializer dropped them on read. Both spots had a
`TODO(0ax1)`.

New ALP-RD files now carry one extra child holding the offsets. The
compressor narrows it, so it's 16 bytes for 16k rows in the btrblocks
golden snapshot. Older files without it read the same as before, and
older readers simply ignore the extra child.

Could a maintainer add the `changelog/feature` label? I can't set labels
from a fork.

I used Claude Code for analysis, debugging, and ideas.

Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
Co-authored-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
…a#11682)

For `col = 'x'` on a `RunEndEncoded<_, Utf8>` column, DataFusion keeps
the column type and wraps the literal in REE, so there is no cast for
vortex-data#10637 to drop, and the converter's supported-type check rejected REE.
The check now looks at the REE values type, as it already does for
dictionaries, so these filters run inside the scan instead of in a
`FilterExec` above it.

Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap
run-end encoding only as the outermost layer, so REE nested in REE or in
a dictionary is declined. DataFusion reads the nulls of a run-end
encoded `IN` list from the `RunArray`'s physical validity, which is
always empty, so for a value not in `(a, b, c, NULL)` it returns false
where SQL returns NULL; such lists are declined so both engines keep the
same rows.

The new test compares Vortex with DataFusion for 26 value types on
plain, dictionary and REE columns, using four-value `IN` lists because
DataFusion rewrites shorter column `IN` lists into comparisons.

Related: vortex-data#10637

Signed-off-by: Ivan Kiselev <ivan.kiselev@dash0.com>
…ta#11630)

Every fused dynamic dispatch launch allocated a device buffer for its
100–200 byte plan, copied the plan into it from pageable host memory,
and tracked the buffer until the kernel finished.

This change now passes the plan by value as a 1 KB `__grid_constant__`
kernel parameter instead; plans that don't fit fall back to unfused
dispatch, like plans over the shared memory limit. The kernel now also
reads the plan, including the scalar ops it re-reads on every tile, from
the constant cache instead of global memory.

Total time per fused dictionary decode (host wall time including the
kernel):

| Elements | Before | After |
|---:|---:|---:|
| 16K | 20.3 µs | 16.4 µs |
| 256K | 20.7 µs | 16.7 µs |
| 4M | 24.6 µs | 21.2 µs |

Kernel time in `dynamic_dispatch_cuda` (100M elements, plan uploaded
outside the timed region):

| Benchmark | Before | After |
|---|---:|---:|
| `for_bitpacked_6bw/dispatch_u32` | 210.9 µs | 159.5 µs |
| `dict_256vals_bp8bw_codes/dispatch_u32` | 240.4 µs | 184.0 µs |
| `dict_nowiden_u32_to_u32/dispatch_u32` | 183.0 µs | 146.9 µs |
| `alp_for_bp_6bw_f32/dispatch_f32` | 228.4 µs | 180.8 µs |
| `alp_for_bp_6bw_f64/dispatch_f64` | 369.3 µs | 360.8 µs |
| `alp_for_bp_6bw_f64_1pct_patches/dispatch_f64` | 452.2 µs | 424.1 µs |

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
- Tracking Issue: vortex-data#11115

Functions use the storage of any extension type today, so comparing two
`Json` values compares their strings. A type needs a way to promise that
its storage answers equality or order correctly before a function can
trust it. This PR adds only the declaration, and vortex-data#11600 makes each
function check it.

`ExtVTable::claims` returns an opaque `Claims` value. A `&'static
[Claim]` slice was the alternative, but the slice could disagree with
`DType::holds` about `Eq`, it had several values for the same promise
(`[Ord]` and `[Eq, Ord]`), and it could only name properties. Private
fields leave room for a custom order key later.

AI assisted with this change.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Arrow export serializes struct fields: each child is decoded and
exported before the next one starts. When a field triggers a
device-to-host readback (string sizes, run-end totals, FSST symbol
tables), the GPU idles while the host waits for the result, and no other
field's work can fill that gap.

Each field now exports on its own CUDA stream via a forked execution
context. The struct's stream waits for every field event before
recording the final sync, so downstream consumers still see a single
ordered export. Error and cancellation cleanup uses per-field
`PendingArrowChildren` guards.

Measured end to end with BenchPress (TPC-H SF10 `lineitem`, GPU read
through the CUDA FFI, hot page cache, 5 iterations, two runs each):

| | Before | After |
|---|---:|---:|
| Read time | 41.85 ms | 38.85 ms |
| Throughput | 196.0 GB/s | 211.4 GB/s |

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Merge vortex-data/vortex develop at 79e75c1.

Conflict resolutions:
- zone_map.rs: keep the fork's large IN-list pruning tests; upstream moved
  the other zone map tests to vortex-scan.
- plan/tests.rs: take upstream; port the fork's fallible ListLayout::try_new
  to the moved lowering tests.
- VortexSparkSession: take upstream's vortex-data#11151, which supersedes the fork's
  version of the same change.
- VortexOutputWriterFactory: keep the fork's public validate covering
  batch.bytes, capacity reset and listLayout.
- VortexWriteOptionValidationTest: upstream V1/V2 structure with the fork's
  option cases; out-of-range batch sizes stay rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
@ChristopherWirt ChristopherWirt changed the title Merge upstream develop into QC develop (2026-11-18) Merge upstream develop into QC develop (2026-11-20) Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.