Repository navigation
Merge upstream develop into QC develop (2026-11-20) - #122
Open
ChristopherWirt wants to merge 130 commits into
Open
ChristopherWirt wants to merge 130 commits into
ChristopherWirt wants to merge 130 commits into
Conversation
…ortex-data#11421) ## Summary First of two stacked drafts: expose dictionary-preserving GPU batches so consumers can postpone string expansion until after combining batches. This PR adds the export API only; the companion PR vortex-data#11422 adds an optional cuDF materialization helper and reports the ~20% end-to-end hot-read gain. A fixed-schema Arrow device stream cannot describe batches whose dictionary encodings or index widths differ. The new iterator returns an independently owned schema and device array for each batch, using the existing `DictionaryExport::Preserve` policy. ## Changes - Add `vx_cuda_scan_path_device_batches`, `vx_cuda_device_batches_next`, and `vx_cuda_device_batches_free`. - Reuse the existing scan options, ordered top-level projection, direct I/O, and batch splitting. - Keep the existing fixed-schema Arrow stream API and its decoded output unchanged. - Document ownership, stream synchronization, thread affinity, end-of-stream, and terminal-error behavior; regenerate the C header. - Add 10 regression tests for varying dictionary/index types, plain batches, nulls, empty batches, projection, independent output lifetimes, invalid inputs, and terminal errors. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
This PR adds benchmarks for the Delta encoding. They are split out of vortex-data#11428 so that its changes can be measured against `develop`, and vortex-data#11428 stacks on this PR. ## Changes `encodings/fastlanes/benches/delta_decode.rs` adds 15 benchmarks. **Walltime (9 benchmarks).** `decode_bitpacked_deltas` decodes a 512 KiB Delta array whose deltas are bit-packed, for u16, u32 and u64. It carries `#[cpu_features]`, so it runs on each walltime CPU-feature leg, where the build selects the transpose and decode kernels. **Simulation (6 benchmarks).** These run once each, on 32 KiB arrays so they stay under 1 ms in CodSpeed's simulation, which runs roughly 13× slower than native: - `decode_primitive_deltas` decodes a u32 Delta array whose deltas are a primitive array. - `probe_once` and `probe_repeated` read 8 rows from 2 chunks of a 2048-row array. `probe_once` uses a one-off probe per row; `probe_repeated` reuses one probe for all 8. - `filter` selects one row in every 64, or one in every 4096. - `take` takes 128 scattered rows. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary Each change to an edition (adding, moving, or removing a member, or freezing an edition) should go in its own PR and show up separately in the release notes. This adds a dedicated `changelog/edition` label. The label check already requires exactly one `changelog/*` label, so an edition PR can't also be labelled as a feature or fix. ## Changes - `.github/workflows/labels.yml`: accept `changelog/edition` as a changelog label. - `.github/release-drafter.yml`: add a "📜 Editions" release-notes section after "Breaks", and make `changelog/edition` trigger a minor version bump, the same as `changelog/feature` and `changelog/break`. - `docs/specs/versioning/editions.md`: under "Maintaining edition records", say that each edition change goes in its own PR labelled `changelog/edition`. - `.github/pull_request_template.md`: add a matching reminder next to the `changelog/break` note. The `changelog/edition` label still needs to be created in the repository's label settings before it can be applied. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…vortex-data#11437) ## Summary `IsSorted` orders nulls first. The non-strict branch of the bool kernel (`check_bool_sorted`) broke that rule: it walked only the indices of valid values, so it dropped nulls instead of ordering them first. The strict branch already handled nulls correctly. It maps each element to `Option`, where `None < Some(_)`, which is what the primitive kernel does in both modes. Two counterexamples showed the bug before this fix: 1. `BoolArray [false, null, true]` reported sorted, but the equivalent `PrimitiveArray [0, null, 1]` reported not sorted. 2. `BoolArray [true, null]` reported sorted, but the same values split into the `ChunkedArray` chunks `[true]` and `[null]` reported not sorted. The accumulator's check at the chunk boundary already sees a valid value followed by a null.
## Summary The SQL benchmarks write Vortex files that we never look inside. This PR adds one artifact per run containing `vx` file stats and layout trees for a sample of each dataset's files, so you can see which encodings and layouts a PR produced on real data. The full trees are too large for logs: hundreds of thousands of lines for ClickBench and the SF100 datasets. So every file is dumped on the runner, but only a sample within a line budget is kept, and the kept dumps from all jobs are merged into one zip retained for 5 days.
…#11441) ## Summary `fastlanes.delta` is still only in the unstable edition. This PR fixes its serialized contract before it is frozen. A review of Delta's serde found two problems: - **Validity is inferred from the child count.** A third child is always read as validity, so the format can't add a child later without it being mistaken for validity. - **The reader is not robust to corrupt input.** - A float dtype panics in `lane_count` before `validate_parts` runs. - `deltas_len` is not tied to `len`, so a tiny array with constant children can claim an unbounded physical length and force a huge allocation on decode. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…a#11443) ## Summary The file-size section disappeared from SQL benchmark PR comments in vortex-data#10987, which moved the baseline from S3 to the benchmarks Postgres. File sizes were never stored there, so `compare-benchmark-jsons.py` found no baseline rows and dropped the section without a message. This PR stores the sizes on develop runs and reads them back on PR runs.
## Summary vortex-data#11443 started storing SQL benchmark file sizes in the benchmarks database, but only for `vortex-file-compressed` and `vortex-compact`. This PR stores every on-disk format, so bench can chart Parquet, DuckDB, Lance and the rest beside Vortex (vortex-data/vortex.dev#222). PR comments still compare only the Vortex formats. ## Changes - `scripts/capture-file-sizes.py` records every format directory that `Format::name()` in `vortex-bench/src/lib.rs` names: `arrow-ipc`, `csv`, `duckdb`, `lance`, `litdata`, `parquet`, `vortex-compact`, `vortex-file-compressed` and `vortex-spatial-native`. Each develop commit adds roughly one row per Parquet file; ClickBench has about 100. - **DuckDB catalogs:** DuckDB keeps a catalog of views (`duckdb.db`, and `duckdb.db.wal` while open) in every format directory it queries (`duckdb-bench/src/lib.rs`). That isn't the format's data, so it's now skipped everywhere except the `duckdb` directory, where the database is the data. This also takes the catalog out of the Vortex totals. Rows recorded today include it, so PR comments based on today's develop commits list `duckdb.db` as a removed file, once. - `scripts/compare-benchmark-jsons.py`: the "File Size Changes" section compares only formats starting with `vortex`. The other formats' files come from the data generators and don't change with a Vortex PR. New tests check that capture records every format, skips the DuckDB catalogs outside `duckdb`, and ignores unknown directories. Another checks that the report leaves out non-Vortex formats. ### Checks - `uv run --no-project --with pytest --with pandas --with numpy --with orjson --with tabulate --with xxhash pytest scripts/tests/test_measurement_id.py scripts/tests/test_benchmark_reporting.py scripts/tests/test_post_ingest.py`: 49 passed. - `uvx ruff format --check` and `uvx ruff check` on the changed files pass, as does `git diff --check`. - Not run: a CI benchmark run. `action/bench-sql` on this PR would exercise capture and the comment. The new formats only reach the database from develop runs. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary - Set the stable toolchain in `rust-toolchain.toml` to `1.100.0`. - Set `NIGHTLY_TOOLCHAIN` to `nightly-2026-11-13` in all 12 workflows that pin it. The new rustfmt makes no changes to the code. ### Clippy 1.100 (`redundant_clone`) - Remove redundant clones in `vortex-array`, `vortex-fastlanes`, `vortex-fuzz`, `vortex-cuda`, `vortex-sqllogictest` and `xtask`. - `vortex-duckdb`: `test_clone_logical_type` tests the FFI `Clone`. Keep the clone and keep the original value alive, and check the type of both values. ### Cargo 1.100 lints - `unused_workspace_dependencies`: - Remove `arrow-arith`. - Add `vortex-elias-fano` to `vortex`, re-export it as `vortex::encodings::elias_fano` and register it in `VortexSession::default()`. It is not in any file edition yet. - `unused_dependencies`: - `datafusion-bench/build.rs` checks `CARGO_CFG_TARGET_OS` and not `#[cfg(target_os)]`, which reads the host OS. The link argument is for the target. - `vortex-web-wasm` puts all its dependencies under `cfg(target_arch = "wasm32")`, because the library is empty on other targets. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
…ata#11184) ## Summary - Tracking Issue: vortex-data#11114 Right now, an extension type validates whatever storage dtype the caller hands it, so one type can have several storage forms, and its metadata doesn't say which one a dtype uses. Each `ExtVTable` now declares its storage from its metadata, so the metadata **deterministically fixes / decides** the whole storage dtype except the top-level nullability. Reads then require the stored storage to equal that declaration exactly, **including nested nullability**. This has to be strict, since arrays decode their storage child with the declared dtype, so a stored form that differs would be misread instead of rejected. Every built-in type accepts the same stored forms as before, so existing files still read. _NB on nested nullability: this means that if we potentially need nulls in the storage, an extension type MUST require its storage type to be nullable, even if in some cases it might not be. I think this is a worthwhile tradeoff for clarity of implementation._ ## Changes Some facts lived only in the storage dtype, like the element type of `Vector` and `FixedShapeTensor` and the dimension of the native spatial types, so they move into the in-memory `Metadata`. `deserialize_metadata` decodes `StoredMetadata` (which is the old `Metadata`, and `metadata_from_storage` combines it with storage to rebuild complete metadata. The serialized bytes don't change. Some other things: - File reads and storage-based dtype, array, and scalar constructors share the same checked path. - `ForeignExtDType` can't derive its storage, since it doesn't know its type, so it keeps the stored storage in its metadata and declares that. - The nested-nullability checks that vortex-data#11183 added to the spatial helpers go away, since the read check now covers them. Follow-up: a read error in a struct field still panics on first access (SF in vortex-data#11114). ## API Changes - `ExtVTable::validate_dtype` becomes `storage_dtype` (plus an optional `validate_metadata`), `deserialize_metadata` returns `StoredMetadata`, and `metadata_from_storage` reconstructs complete metadata from stored metadata and storage. - `ExtDType::try_new` and `try_with_vtable` take a nullability in place of the storage dtype. - Reading an extension dtype whose stored storage differs from its declaration fails, and so does building a scalar or array over the wrong storage. - `FixedShapeTensorMetadata::new` takes the element type, `Vector`'s metadata is `VectorMetadata`, metadata-first construction of native spatial dtypes needs `SpatialMetadata::with_dimension`, and `SpatialMetadata` no longer implements `prost::Message`. - `ExtDType` and `ExtensionArray` gain `try_from_storage` constructors, and `Scalar` gains `try_extension_from_storage`, with explicit-vtable variants. Existing metadata-first constructors remain available. - `FixedShapeTensorStoredMetadata` supplies shape and layout, and `SpatialStoredMetadata` supplies CRS when constructing from storage. - Fallible constructors reject non-nullable null storage, including extensions over null storage, with an error. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary The benchmark jobs run every setup step one after another, even though most of the steps don't depend on each other. This change groups the independent steps into `parallel:` blocks, the same way `ci.yml` already does, to shorten the time a benchmark machine spends before the timed run. It also moves the Polar Signals debuginfo upload out of the build jobs, so the bench jobs no longer wait for it. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
…ake and is_constant kernels (vortex-data#11428) ## Summary FastLanes Delta decoded slowly and had almost no compute kernels: every filter, take, `scalar_at` or `is_constant` decoded the whole array first. Vortex now has the APIs this needs (the probe API and execute kernels). This PR uses them, together with fastlanes 0.8.0's fused unpack-and-undelta and SIMD transpose kernels, so Delta is cheap enough to select by default. ## Changes ### Decode - **Fused decode.** `Delta(BitPacked)` decodes each 1024-value chunk with fastlanes' fused `unchecked_undelta_pack`. The deltas are no longer unpacked into a full intermediate array. This covers patched and block-width bit-packing. - **Sliced arrays** decode only the chunks the slice covers, then trim the result. - **SIMD transposes.** fastlanes is updated to 0.8.0. Transposes are dispatched at runtime through `fearless_simd`: AVX-512 for 8- and 16-bit values; for 32- and 64-bit values, fastlanes' own dispatch, because AVX-512 is slower there on Zen 4. - `DeltaChunks` is a reusable chunk decoder with scratch space. The kernels below share it. ### Kernels - **Probe API.** `scalar_at` decodes only the chunk holding the row. A repeated probe caches the last decoded chunk. - **Filter and take** decode only the chunks that have selected rows. Take falls back to a full decode for large index sets. - **`is_constant`** reads only the stored parts where it can: all-null, mixed-null, or all-zero interior deltas plus constant bases. Only the partial chunks at either end of a slice are decoded. - The chunked take logic in `take_chunks.rs` is shared between BitPacked and Delta. ### Compression - **Back-fill leading nulls per chunk.** `delta_compress` used to carry the previous chunk's last value into a chunk that starts with nulls. Values that restart in every chunk, such as RLE indices, then got wrapping residuals, and the compressor rejected Delta for them. Those leading nulls now take the chunk's first valid value. - **Keep RunEnd off RLE indices.** RLE indices count up within each chunk, so Delta packs them into a few bits. Allowing RunEnd there produced `RLE(indices = RunEnd(ends = Delta))`, which every decode and filter expands twice. This enables the exclusion that had been commented out. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary C and C++ writers can now select a registered edition by name before opening a writer. This lets them target older readers through the same edition-aware writer used by Java and Python. The default file sink uses the selected edition to build its write strategy and picks up the legacy zone-map support from vortex-data#10805. Codex assisted with the implementation, tests, and documentation. ## Changes - Add `vx_session_enable_edition` to the C API. It returns a status code and can report a `vx_error`. - Add `vortex::Session::enable_edition(std::string_view)` to the C++ API. It reports failures through `vortex::VortexException`. - Document how to select an edition before opening a C or C++ writer. - `test_enable_edition` covers selecting an older core edition on a shared session and reporting an unknown edition name. - `test_sink_writes_legacy_zone_map_for_older_edition` checks that a file written through the C sink with `core2025.10.0` has a `vortex.stats` layout. - The C++ session test covers a length-delimited edition name and the exception for an unknown name. --------- Signed-off-by: jinli.zjw <jinli.zjw@alibaba-inc.com>
## Summary - Tracking Issue: vortex-data#10701 vortex-data#11458 and vortex-data#11459 change both the code and the tests of `array.rs`, `assemble.rs`, and `split.rs`. This PR is a purely cosmetic change. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…data#11462) ## Summary Every SQL benchmark that runs DuckDB fails with: ``` duckdb-bench: error while loading shared libraries: libduckdb.so: cannot open shared object file: No such file or directory ``` This started with the Rust 1.100 upgrade (vortex-data#11446). Rust 1.100's cargo moved build script output from `build/<pkg>-<hash>/out` to `build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB cache two levels above `OUT_DIR`, so the library moved: - before: `target/release_debug/build/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so` - now: `target/release_debug/build/vortex-duckdb/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so` The artifact path still names the old location. upload-artifact leaves out an unmatched path without an error, so the bench jobs get `duckdb-bench` without the library that its absolute rpath points to. ## Changes `sql-bench-matrix.yml`, build job: - The artifact path matches the library at any depth under `target/release_debug/build`, so it works with both layouts. It also matches a DuckDB built from source as well as a prebuilt one. upload-artifact still roots the artifact at `target/release_debug`, so the library unpacks at the path the rpath expects. - A new step fails the build job if `libduckdb.so` is missing. A missing library now stops the build, instead of letting every DuckDB benchmark fail later. A sturdier fix would be for `vortex-duckdb/build.rs` to stop deriving its cache location from `OUT_DIR`, since cargo can change that layout again. I kept this PR to the workflow. Checks: - `yamllint --strict -c .yamllint.yaml` passes. - With cargo 1.100, a local build script prints `OUT_DIR=.../target/debug/build/<pkg>/<hash>/out`. - I ran the same fix, as a tar-based packaging step, on vortex-data#11450. There FineWeb, TPC-H SF=1 and SF=10, TPC-DS SF=1 and SF=10, ClickBench and TPC-H on S3 all finished with DuckDB results. Before the fix they failed with the error above, on that PR and on other branches. --- --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary A follow-up to vortex-data#11445. Step timings from recent runs showed CI jobs outside the benchmarks still doing independent setup and checks one step at a time. This groups those steps into `parallel:` blocks, splits the Java job into a matrix, and speeds up the SQL benchmark binary handoff. It also fixes the SQL benchmarks' DuckDB runs, which have been failing since the Rust 1.100 upgrade. ## Changes **SQL benchmarks (`sql-bench-matrix.yml`)** - The build job packs the bench binaries into one tar with multi-threaded zstd and uploads it with `compression-level: 0`, instead of letting upload-artifact zip them with single-threaded zlib. Packing and uploading took 12s, down from 64–90s. Each bench job then spends about 4s unpacking, and every bench job starts 50–75s sooner. The tar also keeps the executable bits, so the `chmod` step is gone. - **Fix:** every DuckDB benchmark failed with `libduckdb.so: cannot open shared object file`, on other branches too. Rust 1.100's cargo moved build-script output from `build/<pkg>-<hash>/out` to `build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB cache two levels above `OUT_DIR`, so the library moved to `build/vortex-duckdb/vortex-duckdb-cache/`, and the hard-coded upload path matched nothing. upload-artifact skipped the missing file silently. The build now searches `target/release_debug/build` for the library and fails if it is missing, because `duckdb-bench` cannot run without it. It loads the library through an absolute rpath, so the archive keeps its relative path. **Pre-merge CI (`ci.yml`)** - Java: a matrix of four jobs. One leg runs `javadoc check` without the Spark compat suites, and one leg runs each Spark compat suite (4.1, 4.2, 4.3). Gradle `--parallel` only splits work across projects, and all three compat suites live in `:vortex-spark-4.0_2.13`, so they ran one after another. A "Java" gate job keeps the required check name. It runs with `always()` and fails if any leg failed or was cancelled, because a skipped required check counts as passed. - License check: `cargo deny check advisories` and `check bans licenses sources` run at the same time. The advisory warning still runs after them, and now has `!cancelled()` so it isn't skipped when both checks fail. - Python (test): after the main suite has built the extension and synced the environment, the datasets-floor, Polars-floor and docs checks run at the same time. - Lightweight checks: the uv setup runs alongside the apt install. **Other PR-path workflows** - `docs-build.yml`: - Rust, JDK and uv install at the same time. - Javadoc (3m41s) builds alongside the Sphinx build (7m15s). - The Javadoc is copied into `docs/_build/html/_static` after the Sphinx build. The docs link to it with raw HTML, and Sphinx only copies `_static` into the output, so the built site is the same. - `publish-dry-runs.yml`, wheel build: setup-prebuild, Zig and uv install at the same time. The Rust cache still waits for setup-prebuild, because its key includes the rustc version. - `web.yml`, `web-build.yml`: Rust, wasm-pack, Bun and Node install at the same time. **Fuzz, package and release workflows** - `run-fuzzer.yml`, `minimize_fuzz_corpus_workflow.yml`: `runs-on/action` and checkout run together, then building or downloading the fuzzer runs alongside the corpus restore. - `fuzz-coverage.yml`: the corpus download (python3 only) runs alongside setup-prebuild. - `close-fixed-fuzzer-issues.yml`: the LLVM setup runs with the cargo-fuzz install. `setup-fuzz-linker` stays on its own, because both it and the LLVM setup may run `apt-get`. - `wasm-fuzz.yml`: the wasm32 build runs with the wabt install. `setup-fuzz-linker` still comes after, because it sets mold `RUSTFLAGS`. - `build-fuzzers.yml`, `package.yml`, `release-binaries.yml`, `publish.yml`, `compat-gen-upload.yml`: `runs-on/action` and checkout run together, and the JDK, Rust, Zig, AWS-credential and artifact-download steps that don't depend on each other run at the same time. **Not changed** - Windows: splitting the wheel build into its own job was tried and reverted. The runner pool keeps one Windows instance warm and a cold one takes 85–105s to boot, which costs more than the 67s it took off the critical path. - Rust tests (cpu-features): "Check CPU features" and the test step compile into the same cargo target directory, so cargo would make them wait for each other. - The CUDA, sanitizer, musl and Rust/Clippy jobs: their time is almost all compiling and running tests. - No matrix caps `max-parallel`, so all matrix legs already run at once. ## Measured | | Before | After | |---|---|---| | SQL bench: package + upload binaries | 64–90s | 12s | | SQL bench: download + unpack, per bench job | 16–20s | 17–31s | | Java, warm Gradle cache | 100s | 81s longest leg + 2s gate | | Java, cold Gradle cache | 546–588s | the compat suites (about 1.5–2 min each) no longer run in series | | License check | 92–110s | 67–84s | | Docs build | 11m33s | 8m39s–10m36s | The `ci.yml` wall time is still set by Windows (about 8.5–10.5 min), so these changes shorten individual jobs, not the whole run. The SQL benchmark changes shorten the bench run directly. On the latest commit the SQL benchmarks all finished, each with a "No clear signal" verdict, as expected for a CI-only change. ## Checks - `yamllint --strict -c .yamllint.yaml .github/` passes. - For every job in every workflow, I parsed the old and new YAML, flattened the `parallel:` groups and compared the steps. They are identical apart from the rewritten Javadoc copy step, the `!cancelled()` on the advisory warning, the Java matrix, and the SQL bench packaging. - I ran Cargo 1.100 locally to confirm the new `OUT_DIR` layout, and checked that the `libduckdb.so` search matches it. - A separate review pass checked each group for shared files, environments, outputs, `PATH`/`GITHUB_ENV` needs and apt-lock conflicts. - CI on the latest commit is green, including all four Java legs, the Java gate, and the integration smoke tests. An earlier smoke-test failure, where the RustFS bucket creation returned an HTTP error after the health check passed, did not recur. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
## Summary TurboQuant stores its Arrow metadata as base64 without an encoding marker. Vortex's fallback for registered Vortex IDs decodes base64 only when marked, so it reads TurboQuant's metadata as literal text and rejects valid arrays, even after a plugin has imported them successfully. An exporter that declines and delegates to storage conversion can hit the same check. ## Changes Successful plugin imports are final because the existing dtype and length checks already validate their result. Export fallback asks the registered importer to decode the field before comparing types, since the same metadata can use different byte encodings. The complete dtype check, apart from outer nullability, and the physical storage check remain. Without a matching importer, the field must contain the source type's serialized metadata. Map entries import as a plain struct because the map's dtype comes from its key and value fields. Running a plugin on the entries field could produce a different dtype from schema inference. Key and value plugins still run. Tests cover native metadata imports and export round trips, invalid metadata, and physical storage mismatches. A byte-collision case checks that equal metadata bytes cannot hide different type parameters. Map tests cover ignored entries plugins and length checks on value imports. ## API Changes Import plugins no longer run on a map's entries field. Extension types on the key and value fields still reach their plugins. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Mutation testing with cargo-mutants (see vortex-data#11471) found code in `vortex-fastlanes` that the tests run but never check. This PR adds tests for two such gaps, keeping only the cases that catch distinct mutants. It changes tests only. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
## Summary Adding a benchmark label (`action/bench-*`) to a stacked PR did nothing, because `pr-bench-dispatch.yml` only triggered for PRs whose base branch is `develop`. ## Changes - Remove the `branches: ["develop"]` filter from the `pull_request` trigger in `.github/workflows/pr-bench-dispatch.yml`. The baseline needs no changes. `scripts/fetch-benchmark-baseline.py` already picks the newest commit reachable from the PR head that has recorded results, and only develop-mode runs write to the benchmarks database. So a stacked PR is compared against `develop`, and the comparison includes every commit in the stack below it. GitHub reads the workflow from the PR's merge commit. An existing stack picks this up only after its parent branch is rebased onto `develop` with this change. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniBC1KY6sxmZBh4hoUwg)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) Several reads and slices binary search a sorted child, such as patch indices or run ends. Some of them built a new searcher, and so a new probe, for every search. RunEnd's probe lookup didn't use the search-sorted API at all: it hand-rolled a binary search that read the ends one scalar at a time. This PR changes only those call sites. Every sorted search now goes through the search-sorted API with one shared probe. That holds even for a single search inside a scalar read: the search reuses the caller's probe instead of building its own. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Robert Kruszewski <github@robertk.io>
…ortex-data#11470) ## Summary In the latest `develop` CodSpeed run ([43502927659](https://github.com/vortex-data/vortex/actions/runs/43502927659)), these benchmarks spent most of their time either building fixtures under the CPU simulation or measuring cases far larger than the rest of their sweep. This PR removes that time without changing what the remaining benchmarks measure. ## Changes - **`vortex-fsst/fsst_take`** (8.5 s in CI) - The fixture searched for a row length that reaches its compressed-size target. It started from an uncompressed estimate and grew the rows 2% at a time, retraining the FSST compressor at every step: 26 to 81 times per case. - It now scales the row length by the compression ratio it just observed, which reaches the target in 2 to 4 compressions. - Every fixture still reaches its target, which is what the compressed-bytes gate under test depends on. The chosen row lengths stay within about 2% of the previous ones, and the codes sizes within about 3%, so expect a small one-time shift on CodSpeed. - **`vortex-file/split_collection`** - The files were written with the BtrBlocks compressor, but split collection walks only the layout tree. They are now written uncompressed. - For every config and the misaligned file, the layout trees and the splits from both `SplitBy::Layout` and `SplitBy::default()` are identical to before. - **`vortex-onpair/decode`** (10.5 s in CI) - The three `(UrlLog, 1_000_000)` cases cost 2.2–2.9 s each. They measure the same kernels as the `(UrlLog, 100_000)` cases at ten times the size. - They now run only locally (`#[cfg(not(codspeed))]`), as `split_collection` already does for its scaling cases. - CodSpeed measures 9 of the 12 benchmarks, and `scripts/codspeed_bench_counts.json` is updated to match. These three benchmarks lose their CodSpeed history. Left alone on purpose: - **`vortex-spatial/binary_predicates`** (4.5 s): locally each benchmark after the first takes about 0.01 s. Its CI cost is the measured iteration under instrumentation, which the file sizes deliberately against its per-iteration budget. - **`vortex-arrow/to_arrow`** (29 s over 130 benchmarks): locally each case takes 0.02–0.04 s including setup, so its CI cost is also mostly measured work. Sharing fixtures between its benchmarks could let one benchmark warm cached statistics for the next. ## Measurements Local `valgrind --tool=callgrind` runs of the CodSpeed builds (`--cfg codspeed`, `VORTEX_BENCH_VARIANT=simulation`). These include setup but not CodSpeed's instrumentation, so they show the setup savings rather than CI times. | Binary | Before | After | |---|---:|---:| | `fsst_take` | 1.84 s | 0.65 s | | `fsst_take`, the 18 `take` cases | 1.14 s | 0.27 s | | `split_collection` `warm` (first to build the file) | 0.11 s | 0.02 s | | `onpair/decode` | 2.07 s | 0.78 s | ## Checks - Built the three benches with `--cfg codspeed`: no warnings. CodSpeed now runs 9 `decode`, 24 `fsst_take` and 8 `split_collection` benchmarks. - Compared both fixture searches for every `fsst_take` case, and compared trees and splits for every `split_collection` file with a temporary debug `main`. - `cargo +nightly-2026-09-10 fmt --check` on the changed bench files. - `pytest scripts/tests/test_codspeed_shards.py` (22 passed). - I did not run clippy or the crate test suites. --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniBAqXtmDDNUP9pHsfSo)_ --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
## Summary - Tracking Issue: vortex-data#11359 `ArrayRef::slice` returns `Canonical::empty(dtype)` for an empty range, and `Canonical::empty` panicked for `DType::Variant`, so slicing any Variant array to an empty range panicked instead of returning an empty array with the same dtype. This is the panic reported in vortex-data#11359, and it also blocks the Arrow round-trip coverage for empty containers with Variant children that vortex-data#11361 works around. ## Changes - `Canonical::empty` now builds a Variant empty instead of panicking. A Variant array needs Variant-typed core storage, and the core storage must not be a `VariantArray`, because `VariantArray::try_new` would call back into `Canonical::empty`. The empty is a zero-length `ConstantArray` holding a null Variant scalar; a non-nullable Variant empty casts from the nullable one, which is safe because there are no null values to drop. Signed-off-by: ump45nose <52391318+ump45nose@users.noreply.github.com>
This PR contains the following updates: | Update | Change | |---|---| | lockFileMaintenance | All locks refreshed | 🔧 This Pull Request updates lock files to use the latest dependency versions. --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - Between 12:00 AM and 03:59 AM, only on Monday (`* 0-3 * * 1`) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://redirect.github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://mend.io/renovate/). View the [repository job log](https://developer.mend.io/github/vortex-data/vortex). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4yMjguMCIsInVwZGF0ZWRJblZlciI6IjQ0LjIyOC4wIiwidGFyZ2V0QnJhbmNoIjoiZGV2ZWxvcCIsImxhYmVscyI6WyJjaGFuZ2Vsb2cvY2hvcmUiXX0=--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…data#11410) ## Summary `VortexDataset.take` failed with "buffer values must be strictly increasing" for `take([7, 2])` and `take([7, 2, 7])`, while pyarrow's `Dataset.take` returns the rows in the requested order. ## Changes - When the indices are not strictly increasing and no row filter applies, scan the sorted distinct rows once, then gather them back into the requested order with Vortex `Array.take`. Sorted input keeps the single-scan path. - With a filter (the `filter` argument or `Dataset.filter`) the surviving rows cannot be mapped back to positions, so strictly increasing indices are still required. The docstring says so. --------- Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…ortex-data#11384) ## Summary `VortexDataset.head(n)` raised `RuntimeError: OutOfBounds: stop 10 > length 5` whenever fewer than `n` rows matched, where a pyarrow Parquet dataset returns the rows it has: `head(10)` on a 5-row file, or `head(10, filter=pc.field("x") > 3)` matching one row. `ds.scanner(batch_size=N).head(k)` always raised `ValueError: batch_size is not supported`, although `to_batches` takes `batch_size`. ## Changes `head` now reads through the record batch reader that `to_batches` uses, with `num_rows` as a new scan `limit`. A short result is returned as is, and the scan stops once it has enough rows: on a 20M-row file, `head(10, filter=pc.field("i") > 5)` grows peak RSS by 20 MiB instead of 109 MiB. --------- Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
Ray Data could read Vortex files but had no distributed write path. This adds `VortexDatasink`, so `ds.write_datasink(VortexDatasink(path))` streams each write task to one Vortex file instead of collecting data on the driver or switching formats. - Tracking Issue: vortex-data#11414 Signed-off-by: jackylee-ch <qcsd2011@gmail.com>
…1436) Add a Pixi environment with the dependencies needed to build and test CUDA/cuDF C++ code. - Include cuDF, GTest, CMake/Ninja, and C++/CUDA compilers. - Check in a lockfile for Linux x86-64 and AArch64, with CUDA pinned to 13.0 for CI driver compatibility. - Document how to install Pixi and enter the environment. This PR only adds the environment and documentation. AMI installation and wiring integration tests into CI are separate changes. --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
`cargo-mutants` skips macro bodies, which hides VarBin’s choice between validating the whole byte range and checking non-null strings individually. A generic helper exposes that decision and the per-string loop to mutation testing. Follow-up to vortex-data#11605. Stacked on vortex-data#11606. Prepared and opened with Codex. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
`cargo-mutants` skips macro bodies, which hides the choice between chunked and flat patch searches. Generic helpers expose that decision and the flat search’s offset handling to mutation testing. Follow-up to vortex-data#11605. Stacked on vortex-data#11607. Prepared and opened with Codex. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The file opener and the Scan API adapter both convert projected Vortex chunks into Arrow record batches. Sharing that conversion keeps the two paths consistent while each caller still chooses its target schema and runs the conversion on its existing CPU task. Stacked on vortex-data#11557. Part of vortex-data#2254. This change was prepared with assistance from OpenAI Codex. Signed-off-by: Nicholas Gates <nick@nickgates.com>
- Tracking Issue: vortex-data#10733 All DecimalByteParts v2 data lives in its children. The reader currently accepts and ignores buffers attached to the node itself. Reject those buffers before v2 enters preview so the reader enforces the same shape the writer emits. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
) ## Summary - Tracking Issue: vortex-data#11394 `IS [NOT] DISTINCT FROM` filters fail against a Vortex scan in DuckDB with `Binder Error: cannot convert DUCKDB_VX_EXPR_TYPE_COMPARE_DISTINCT_FROM`. So do `IS [NOT] TRUE` and `IS [NOT] FALSE`, which DuckDB rewrites to them. `can_push_expression` accepts every `BoundComparison`, but the converter only handles the six operators in `Operator`. This lowers the two operators in the converter so the filter is evaluated by the scan. ## Changes - `vortex-duckdb/src/convert/expr.rs`: lower `IS NOT DISTINCT FROM` as `(l IS NULL AND r IS NULL) OR (l IS NOT NULL AND r IS NOT NULL AND l = r)`, and `IS DISTINCT FROM` as its negation. Unlike `=`, the result is never NULL, so it is correct under `NOT` and `OR`. - `vortex-sqllogictest/slt/is_distinct_from.slt`: covers a constant, `NULL`, column against column, `NOT (...)` and `IS [NOT] TRUE`. On DuckDB it also asserts through `EXPLAIN` that no `FILTER` node remains. --------- Signed-off-by: Ayush Mishra <ayushm1906@gmail.com>
…#11611) Use the shared driver for the DataFusion V2 file opener and remove its duplicate scheduler. The opener still waits for scan preparation so failures are reported as file-open errors, and Arrow conversion still runs on the task that produces each batch. When several chunks run on the same scheduler, such as with `scan_concurrency=0`, each chunk needs its own queue for pending reads. The shared driver used to give these chunks the same I/O session, so when one chunk finished and cleared the session, it could discard another chunk’s pending reads. Give each planner root its own session over the same file source, preserving the isolation DataFusion already had for inline chunks. Stacked on vortex-data#11610. Part of vortex-data#9062. This change was prepared with assistance from OpenAI Codex. --------- Signed-off-by: Nicholas Gates <nick@nickgates.com>
- Tracking Issue: vortex-data#10733 The existing decimal wire format can preserve narrow storage only when `Widen` wraps a decimal leaf. Integer arrays and decimals with encoded children need a wire ID that carries the child dtype so readers can interpret the narrow storage directly. No edition permits `vortex.widen` yet. A decimal leaf still writes `vortex.decimal`, even when the target permits `vortex.widen`, because `ArrayPlugin::serialize` does not see the permitted IDs and so cannot choose the ID by edition. Those arrays stay readable under existing editions, but their reads keep the range scan of the legacy decimal reader. When the target does not permit `vortex.widen`, serialization returns an error instead of writing the canonical array, as `develop` already does for these arrays. This change was prepared with AI assistance. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…#11617) - Tracking Issue: vortex-data#11114 Vortex declares the nullability of every nested level, and a non-nullable Vortex array has no validity at all. Arrow is looser: a level whose field is non-nullable can still carry null bits, for example under a null parent row, and the declared storage of an extension type can be stricter than the Arrow fields it imports from. So the import has to put a real value in each of those slots: ```text Arrow Vortex struct (nullable) [ valid, null ] struct (nullable) [ valid, null ] └── x (not null) [ 1, null ] └── x (not null) [ 1, 0 ] ``` vortex-data#11223 and vortex-data#11497 handled this by passing a mask of hidden slots down the tree, dropping a null that a null ancestor hides and failing on any other. The mask only went through structs, lists, and fixed-size lists, so the same `x` below a list view, a map, a run-end array, or a dictionary failed to import, and the `vortex_arrow::convert` functions never used the mask at all. vortex-data#11602 extended it to the other levels, which showed how much the approach costs: list views can overlap, a run-end value covers many rows, and many dictionary keys share one value, so each level needs its own rule for mapping rows to children. This drops the mask, and each level now only looks at its own declared nullability. A non-nullable leaf fills its nulls with the zero of its storage type, and a non-nullable container drops its validity and keeps its contents, so its children follow their own fields. Both import paths share this rule. Importers still get the raw Arrow array. Most go through `from_arrow_storage`, which applies the rule with the nullability of the storage dtype, and the UUID and Parquet Variant importers, which convert validity themselves, now use the new public `level_validity`. A non-nullable dictionary points its null keys at entry zero, and a dictionary with keys but no values (as `new_null_array` builds) gets one entry by importing a single null row of the value type under the same rule. Behavior changes: - A null in a valid row of a non-nullable level now imports as zero instead of failing, so a null element in a valid `Vector` row becomes `0.0`. - Inside extension storage, the fill uses the storage zero without asking the extension, so a non-nullable JSON field under a null row holds `""`. - `vortex_arrow::convert` now imports run-end arrays, since it shares the run-end import with the session. Replaces vortex-data#11602. AI assisted with this change. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
- Tracking Issue: vortex-data#11114 A cast between an extension dtype and any other dtype reinterprets the storage instead of failing, so `Timestamp(ms)` casts to `u8`, and a constant `Timestamp(ms)` of 1000 casts to `Timestamp(ns)` of 1000, which turns one second into one microsecond. A conversion that changes what a value means belongs to the type, so the generic cast now only changes the top-level nullability or builds typed nulls from an untyped `Null`, and code that wants the raw storage reads it with `ext_storage`. `Cast::return_dtype` checks the rule, so a bad cast fails when the expression is built instead of failing in one kernel but not another. Breaking: a cast from or to an extension dtype, including one nested in a struct, list, or map, fails unless it only changes nullability, so serialized expressions that hold such a cast no longer load. Breaking: `Cast::new` is now `Cast::try_new`. Breaking: an Arrow export of an extension array fails when the target neither names the type nor has an Arrow type that imports as it, such as `Int64` for a timestamp, a nested field without metadata, or the target of `Datum::try_new_with_target_datatype`. Apply `ext_storage` to export the storage. Breaking: Python no longer builds extension scalars or literals from raw integers. Prepared with AI assistance. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
…a#11615) vortex-data#11595 removed `vortex.decimal_byte_parts.v2` from the `unstable` edition because `between` and aggregates decode to canonical form when lower parts exist. That is a performance gap, not a wire problem, and `unstable` has no compatibility guarantee, so the encoding goes back in and the decompression kernels can follow. `vortex.widen` gets a wire ID in vortex-data#11597. Adding it to `unstable` lets writers that enforce editions emit integer `Widen` arrays and decimal `Widen` arrays over encoded children. Without it, those writes return an error. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
## Summary - Tracking Issue: vortex-data#11420 A buffer that grows by appending can have up to 2x its length in capacity, and `freeze` keeps all of it. This adds `BufferMut::shrink_to_fit`, which uses `realloc` to release the spare capacity in place, so callers such as `FsstSink::finish` can trim a buffer before they freeze it. - `Allocation::shrink` forwards to `Allocator::shrink`. If the allocator refuses, the allocation does not change. - `shrink_to_fit` keeps the data at its offset, and moves it back to an aligned offset if `realloc` misaligns it. A sliced buffer is copied once into a new allocation. An empty buffer becomes unallocated. A second call does nothing. - Tests use an allocator that moves each shrunk block to a misaligned address. New `shrink_to_fit` and `shrink_by_copy` benchmarks are in `benches/allocation.rs`. On glibc, `shrink_to_fit` is 30–550x faster than a copy (for example, 4 MiB: 663 ns vs 366 µs). mimalloc copies on `realloc`, so there it costs the same as a copy. ## API Changes Adds `BufferMut::shrink_to_fit`. After a shrink, the data has only the buffer's own alignment, not the `DEFAULT_ALIGNMENT` over-alignment. Growth already has the same guarantee. --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Robert Kruszewski <github@robertk.io>
…city (vortex-data#11434) ## Summary - Tracking Issue: vortex-data#11420 `fsst_compress` builds the codes buffer by appending one batch at a time, and `BufferMut` growth at least doubles the capacity. `freeze` keeps that allocation, so a compressed FSST array could hold up to about 1.9x its `nbytes()`, depending on where the last batch landed relative to the last doubling. OnPair has the same gap: `onpair::compress_rows` grows the codes `Vec` from empty, and `Buffer::from(Vec)` keeps its whole allocation. ## Changes - FSST: at the end of `FsstSink::finish`, when the codes buffer's spare capacity is more than a quarter of its length, copy the codes into an exactly sized buffer (same alignment, the context's allocator) before freezing. Otherwise the buffer is frozen as before. - OnPair: in `onpair_compress`, call `shrink_to_fit` on the codes under the same condition, before they become a `Buffer`. The dictionary bytes were already copied into an exactly sized buffer. - Add a `codes_buffer_has_little_spare_capacity` `rstest` regression to each crate, over 8,192 random hex strings of several lengths. Each one unwraps the uniquely owned codes buffer with `try_into_mut` and checks that the spare capacity is at most a quarter of the length. Both fail on `develop` and pass with this change. --------- Signed-off-by: Shu Chen <338845574+shu-chen-dot-ca@users.noreply.github.com> Co-authored-by: Robert Kruszewski <github@robertk.io>
…tex-data#11430) ## Summary Follow-up to vortex-data#11418 and vortex-data#11423. The `#[array_slots]` accessors now return references borrowed for the array, or for the `ArrayView`'s lifetime `'a`. So there's no longer a reason to index `slots()` by `XSlots` constants and unwrap by hand, or to keep hand-written accessors that duplicate the generated ones. ## API Changes - Removed `PrimitiveArrayExt::validity_child`, `DecimalArrayExt::validity_child`, `InterleaveArrayExt::array_indices`, `InterleaveArrayExt::row_indices` and `ChunkedArrayExt::chunk_offsets_array`. The generated accessors replace them, so callers need the encoding's `…ArraySlotsExt` trait in scope. Use `chunk_offsets()` in place of `chunk_offsets_array()`. - Newly public: `InterleaveSlots`, `InterleaveSlotsView` and `InterleaveArraySlotsExt` from `arrays::interleave`, and `ChunkedArraySlotsExt` from `arrays::chunked`. Signed-off-by: Robert Kruszewski <robert@spiraldb.com>
## Summary Baseline benchmarks for taking `Patches` with `PiecewiseSequenceArray` indices, ahead of the stacked follow-up vortex-data#11547, which takes the runs without expanding them. Landing the benchmarks first gives that change a recorded baseline to compare against. ## Changes - Adds `take_piecewise` to `vortex-array/benches/take_patches.rs`. It takes patches at density 0.1 with ascending runs of contiguous indices, covering half of a 4096-row array, for run lengths 1, 16 and 256. - Inputs go through `mark_optimized`, like the existing benchmarks in this file. --------- Signed-off-by: Robert Kruszewski <github@robertk.io>
Add a `filter_bool` benchmark for validity bitmaps. Validity bitmaps are mostly set, even when the selection is random. The existing sweeps use only random source bitmaps. - Add `density_sweep_validity` at 4 densities: 0.02, 0.1, 0.5, and 0.99. Signed-off-by: Robert Kruszewski <github@robertk.io>
The dynamic dispatch kernel unpacks each 1024-element FastLanes chunk with one thread per lane, but 32- and 64-bit chunks have only 32 and 16 lanes, so half or three quarters of each 64-thread block waited at the barrier during the unpack. This splits those lanes into 16-row slices so every thread decodes one, and spreads the work over all chunks of a stage. The generator emits the per-row decode code once, as slices, and builds the whole-lane decoders used by the standalone kernels from them. Fused u32 and u64 kernels in `dynamic_dispatch_cuda` are 11–19% faster, e.g. `alp_for_bp_6bw_f64` at 100M elements goes from 369 µs to 299 µs. Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…x-data#11634) `CudaExecutionCtx::copy_to_device` and `ensure_on_device` waited for a stream callback after enqueueing each copy. That callback fires only once *all* earlier work on the stream has finished, so every small host-to-device copy during decode and Arrow export (e.g. FSST symbol tables, binary-export pointer tables) stalled the host until the GPU drained, leaving the GPU idle while the host queued the next kernels. The wait isn't needed: later work on the same stream is already ordered after the copy, and readers on other streams wait for the buffer's recorded write. Both methods now return the buffer as soon as the copy is enqueued. `DeviceReadAt`, which hands buffers across streams, keeps the waiting copy. Measured end to end with BenchPress (TPC-H SF10 `lineitem`, GPU read through the CUDA FFI, hot page cache, 5 iterations, two runs each): | | Before | After | |---|---:|---:| | Read time | 41.85 ms | 40.25 ms | | Throughput | 196.0 GB/s | 203.9 GB/s | Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
…ortex-data#11631) The fused kernel runs one block per 2048 output elements, and every block first decodes the plan's input stages (dictionary values, run ends) into its own shared memory. With large dictionaries or many runs, most of the kernel's time went into decoding the same input stages over and over. This adds a `loop_ranges` kernel that launches only as many blocks as fit on the GPU at once and loops each over several 2048-element ranges. Each block decodes the input stages once and reuses them for all its ranges, cutting redundant decodes by roughly 12× at 100M elements. Plans with fewer than 64 input elements keep the original kernel, where the decode is cheap and looping doesn't pay off. `dynamic_dispatch_cuda` at 100M elements: | Benchmark | Before | After | |---|---:|---:| | `dict_4096vals_bp12bw_codes/dispatch_u32` | 1142.8 µs | 673.9 µs | | `dict_4096vals_bp12bw_codes/dispatch_u64` | 2348.7 µs | 1377.9 µs | | `runend_1024runs/dispatch_u32` | 370.7 µs | 206.6 µs | | `dict_1024vals_bp10bw_codes/dispatch_u32` | 341.9 µs | 312.2 µs | --------- Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Moves the segment types and readers into a new `vortex_io::segments` module: `SegmentId`, `SegmentSource` and `SegmentLocation` from vortex-scan, the coalescing `FileSegmentSource` with its read driver, `BufferSegmentSource` and `RequestMetrics` from vortex-file, and `SharedSegmentSource` from vortex-layout. The I/O layer can then read a file's segments itself, which vortex-data#11623 needs for its session-wide scheduler, without depending on vortex-file or vortex-layout. Every item stays re-exported from its old path, and behaviour is unchanged. The only edits beyond imports: `FileSegmentSource` and `BufferSegmentSource` take `SegmentLocation` instead of vortex-file's `SegmentSpec` (same fields), so `Footer::segment_locations()` now does the conversion that `VortexFile` did; `SegmentLocation::target()` becomes a private free function in vortex-scan because the type now lives in another crate; and two casts and a test `cfg` change to satisfy vortex-io's stricter lints. Written with Claude Code. The vortex-io, vortex-scan, vortex-layout and vortex-file tests and clippy pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Nicholas Gates <nick@nickgates.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
On `develop`, a V2 scan that reads one field from a nullable struct can still plan every child. The scan fails if a sibling uses a layout that V2 cannot execute. This PR plans only the requested fields and the validity needed to preserve their nulls. For `address.zip`, the scan can skip `address.city` and an unused sibling list. A null `address` still produces a null `zip`. Related to vortex-data#10952, which handles deep projections in the classic reader. This change was implemented and reviewed with OpenAI Codex. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Delta's serialized form was settled in vortex-data#11441, and with the fused decode and kernels (vortex-data#11428) and the sampled estimator (vortex-data#11453) it is cheap enough to offer for broad opt-in use. This moves `fastlanes.delta` from `unstable` into the draft `preview2026.08.0` edition, so Delta's wire format becomes fixed when that edition freezes. Writers that enable only `core` still never write Delta. Benchmark sessions now enable `preview` as well as `unstable` through `enable_unstable_encodings`, so every benchmark keeps writing Delta. This also enables `preview`'s other member, the `vortex.uncompressed_size_in_bytes` aggregate, in benchmark files. PR benchmarks against `develop`, which already writes Delta through `unstable`, are flat. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…11151) A malformed `batch.size` or `vortex.write.batch.size`, a negative `vortex.workerThreads`, or a `vortex.session.provider` class that can't be used fails only inside write tasks. On a V1 overwrite, Spark has already deleted the old files by then. The connector also ignores `vortex.workerThreads` once the JVM's worker pool is sized. --------- Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com> Signed-off-by: Robert Kruszewski <github@robertk.io>
Part of vortex-data#11516 This is the first step toward an ALP-RD GPU kernel, as @0ax1 suggested on the issue. The CUDA patch loader needs `chunk_offsets` to find a chunk's patches in constant time, but the ALP-RD encoder never wrote them, and the deserializer dropped them on read. Both spots had a `TODO(0ax1)`. New ALP-RD files now carry one extra child holding the offsets. The compressor narrows it, so it's 16 bytes for 16k rows in the btrblocks golden snapshot. Older files without it read the same as before, and older readers simply ignore the extra child. Could a maintainer add the `changelog/feature` label? I can't set labels from a fork. I used Claude Code for analysis, debugging, and ideas. Signed-off-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com> Co-authored-by: Vishal Kotak <22182951+vishalkotak@users.noreply.github.com>
…a#11682) For `col = 'x'` on a `RunEndEncoded<_, Utf8>` column, DataFusion keeps the column type and wraps the literal in REE, so there is no cast for vortex-data#10637 to drop, and the converter's supported-type check rejected REE. The check now looks at the REE values type, as it already does for dictionaries, so these filters run inside the scan instead of in a `FilterExec` above it. Two REE shapes stay in DataFusion. Arrow's comparison kernels unwrap run-end encoding only as the outermost layer, so REE nested in REE or in a dictionary is declined. DataFusion reads the nulls of a run-end encoded `IN` list from the `RunArray`'s physical validity, which is always empty, so for a value not in `(a, b, c, NULL)` it returns false where SQL returns NULL; such lists are declined so both engines keep the same rows. The new test compares Vortex with DataFusion for 26 value types on plain, dictionary and REE columns, using four-value `IN` lists because DataFusion rewrites shorter column `IN` lists into comparisons. Related: vortex-data#10637 Signed-off-by: Ivan Kiselev <ivan.kiselev@dash0.com>
…ta#11630) Every fused dynamic dispatch launch allocated a device buffer for its 100–200 byte plan, copied the plan into it from pageable host memory, and tracked the buffer until the kernel finished. This change now passes the plan by value as a 1 KB `__grid_constant__` kernel parameter instead; plans that don't fit fall back to unfused dispatch, like plans over the shared memory limit. The kernel now also reads the plan, including the scalar ops it re-reads on every tile, from the constant cache instead of global memory. Total time per fused dictionary decode (host wall time including the kernel): | Elements | Before | After | |---:|---:|---:| | 16K | 20.3 µs | 16.4 µs | | 256K | 20.7 µs | 16.7 µs | | 4M | 24.6 µs | 21.2 µs | Kernel time in `dynamic_dispatch_cuda` (100M elements, plan uploaded outside the timed region): | Benchmark | Before | After | |---|---:|---:| | `for_bitpacked_6bw/dispatch_u32` | 210.9 µs | 159.5 µs | | `dict_256vals_bp8bw_codes/dispatch_u32` | 240.4 µs | 184.0 µs | | `dict_nowiden_u32_to_u32/dispatch_u32` | 183.0 µs | 146.9 µs | | `alp_for_bp_6bw_f32/dispatch_f32` | 228.4 µs | 180.8 µs | | `alp_for_bp_6bw_f64/dispatch_f64` | 369.3 µs | 360.8 µs | | `alp_for_bp_6bw_f64_1pct_patches/dispatch_f64` | 452.2 µs | 424.1 µs | Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
- Tracking Issue: vortex-data#11115 Functions use the storage of any extension type today, so comparing two `Json` values compares their strings. A type needs a way to promise that its storage answers equality or order correctly before a function can trust it. This PR adds only the declaration, and vortex-data#11600 makes each function check it. `ExtVTable::claims` returns an opaque `Claims` value. A `&'static [Claim]` slice was the alternative, but the slice could disagree with `DType::holds` about `Eq`, it had several values for the same promise (`[Ord]` and `[Eq, Ord]`), and it could only name properties. Private fields leave room for a custom order key later. AI assisted with this change. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Arrow export serializes struct fields: each child is decoded and exported before the next one starts. When a field triggers a device-to-host readback (string sizes, run-end totals, FSST symbol tables), the GPU idles while the host waits for the result, and no other field's work can fill that gap. Each field now exports on its own CUDA stream via a forked execution context. The struct's stream waits for every field event before recording the final sync, so downstream consumers still see a single ordered export. Error and cancellation cleanup uses per-field `PendingArrowChildren` guards. Measured end to end with BenchPress (TPC-H SF10 `lineitem`, GPU read through the CUDA FFI, hot page cache, 5 iterations, two runs each): | | Before | After | |---|---:|---:| | Read time | 41.85 ms | 38.85 ms | | Throughput | 196.0 GB/s | 211.4 GB/s | Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Merge vortex-data/vortex develop at 79e75c1. Conflict resolutions: - zone_map.rs: keep the fork's large IN-list pruning tests; upstream moved the other zone map tests to vortex-scan. - plan/tests.rs: take upstream; port the fork's fallible ListLayout::try_new to the moved lowering tests. - VortexSparkSession: take upstream's vortex-data#11151, which supersedes the fork's version of the same change. - VortexOutputWriterFactory: keep the fork's public validate covering batch.bytes, capacity reset and listLayout. - VortexWriteOptionValidationTest: upstream V1/V2 structure with the fork's option cases; out-of-range batch sizes stay rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Christopher Wirt <cwirt@quantcast.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merge
vortex-data/vortex:developat79e75c1d0a080136254734366de4ee7d761717abinto QCdevelop, preserving upstream ancestry. This brings in 128 upstream commits, in two merges:8434405de, no conflicts.79e75c1d0, with the conflicts resolved below.Changes
vortex-scan, experimental index/table crates, bulk Java list literals, Python dataset fixes, and a Ray Data sink.Conflict resolutions (2026-11-20)
zone_map.rs: kept the fork's large IN-list pruning tests. Upstream moved the other zone map tests tovortex-scan.vortex-scan/src/plan/tests.rs: took upstream's version. Ported the fork's fallibleListLayout::try_newto the movedvortex-layout/src/lowering/tests.rs.VortexSparkSession: took upstream's fix(spark): reject unusable write options on the driver vortex-data/vortex#11151, which replaces the fork's earlier version of that change. A provider class the driver can't load is now left for the executors to load.VortexOutputWriterFactory: kept the fork's publicvalidate, which also checksbatch.bytes, capacity reset andlistLayout.VortexWriteOptionValidationTest: uses upstream's V1/V2 test structure with the fork's option cases. Out-of-range batch sizes are still rejected, as on the fork; upstream falls back to the default instead.Testing
nightly-2026-11-13rustfmt onvortex-layout/vortex-scanandspotlessApplyon the Spark module. Neither produced changes.🤖 Generated with Claude Code