Skip to content

Validate List and ListView arrays with an explicit execution context - #10594

Merged
robert3005 merged 3 commits into
developfrom
rk/legacy-session-list
Oct 20, 2026
Merged

robert3005 merged 3 commits into
developfrom
rk/legacy-session-list

Conversation

@robert3005

@robert3005 robert3005 commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor
  • ListArray::{new, try_new} and ListViewArray::{new, try_new} take an ExecutionCtx. They validate offsets and sizes (sorted and in bounds) with it, instead of with legacy_session().
  • Deserialization validates with a context from its session.
  • These accessors take a ctx:
    • List and ListView: offset_at, size_at, list_elements_at, sliced_elements
    • Map: entries_at, entry_count_at
  • These build valid lists by construction and use the unchecked constructors, with a SAFETY comment:
    • the List builder
    • the Arrow list, list-view and map imports, which rely on Arrow's offset invariants
    • the DuckDB list import
    • the spatial storage builders
  • Plugin-aware Arrow imports check that an ArrowImportVTable returns as many elements as it was given. The list, list-view and map imports reuse Arrow's offsets into the imported child, so a plugin that changed the child's length would otherwise produce an invalid list.
  • Copying a CUDA list view to the host keeps the checked ListViewArray::try_new, now with an explicit context. Device-resident offsets and sizes skip range validation when the list is built, so this is where invalid ranges are caught.

part of #9446

@robert3005 robert3005 added the changelog/break A breaking API change label Oct 19, 2026
@robert3005
robert3005 marked this pull request as ready for review October 19, 2026 15:36
@robert3005
robert3005 force-pushed the rk/legacy-session-varbin branch from 7281c06 to 8105637 Compare October 20, 2026 11:25
@connortsui20
connortsui20 self-requested a review October 20, 2026 11:28
@robert3005
robert3005 force-pushed the rk/legacy-session-varbin branch from 8105637 to cbb2829 Compare October 20, 2026 11:42
Base automatically changed from rk/legacy-session-varbin to develop October 20, 2026 11:59
The List and ListView constructors validated their offsets and sizes, and
`offset_at`, `size_at`, `list_elements_at` and `sliced_elements` read
them, with the hidden global `legacy_session()`. Take an `ExecutionCtx`
instead, as do Map `entries_at` and `entry_count_at`, and validate during
deserialization with a context from the session.

The List builder, the Arrow and DuckDB list imports and the spatial storage
builders build valid lists by construction and use the unchecked
constructors.

Plugin-aware Arrow imports now check that an importer preserves the length
of the array it imports, since list, list view and map imports reuse Arrow's
offsets into the imported child without checking them again. Copying a CUDA
list view to the host keeps the checked constructor, because device-resident
offsets and sizes skip range validation when the list is built.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1zHpE1K1Y9HvyU4kt24Tq
@robert3005
robert3005 force-pushed the rk/legacy-session-list branch from 931c193 to 5389bac Compare October 20, 2026 12:20
@robert3005
robert3005 enabled auto-merge (squash) October 20, 2026 12:24
@codspeed

codspeed Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 4 benchmarks

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 19 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

ℹ️ 40 benchmarks have no flame graph

The flamegraph could not be generated because we failed to resolve the symbols in the profiling trace. This happens when we fail to harvest information about language frames.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 8 improved benchmarks
❌ 4 regressed benchmarks
✅ 2396 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation bench_compare_sliced_dict_primitive[(2000, 10000)] 13.9 µs 68 µs -79.57%
❌ Simulation density_sweep_dense_runs[0.001] 44.2 µs 62.7 µs -29.41%
❌ ⚠️ Simulation test_repeated_scan_scalar_at[int] 244.2 µs 279 µs -12.48%
❌ WallTime words_gather_dispatch_neon[65536] 1.8 µs 2 µs -10.01%
⚡ Simulation column_x_constant 580.4 µs 153.3 µs ×3.8
⚡ Simulation column_x_column 169.2 µs 107.9 µs +56.92%
⚡ Simulation nullable_columns 183.6 µs 124.5 µs +47.5%
⚡ ⚠️ Simulation random_i128[0.01] 5.9 µs 4.4 µs +34.79%
⚡ WallTime bitpack_blocked_compress_avx2 7.6 µs 6.7 µs +13.11%
⚡ Simulation eight_points 639.9 µs 573.6 µs +11.55%
⚡ ⚠️ Simulation test_scan_filter[int] 3.4 ms 3.1 ms +11.51%
⚡ Simulation sixty_four_points 711 µs 639.8 µs +11.12%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing rk/legacy-session-list (68b6865) with develop (7216cb5)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

The C++ field accessor no longer takes a session (#10611).
Same change as #10638.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1zHpE1K1Y9HvyU4kt24Tq
A plugin that declines an import hands back an array that the canonical
conversion then imports. Check that it kept the original length too, since
list, list view and map imports reuse the Arrow offsets into it.

Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y1zHpE1K1Y9HvyU4kt24Tq
@robert3005
robert3005 merged commit c48457b into develop Oct 20, 2026
88 of 89 checks passed
@robert3005
robert3005 deleted the rk/legacy-session-list branch October 20, 2026 13:03
mhk197 added a commit that referenced this pull request Oct 23, 2026
## Summary

Develop doesn't compile vortex-file's tests. #10533 added
`nested_empty_list_roundtrip`, which calls `ListArray::try_new` with
three arguments, but #10594 had already added a fourth, the execution
context. Both passed CI on their own. Since #10533 merged, the Rust
build, lint and test jobs fail on develop and on every PR.

## Changes

- `nested_empty_list_roundtrip` creates an execution context and passes
it to both `ListArray::try_new` calls, as
`nested_list_of_list_roundtrip` just above it already does.

## Testing

- `cargo nextest run -p vortex-file -E
'test(/nested_empty_list_roundtrip|nested_list_of_list_roundtrip/)'` (5
passed)
- `cargo clippy -p vortex-file --all-targets --all-features -- -D
warnings`
- `cargo +nightly-2026-09-10 fmt -p vortex-file --check`

Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants