Skip to content

Add a layout for struct-backed extensions - #11640

Open
connortsui20 wants to merge 1 commit into
developfrom
ct/extension-storage-layout
Open

connortsui20 wants to merge 1 commit into
developfrom
ct/extension-storage-layout

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

This PR adds vortex.extension, an experimental layout for struct-backed extension types. On develop, the file writer stores these types in a leaf layout, so a scan reads the whole value even when it needs only one storage field. The new layout keeps the extension dtype and wraps a struct layout that stores the fields separately.

For a Point backed by {x, y}, get_item("x", ext_storage(root())) can read x without reading y, and a filter on x can use its zone map. A nullable Point gets field zone maps once #11641 lands, as any nullable struct does. A whole-value read still returns a Point.

The default writer requires the unstable edition and per-type opt-in. Splitting the storage removes whole-value zone maps, so each type chooses whether to use the new layout. Prototype design.

ForeignExtDType is public under the _test-harness feature, so that the layout and file tests can build extension dtypes over any storage dtype without a vtable for each shape.

This change was implemented and reviewed with OpenAI Codex and Claude Code.

@connortsui20 connortsui20 self-assigned this Nov 19, 2026
@connortsui20 connortsui20 changed the title Write opt-in extension storage as a structural layout Write opted-in extensions with independently readable fields Nov 19, 2026
@connortsui20 connortsui20 changed the title Write opted-in extensions with independently readable fields Add a layout for struct-backed extensions Nov 19, 2026
@connortsui20
connortsui20 force-pushed the ct/extension-storage-layout branch 3 times, most recently from 36f0650 to 026f828 Compare November 20, 2026 20:36
@connortsui20
connortsui20 marked this pull request as ready for review November 20, 2026 20:40
A struct-backed extension written as a leaf makes a field query load every storage field. Let types opt into a structural layout so explicit storage reads can use the existing struct reader and planner without losing the extension dtype.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/extension-storage-layout branch from 026f828 to 24fd9da Compare November 20, 2026 21:12
@codspeed

codspeed Bot commented Nov 20, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 14.56%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

❌ 1 regressed benchmark
✅ 1981 untouched benchmarks
⏩ 58 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime scalar_subtract_neon 12.6 µs 14.7 µs -14.56%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/extension-storage-layout (24fd9da) with develop (967bcb6)

Open in CodSpeed

Footnotes

  1. 58 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 added a commit that referenced this pull request Nov 20, 2026
The branch carried the prototype versions of the changes that the open
PRs make, and those PRs have moved on in review. For example, #11640
now names the layout `vortex.extension`. This merge takes the tree of a
stack on develop at 967bcb6, which holds the PR heads and ports of
the two changes that have no PR yet:

- #11640 at 24fd9da.
- #11710 at add4115.
- #11639 at 4f25e2f. Its newer head, 115318b, fails three of its
  own tests.
- #11642 at d16038c.
- Zone maps of bounds beneath nullable structs.
- Single fields of list elements on both scans.

A merge instead of a force-push keeps every commit that the branch
published reachable.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch was successfully deployed

1 active deployment
vercel-docs — 24fd9daa Deployed Nov 20, 2026 by connortsui20 via Publish preview / Deploy #11754
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant