Skip to content

[DO NOT MERGE] Research notes on extension types and casts - #10633

Draft
connortsui20 wants to merge 61 commits into
developfrom
ct/ext-types-research
Draft

connortsui20 wants to merge 61 commits into
developfrom
ct/ext-types-research

Conversation

@connortsui20

@connortsui20 connortsui20 commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Research notes on the Vortex type system: extension types and casts. This PR exists so the notes can be read and discussed on GitHub. It is not meant to merge.

For review, start with research/decisions.md. It records the settled decisions and the open questions, each with a recommendation, and the other notes follow it. The topic has a short summary (research/extension-types/) and an agents/ directory with the detailed report, file and line references against develop at 5fc5cb8, and reproductions.

The main decisions so far:

  • Extension types are opaque by default. A type implements its behavior as rewrite rules on arrays, registered in the session with the type, and a rewrite rule lifts generic structural arrays such as Constant and Dict out from under the extension wrapper.
  • ExtensionArray::try_new checks values, and a new new_unchecked serves kernels.
  • A cast exists only where a rule for that pair of dtypes exists.

The Decimal notes, Widen plan, and type roadmap moved to #11237.

Rendered

@connortsui20 connortsui20 added the do not merge Pull requests that are not intended to merge label Oct 20, 2026
@vortex-data vortex-data deleted a comment from codspeed Bot Oct 23, 2026
@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 2318 untouched benchmarks


Comparing ct/ext-types-research (fa50f1f) with develop (996bb93)

Open in CodSpeed

connortsui20 added a commit that referenced this pull request Oct 26, 2026
## Summary

- Closes #10750
- Replaces #10746

The docs on `Validity::cast_nullability` read as if every `CastReduce`
rule that removes nullability must call
`Validity::trivially_cast_nullability` and pair with a `CastKernel`.
Most push-down rules (FoR, ALP, ALP-RD, ZigZag, RLE, RunEnd,
DateTimeParts, DecimalByteParts, Chunked, Sparse, Extension) instead
cast the child that holds their validity, and the cast of that child
already fails on nulls, so no result is wrong. Guarding each rule, as
#10746 does for DecimalByteParts, adds a kernel per encoding and delays
the rewrite until execution without changing any result.

The guard in #10746 comes from the closed #10230, where a canonical
decimal held a lazy child that a cast to canonical did not execute, by
way of the FX2 notes in #10633. `develop` keeps one buffer per decimal,
so that reason no longer applies.

## Changes

`CastReduce` now owns the contract and documents both forms of rule. A
rule that builds a new `Validity` keeps the guard and the kernel, and a
rule that pushes the cast into its children needs neither when the nulls
of those children are exactly the null rows of the array. `Dict` is the
documented exception, because its values can hold nulls that no code
references. No new tests are needed: every push-down encoding listed
above already runs `test_cast_conformance` on a partially valid array,
and that suite asserts that the cast to non-nullable fails.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01V1YHcosAAKufkyKPorUC2n)_
@connortsui20
connortsui20 force-pushed the ct/ext-types-research branch from fe6586e to 0c062ea Compare October 26, 2026 15:17
claude and others added 13 commits October 26, 2026 11:17
Research notes on why extension types must be opaque, what is wrong on
develop, a proposed design, urgent changes, and an implementation plan.
The agents directory holds the detailed findings catalog, design notes,
and the throwaway probe that reproduced the findings.

Signed-off-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Aligns the short-term nullability rule across the design documents, fixes
the count of typed wrappers in vortex-spatial, and uses a checked multiply
in the unit cast example.

Signed-off-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Research notes on making Decimal an opaque wrapper over an integer
child, the choice between an integer child and a FixedSizeBinary
child, a wide-integer tier, FixedSizeBinary as its own type, and a
dependency graph for the type work. The agents directory holds the
full report and the call-site and PType inventories.

The extension type notes now link to the Decimal notes, record the
position of the wrapper relative to generic encodings as an open
question, and state which parts of the Decimal work wait for the
extension phases.

Signed-off-by: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Derives are now a property of one extension dtype. A derive applies
only when every operand in a derived position has that dtype.
Operations across dtypes or metadata need explicit compatibility,
coercion or rewrite rules, and the notes do not assume them.

The design notes now define these parts:

- A Rust contract for Scalar equality and order that keeps PartialEq
  and PartialOrd consistent.
- Exact rules for the removal of ext_storage and ext_wrap pairs, for
  checked and unchecked wraps.
- A validation model with validated and unvalidated arrays, and a
  default validate_array that checks each value.
- Function declarations that authorize type-aware behavior, and a
  precise storage-only invariant.
- The position of the wrapper relative to generic encodings, as an
  open decision with a labeled recommendation.

A follow-up probe ran in vortex-btrblocks. It confirms the
DateTimeParts compare bug (EXT-Q3), the Decimal to f32 cast
disagreement, and a new finding about scalar storage checks
(EXT-V7). The notes also correct the U5 and U6 node mapping, a claim
about float equality, and a claim about the decimal wire form.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
U1 now also removes the storage target case in ExtScalar::cast.
Without that change, constants and the row path still cast an
extension value to its storage dtype after the array path stops.

The Decimal rewrite of a cast to an integer keeps the current rule: a
value outside the target range fails before truncation. Only min and
max wrap their result again, because is_sorted and is_constant return
Bool.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The new research/decisions.md records the decisions from the review
on 2026-10-19 and the questions that stay open. The other notes
follow it.

Extension types now implement their behavior as rewrite rules on
arrays. The rules are registered with the type, and there is no
ext_wrap expression. A lift rule pulls generic structural arrays,
such as Constant and Dict, out from under the extension wrapper.
ExtensionArray::try_new checks values, and a new new_unchecked
serves kernels.

Decimal stays a leaf with one buffer and no data child. The
precision fixes the buffer width, so each decimal dtype has one
canonical encoding. A Narrow encoding stores a column at a lower
precision with the same scale. A reader wraps the narrow most
significant part of an old DecimalByteParts array in an integer
Narrow, without a copy.

The notes also list two costs of the one-width rule: an old I128
leaf at precision 39 to 76 needs a widening copy, and an Arrow
import at another width copies its buffer.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Add decisions D21 to D24 and update D8, D9, D11, D16, D17 and D20
after a confirmation review of the decimal and Narrow design.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
#10223, #10332 and #10333 closed on 2026-10-23, and new PRs replace
them. Note that the local rewrite needs a rebase over #10681 and new
branch names.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Split the work into 22 PRs with one brief for each, and record decisions
D25 to D35 from the planning review.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Fix the dependency data, mark the PRs that share files, and explain how
agents reach the briefs and rebase stacks. Record D36 and D37, rename the
canonical width function, and let agents run any command the work needs.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Treat the local rewrite as a reference for tests and edge cases, not as
a base (D38). Build every Narrow aggregate accumulator from the parent
dtype (D36), and explain D36 and D37 in the design notes.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Lists the draft PRs and follow-up issues opened on 2026-10-23, and notes in the CD6 brief that it must update the round-trip check that CD2 adds.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Records why FX2 was dropped (#10798 documents the cast-rule contract), the merged CD2 fixtures, the error types without categories (#10080), and the FoR scheme change (#10757). The tracking issues #10701 and #10733 now hold the state of each PR, so part 4 points to them instead of a status table.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/ext-types-research branch from 0c062ea to f3f42a2 Compare October 26, 2026 15:18
A preflight review checked the NR1 to NR4 briefs against develop. This commit
applies its corrections and the decisions made with it:

- D26: the serialization workaround is crate-private, in
  ArraySerializationTree::try_new. No public trait method is added.
- D36: an aggregate on a Narrow runs in the accumulator of the parent dtype.
  Kernels build the exact parent partial and never cast a partial state. The
  sum kernel serves only Sum with skip_nans.
- NR1 adds a take kernel instead of a TakeReduce rule (D25), names the
  helpers that NR4 reuses, and adds conformance tests.
- NR2 fixes its session, its input recipe and its workloads.
- NR4 declines operands that are neither a Narrow nor a constant.
- The stack lives in vortex2 and uses gh stack v0.0.8.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A registered VTable also gets a reader entry under its wire ID. For
Narrow, which has no wire form yet, that entry can only fail, so it
hides the allow_unknown foreign fallback and lists vortex.narrow in
footers written without editions. NR1 (#10864) therefore leaves Narrow
out of ArraySession, lowers every Narrow before the serializer lookup,
and makes Narrow::serialize panic with a TODO. WR1 adds the
registration together with the working deserializer.

D26, the NR1 brief and the wire brief now say so. The NR1 brief also
gets the final deprecation note of encode_values.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
NR1 (#10864) no longer lowers a Narrow in ArraySerializationTree. In
NR1 nothing outside the tests creates a Narrow, so the branch protected
no real path, and a branch specific to Narrow does not belong in the
generic serializer while nothing needs it. Serializing a Narrow now
fails like it does for any other unregistered in-memory encoding.

D26 is now deferred to CD4, the first PR where a Narrow reaches writers
in ordinary use. CD4 chooses between a lowering specific to Narrow, a
generic lowering for arrays that cannot be written, and no lowering.
The NR1 brief, the CD4 and WR1 briefs, the design and the plan now say
so.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
NR1 (#10864) gave vortex-array/src/arrays/narrow/ the layout of the
other encodings: one file per operation in compute/, and the vtable in
vtable/. The integer Narrow brief now describes that layout, and the
NR3, NR4 and NR5 file tables put their code there. The CD4 and wire
briefs point to the same section.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 mentioned this pull request Nov 2, 2026
9 tasks done
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and
#11016, and the DataFusion literal fix #11067 merged right after U2. So
order constraint 1 holds, and phase 1 starts from develop. The graph
shows these nodes as merged, and parts 2 and 4 name the PR that fixed
each finding.

#10953 now keeps i32 days and widens only the days part to i64 when a
value does not fit. It has no review, and it conflicts with the
DateTimeParts compare kernel that U2 changed. #11056 still waits on its
request for changes. #10661 replaces the closed #10482 for file sort
statistics.

In the type roadmap, only the N1 sentence changes.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 mentioned this pull request Nov 2, 2026
19 of 57 tasks
…11114

Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has
only `None`, `Structural` and `Specific`, and the built-in functions
answer `Specific` for fixed lists of types that match the claims of X16
(new decision X35). The claims, `ExtensionUse::Storage`,
`ResultMapping`, the generic storage rewrite and the storage paths move
to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on
#11115.

The declarations of phase 2 now land before the typing check of phase
3. Before, the check could have rejected timestamp compares until TY1
landed. U9 wraps the results of the storage pushdown with a fixed
mapping, so it no longer waits for `ResultMapping`.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Adds EXT-V10: the `Display` of `Date` panics on valid storage values
that jiff cannot hold. X31 now says so, and the plan gives the finding
a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953
on 2026-11-03, the local review commits, the removal of the
cached-bounds fast path, and the benchmark against `develop`.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
SR merges into SD, because a storage declaration without the exact read
check reads an old form silently wrong. SD now needs SN1, which only
tightens the spatial validators. SN2 takes every Arrow importer of a
known type and builds on DP, which peels dictionary fields before the
importers run. V10 and DS become their own PRs.

Also record that #10953 is pushed at 7943b4e and waits for an
approval, that the 12.9% millisecond slowdown was measured at
d6e73bd, and that #11056 merged.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187)
merged on 2026-11-04. The dependency graph marks the three merged and
moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their
open PRs (#11184, #11185, #11186 and #11188), and it drops #10408,
which closed without a merge. The text of parts 4 and 5 and the Fix
lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state.

Also record that AX, DS and SN2 are implemented on branches without a
PR, and add the approved one-pass design for the masked nulls of SN2.

Part 2 and decision X31 also record the fixes: #10953 made every i64
legal, #11183 made the spatial validators reject nullable nested
levels, #11187 gave the Date display its fallback, and #10408 closed.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
DP (#11185) merged on 2026-11-04. SN2, AX and DS are open drafts as

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
#11223, #11224 and #11225, so the graph and parts 4 and 5 link them.
SN3 drops masked nulls below every nested level after SN2. TY5 declares the
type-specific functions, which the typing check needs. DS2 rejects DataFusion
columns that mix an extension type and plain storage. #11114 now lists all
three.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@vercel

vercel Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
vortex-docs Ready Ready Preview Nov 4, 2026 5:54pm UTC
vortex-explorer Ready Ready Preview Nov 4, 2026 5:54pm UTC

Request Review

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20 connortsui20 changed the title [DO NOT MERGE] Research notes on extension types, Decimal and casts [DO NOT MERGE] Research notes on extension types and casts Nov 4, 2026
The maintainer closed #11188 on 2026-11-04, because naming the value fields
by index is a half-measure and SP4 removes the adapter anyway. Issue #11243
tracks the bug until SP4. X12, part 4, the plan, and the findings now say
so, and part 3 asks whether SP1 needs a positional `Fields` mapping at all.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Record the merged AX and DS, the new RT, SF and AF nodes, and the related PRs that closed or merged. Mark the nodes that wait for a new design of the function declaration, since FD (#11186) closed after review.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Remove ExtensionUse, the result mappings, the generic storage rewrite and
the hardcoded type lists (X36 to X38). Each function checks the claims
that it needs in its own return_dtype and runs on storage itself. Code
that builds no node checks DType::holds directly.

Rewrite the implementation plan with the new PR IDs (CL, TY, ST), add
findings EXT-A4 (bloom filter) and EXT-A5 (NaN guard), and add the agent
plan in 06-agent-plan.md.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Record the answers of 2026-11-17: Codex-managed worktrees, draft PRs and issue plan updates after review, work on develop beside the crate moves, and scoped checks at the tip.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
ExtensionArray::new_unchecked already exists with a contract that covers only the storage dtype. A debug value check in it would reject constructions that develop accepts, so VC1 owns the contract and the check.
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The notes still described a `&'static [Claim]` list with `Claim::implies`, which the PR replaced with the opaque `Claims` value.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/docs A docs change do not merge Pull requests that are not intended to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants