Repository navigation
[DO NOT MERGE] Research notes on extension types and casts - #10633
Draft
connortsui20 wants to merge 61 commits into
Draft
connortsui20 wants to merge 61 commits into
connortsui20 wants to merge 61 commits into
Conversation
This was referenced Oct 23, 2026
Merging this PR will not alter performance1 warning
Comparing |
This was referenced Oct 23, 2026
connortsui20
added a commit
that referenced
this pull request
Oct 26, 2026
## Summary - Closes #10750 - Replaces #10746 The docs on `Validity::cast_nullability` read as if every `CastReduce` rule that removes nullability must call `Validity::trivially_cast_nullability` and pair with a `CastKernel`. Most push-down rules (FoR, ALP, ALP-RD, ZigZag, RLE, RunEnd, DateTimeParts, DecimalByteParts, Chunked, Sparse, Extension) instead cast the child that holds their validity, and the cast of that child already fails on nulls, so no result is wrong. Guarding each rule, as #10746 does for DecimalByteParts, adds a kernel per encoding and delays the rewrite until execution without changing any result. The guard in #10746 comes from the closed #10230, where a canonical decimal held a lazy child that a cast to canonical did not execute, by way of the FX2 notes in #10633. `develop` keeps one buffer per decimal, so that reason no longer applies. ## Changes `CastReduce` now owns the contract and documents both forms of rule. A rule that builds a new `Validity` keeps the guard and the kernel, and a rule that pushes the cast into its children needs neither when the nulls of those children are exactly the null rows of the array. `Dict` is the documented exception, because its values can hold nulls that no code references. No new tests are needed: every push-down encoding listed above already runs `test_cast_conformance` on a partially valid array, and that suite asserts that the cast to non-nullable fails. --- _Generated by [Claude Code](https://claude.ai/code/session_01V1YHcosAAKufkyKPorUC2n)_
connortsui20
force-pushed
the
ct/ext-types-research
branch
from
October 26, 2026 15:17
fe6586e to
0c062ea
Compare
Research notes on why extension types must be opaque, what is wrong on develop, a proposed design, urgent changes, and an implementation plan. The agents directory holds the detailed findings catalog, design notes, and the throwaway probe that reproduced the findings. Signed-off-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Aligns the short-term nullability rule across the design documents, fixes the count of typed wrappers in vortex-spatial, and uses a checked multiply in the unit cast example. Signed-off-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Research notes on making Decimal an opaque wrapper over an integer child, the choice between an integer child and a FixedSizeBinary child, a wide-integer tier, FixedSizeBinary as its own type, and a dependency graph for the type work. The agents directory holds the full report and the call-site and PType inventories. The extension type notes now link to the Decimal notes, record the position of the wrapper relative to generic encodings as an open question, and state which parts of the Decimal work wait for the extension phases. Signed-off-by: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BKSPCwTaE2gzKwpxokMGXb
Derives are now a property of one extension dtype. A derive applies only when every operand in a derived position has that dtype. Operations across dtypes or metadata need explicit compatibility, coercion or rewrite rules, and the notes do not assume them. The design notes now define these parts: - A Rust contract for Scalar equality and order that keeps PartialEq and PartialOrd consistent. - Exact rules for the removal of ext_storage and ext_wrap pairs, for checked and unchecked wraps. - A validation model with validated and unvalidated arrays, and a default validate_array that checks each value. - Function declarations that authorize type-aware behavior, and a precise storage-only invariant. - The position of the wrapper relative to generic encodings, as an open decision with a labeled recommendation. A follow-up probe ran in vortex-btrblocks. It confirms the DateTimeParts compare bug (EXT-Q3), the Decimal to f32 cast disagreement, and a new finding about scalar storage checks (EXT-V7). The notes also correct the U5 and U6 node mapping, a claim about float equality, and a claim about the decimal wire form. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
U1 now also removes the storage target case in ExtScalar::cast. Without that change, constants and the row path still cast an extension value to its storage dtype after the array path stops. The Decimal rewrite of a cast to an integer keeps the current rule: a value outside the target range fails before truncation. Only min and max wrap their result again, because is_sorted and is_constant return Bool. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The new research/decisions.md records the decisions from the review on 2026-10-19 and the questions that stay open. The other notes follow it. Extension types now implement their behavior as rewrite rules on arrays. The rules are registered with the type, and there is no ext_wrap expression. A lift rule pulls generic structural arrays, such as Constant and Dict, out from under the extension wrapper. ExtensionArray::try_new checks values, and a new new_unchecked serves kernels. Decimal stays a leaf with one buffer and no data child. The precision fixes the buffer width, so each decimal dtype has one canonical encoding. A Narrow encoding stores a column at a lower precision with the same scale. A reader wraps the narrow most significant part of an old DecimalByteParts array in an integer Narrow, without a copy. The notes also list two costs of the one-width rule: an old I128 leaf at precision 39 to 76 needs a widening copy, and an Arrow import at another width copies its buffer. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Add decisions D21 to D24 and update D8, D9, D11, D16, D17 and D20 after a confirmation review of the decimal and Narrow design. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Split the work into 22 PRs with one brief for each, and record decisions D25 to D35 from the planning review. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Fix the dependency data, mark the PRs that share files, and explain how agents reach the briefs and rebase stacks. Record D36 and D37, rename the canonical width function, and let agents run any command the work needs. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Treat the local rewrite as a reference for tests and edge cases, not as a base (D38). Build every Narrow aggregate accumulator from the parent dtype (D36), and explain D36 and D37 in the design notes. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Lists the draft PRs and follow-up issues opened on 2026-10-23, and notes in the CD6 brief that it must update the round-trip check that CD2 adds. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Records why FX2 was dropped (#10798 documents the cast-rule contract), the merged CD2 fixtures, the error types without categories (#10080), and the FoR scheme change (#10757). The tracking issues #10701 and #10733 now hold the state of each PR, so part 4 points to them instead of a status table. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/ext-types-research
branch
from
October 26, 2026 15:18
0c062ea to
f3f42a2
Compare
A preflight review checked the NR1 to NR4 briefs against develop. This commit applies its corrections and the decisions made with it: - D26: the serialization workaround is crate-private, in ArraySerializationTree::try_new. No public trait method is added. - D36: an aggregate on a Narrow runs in the accumulator of the parent dtype. Kernels build the exact parent partial and never cast a partial state. The sum kernel serves only Sum with skip_nans. - NR1 adds a take kernel instead of a TakeReduce rule (D25), names the helpers that NR4 reuses, and adds conformance tests. - NR2 fixes its session, its input recipe and its workloads. - NR4 declines operands that are neither a Narrow nor a constant. - The stack lives in vortex2 and uses gh stack v0.0.8. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A registered VTable also gets a reader entry under its wire ID. For Narrow, which has no wire form yet, that entry can only fail, so it hides the allow_unknown foreign fallback and lists vortex.narrow in footers written without editions. NR1 (#10864) therefore leaves Narrow out of ArraySession, lowers every Narrow before the serializer lookup, and makes Narrow::serialize panic with a TODO. WR1 adds the registration together with the working deserializer. D26, the NR1 brief and the wire brief now say so. The NR1 brief also gets the final deprecation note of encode_values. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
NR1 (#10864) no longer lowers a Narrow in ArraySerializationTree. In NR1 nothing outside the tests creates a Narrow, so the branch protected no real path, and a branch specific to Narrow does not belong in the generic serializer while nothing needs it. Serializing a Narrow now fails like it does for any other unregistered in-memory encoding. D26 is now deferred to CD4, the first PR where a Narrow reaches writers in ordinary use. CD4 chooses between a lowering specific to Narrow, a generic lowering for arrays that cannot be written, and no lowering. The NR1 brief, the CD4 and WR1 briefs, the design and the plan now say so. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
NR1 (#10864) gave vortex-array/src/arrays/narrow/ the layout of the other encodings: one file per operation in compute/, and the vtable in vtable/. The integer Narrow brief now describes that layout, and the NR3, NR4 and NR5 file tables put their code there. The CD4 and wire briefs point to the same section. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
9 tasks done
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and #11016, and the DataFusion literal fix #11067 merged right after U2. So order constraint 1 holds, and phase 1 starts from develop. The graph shows these nodes as merged, and parts 2 and 4 name the PR that fixed each finding. #10953 now keeps i32 days and widens only the days part to i64 when a value does not fit. It has no review, and it conflicts with the DateTimeParts compare kernel that U2 changed. #11056 still waits on its request for changes. #10661 replaces the closed #10482 for file sort statistics. In the type roadmap, only the N1 sentence changes. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
19 of 57 tasks
…11114 Each node of the graph is now one PR, with the ID that the PR plan in #11114 uses. The claims node points to #11115, the old phase 1 node splits into NU and CL, and the shared pushdown and the lift rule become separate low-priority nodes. The graph also links #10661, #10686 and #10793 as open PRs to coordinate with. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Opacity in #11114 now lands with hardcoded opt-ins. `ExtensionUse` has only `None`, `Structural` and `Specific`, and the built-in functions answer `Specific` for fixed lists of types that match the claims of X16 (new decision X35). The claims, `ExtensionUse::Storage`, `ResultMapping`, the generic storage rewrite and the storage paths move to #11115 as CL1 to CL3 and SP1 to SP7. So #11114 no longer depends on #11115. The declarations of phase 2 now land before the typing check of phase 3. Before, the check could have rejected timestamp compares until TY1 landed. U9 wraps the results of the storage pushdown with a fixed mapping, so it no longer waits for `ResultMapping`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Adds EXT-V10: the `Display` of `Date` panics on valid storage values that jiff cannot hold. X31 now says so, and the plan gives the finding a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953 on 2026-11-03, the local review commits, the removal of the cached-bounds fast path, and the benchmark against `develop`. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
SR merges into SD, because a storage declaration without the exact read check reads an old form silently wrong. SD now needs SN1, which only tightens the spatial validators. SN2 takes every Arrow importer of a known type and builds on DP, which peels dictionary fields before the importers run. V10 and DS become their own PRs. Also record that #10953 is pushed at 7943b4e and waits for an approval, that the 12.9% millisecond slowdown was measured at d6e73bd, and that #11056 merged. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187) merged on 2026-11-04. The dependency graph marks the three merged and moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their open PRs (#11184, #11185, #11186 and #11188), and it drops #10408, which closed without a merge. The text of parts 4 and 5 and the Fix lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state. Also record that AX, DS and SN2 are implemented on branches without a PR, and add the approved one-pass design for the masked nulls of SN2. Part 2 and decision X31 also record the fixes: #10953 made every i64 legal, #11183 made the spatial validators reject nullable nested levels, #11187 gave the Date display its fallback, and #10408 closed. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
SN3 drops masked nulls below every nested level after SN2. TY5 declares the type-specific functions, which the typing check needs. DS2 rejects DataFusion columns that mix an extension type and plain storage. #11114 now lists all three. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The maintainer closed #11188 on 2026-11-04, because naming the value fields by index is a half-measure and SP4 removes the adapter anyway. Issue #11243 tracks the bug until SP4. X12, part 4, the plan, and the findings now say so, and part 3 asks whether SP1 needs a positional `Fields` mapping at all. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Record the merged AX and DS, the new RT, SF and AF nodes, and the related PRs that closed or merged. Mark the nodes that wait for a new design of the function declaration, since FD (#11186) closed after review. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/ext-types-research
branch
from
November 16, 2026 16:03
d570d3a to
fb36802
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Remove ExtensionUse, the result mappings, the generic storage rewrite and the hardcoded type lists (X36 to X38). Each function checks the claims that it needs in its own return_dtype and runs on storage itself. Code that builds no node checks DType::holds directly. Rewrite the implementation plan with the new PR IDs (CL, TY, ST), add findings EXT-A4 (bloom filter) and EXT-A5 (NaN guard), and add the agent plan in 06-agent-plan.md. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Record the answers of 2026-11-17: Codex-managed worktrees, draft PRs and issue plan updates after review, work on develop beside the crate moves, and scoped checks at the tip. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
ExtensionArray::new_unchecked already exists with a contract that covers only the storage dtype. A debug value check in it would reject constructions that develop accepts, so VC1 owns the contract and the check. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The notes still described a `&'static [Claim]` list with `Claim::implies`, which the PR replaced with the opaque `Claims` value. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Research notes on the Vortex type system: extension types and casts. This PR exists so the notes can be read and discussed on GitHub. It is not meant to merge.
For review, start with
research/decisions.md. It records the settled decisions and the open questions, each with a recommendation, and the other notes follow it. The topic has a short summary (research/extension-types/) and anagents/directory with the detailed report, file and line references againstdevelopat5fc5cb8, and reproductions.The main decisions so far:
ConstantandDictout from under the extension wrapper.ExtensionArray::try_newchecks values, and a newnew_uncheckedserves kernels.The Decimal notes,
Widenplan, and type roadmap moved to #11237.Rendered