Skip to content

Stop casts from reinterpreting extension storage - #11599

Merged
connortsui20 merged 3 commits into
developfrom
ct/ext-no-storage-casts
Nov 19, 2026
Merged

connortsui20 merged 3 commits into
developfrom
ct/ext-no-storage-casts

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

A cast between an extension dtype and any other dtype reinterprets the storage instead of failing, so Timestamp(ms) casts to u8, and a constant Timestamp(ms) of 1000 casts to Timestamp(ns) of 1000, which turns one second into one microsecond. A conversion that changes what a value means belongs to the type, so the generic cast now only changes the top-level nullability or builds typed nulls from an untyped Null, and code that wants the raw storage reads it with ext_storage. Cast::return_dtype checks the rule, so a bad cast fails when the expression is built instead of failing in one kernel but not another.

Breaking: a cast from or to an extension dtype, including one nested in a struct, list, or map, fails unless it only changes nullability, so serialized expressions that hold such a cast no longer load.

Breaking: Cast::new is now Cast::try_new.

Breaking: an Arrow export of an extension array fails when the target neither names the type nor has an Arrow type that imports as it, such as Int64 for a timestamp, a nested field without metadata, or the target of Datum::try_new_with_target_datatype. Apply ext_storage to export the storage.

Breaking: Python no longer builds extension scalars or literals from raw integers.

Prepared with AI assistance.

@connortsui20 connortsui20 mentioned this pull request Nov 18, 2026
19 of 57 tasks
@connortsui20
connortsui20 force-pushed the ct/ext-no-storage-casts branch from 5ba4aa0 to c2157ae Compare November 18, 2026 20:59
@connortsui20
connortsui20 marked this pull request as ready for review November 18, 2026 21:08
@connortsui20 connortsui20 added changelog/fix A bug fix changelog/break A breaking API change and removed changelog/fix A bug fix labels Nov 18, 2026
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/ext-no-storage-casts branch from 60e04d4 to 9d44696 Compare November 18, 2026 22:04
@codspeed

codspeed Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 15.45%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 3 improved benchmarks
❌ 9 regressed benchmarks
✅ 1963 untouched benchmarks
⏩ 53 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Walltime compare_u8_neon 1.6 µs 3.1 µs -47.91%
❌ Walltime compare_int_constant_neon 3.3 µs 4.4 µs -26.51%
❌ Walltime compare_int_eq_neon 3.9 µs 5.1 µs -23.94%
❌ Walltime compare_int_neon 3.9 µs 4.9 µs -19.83%
❌ Walltime compare_u64_neon 3.9 µs 4.9 µs -19.79%
❌ Walltime compare_int_nullable_neon 4.3 µs 5.3 µs -19.72%
❌ Walltime compare_f32_neon 4.2 µs 5.1 µs -19.05%
❌ Walltime compare_float_neon 6 µs 7 µs -13.74%
❌ Walltime deferred_bool_neon[i32, PerRowPerRow] 4.1 µs 4.6 µs -10.88%
⚡ CPU Simulation density_sweep_random[0.02] 56.4 µs 49.4 µs +14.18%
⚡ CPU Simulation density_sweep_random[0.05] 122.5 µs 108.4 µs +13.05%
⚡ CPU Simulation filter_random_by_random[10000] 8 µs 7.2 µs +10.48%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ct/ext-no-storage-casts (8d13b1d) with develop (dc2ac56)

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005 robert3005 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember we added this in #7469, I think this is right though

@connortsui20

Copy link
Copy Markdown
Member Author

I think we have to figure out what we allow as a cast. It's no longer clear to me that the behavior of #7469 is even correct in the first place... We should look at what the SQL semantics should be and manually add the cast kernel.

@robert3005

Copy link
Copy Markdown
Contributor

I think we want explicit functions instead of implicit cast behaviour. So yes I think this is correct change

Polars 1.x serializes a time zone aware datetime literal as a cast of its
wall-clock time to a naive datetime, followed by `replace_time_zone`. Vortex
casts no longer change a temporal dtype, so fold the cast into the literal
instead of leaving the whole predicate to Polars.

Claude-Session: https://claude.ai/code/session_01J3PQtBmPgQ84dGUc5TdHXw
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 enabled auto-merge (squash) November 19, 2026 02:09
The pyarrow stubs constrain the time zone of `pa.timestamp` to exactly `str`
or `None`, so ty rejects an argument typed `str | None`. Branch on `None` and
return `pa.DataType`, which is all the callers need. Also drop the `UTC`
import that the test no longer uses.

Claude-Session: https://claude.ai/code/session_01J3PQtBmPgQ84dGUc5TdHXw
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 merged commit f5b7712 into develop Nov 19, 2026
73 of 74 checks passed
@connortsui20
connortsui20 deleted the ct/ext-no-storage-casts branch November 19, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants