Repository navigation
Stop casts from reinterpreting extension storage - #11599
Conversation
5ba4aa0 to
c2157ae
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
60e04d4 to
9d44696
Compare
Merging this PR will degrade performance by 15.45%2 warnings
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
robert3005
left a comment
There was a problem hiding this comment.
I remember we added this in #7469, I think this is right though
|
I think we have to figure out what we allow as a cast. It's no longer clear to me that the behavior of #7469 is even correct in the first place... We should look at what the SQL semantics should be and manually add the cast kernel. |
|
I think we want explicit functions instead of implicit cast behaviour. So yes I think this is correct change |
Polars 1.x serializes a time zone aware datetime literal as a cast of its wall-clock time to a naive datetime, followed by `replace_time_zone`. Vortex casts no longer change a temporal dtype, so fold the cast into the literal instead of leaving the whole predicate to Polars. Claude-Session: https://claude.ai/code/session_01J3PQtBmPgQ84dGUc5TdHXw Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
The pyarrow stubs constrain the time zone of `pa.timestamp` to exactly `str` or `None`, so ty rejects an argument typed `str | None`. Branch on `None` and return `pa.DataType`, which is all the callers need. Also drop the `UTC` import that the test no longer uses. Claude-Session: https://claude.ai/code/session_01J3PQtBmPgQ84dGUc5TdHXw Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
A cast between an extension dtype and any other dtype reinterprets the storage instead of failing, so
Timestamp(ms)casts tou8, and a constantTimestamp(ms)of 1000 casts toTimestamp(ns)of 1000, which turns one second into one microsecond. A conversion that changes what a value means belongs to the type, so the generic cast now only changes the top-level nullability or builds typed nulls from an untypedNull, and code that wants the raw storage reads it withext_storage.Cast::return_dtypechecks the rule, so a bad cast fails when the expression is built instead of failing in one kernel but not another.Breaking: a cast from or to an extension dtype, including one nested in a struct, list, or map, fails unless it only changes nullability, so serialized expressions that hold such a cast no longer load.
Breaking:
Cast::newis nowCast::try_new.Breaking: an Arrow export of an extension array fails when the target neither names the type nor has an Arrow type that imports as it, such as
Int64for a timestamp, a nested field without metadata, or the target ofDatum::try_new_with_target_datatype. Applyext_storageto export the storage.Breaking: Python no longer builds extension scalars or literals from raw integers.
Prepared with AI assistance.