Skip to content

test: replace arrow_cast with :: casts where a SQL type is equivalent - #76

Open
andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:test-arrow-cast-to-double-colon
Open

andygrove wants to merge 4 commits into
apache:mainfrom
andygrove:test-arrow-cast-to-double-colon

Conversation

@andygrove

@andygrove andygrove commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

N/A. Follow-up to #54 (the PySpark runner skips any query that calls arrow_cast, because Spark has no such function).

Rationale for this change

Many .slt queries use arrow_cast only to pick a type that has a plain SQL name. Writing those as :: casts makes the tests easier to read, and Spark can run them, so the PySpark runner checks these queries instead of skipping them.

What changes are included in this PR?

arrow_cast(x, 'T') becomes x::SQLTYPE in 58 files (482 lines) when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name:

Arrow type SQL type
Int8 / Int16 / Int32 / Int64 TINYINT / SMALLINT / INT / BIGINT
Float32 / Float64 FLOAT / DOUBLE
Boolean, Binary, Date32 BOOLEAN, BINARY, DATE
Utf8View STRING
Decimal128(p, s), p ≤ 38 DECIMAL(p, s)
List(Int32) ARRAY<INT>

:: binds tighter than unary minus, so negative literals and expressions are wrapped in parentheses, for example (-128)::TINYINT. Without them, -128::TINYINT casts 128 first and overflows.

Spark cannot cast an integer to DATE, so an arrow_cast(n, 'Date32') of an integer day count becomes date_from_unix_date(n) rather than n::DATE. A test that reads a DATE result back as a day count uses unix_date(...), because Spark's CAST of a DATE to INT returns NULL.

arrow_cast is kept where no SQL type gives the same Arrow type:

  • Utf8, because STRING and VARCHAR map to Utf8View by default
  • unsigned integers, Float16, Large*, *View, Date64, Time32, Decimal32/Decimal64/Decimal256, FixedSize*, Dictionary, Struct and Map
  • timestamps with a precision or a time zone, and Time64
  • Utf8View casts of a column that is already Utf8View (in bround.slt and positive.slt), where column1::STRING gets the same name as plain column1 and DataFusion rejects the duplicate projection

Are these changes tested?

Only test files changed. cargo test -p datafusion-spark-sqllogictest passes, apart from math/expm1.slt, which also fails on main on macOS and is fixed by #359. The PySpark runner now checks the converted queries on Spark 4.2, and none of them adds a new difference from Spark.

Are there any user-facing changes?

No.

Rewrite arrow_cast(x, 'T') as x::SQLTYPE in the sqllogictest files when
DataFusion maps the SQL type to exactly the same Arrow type and Spark
accepts the type name: Int8/16/32/64, Float32/64, Boolean, Binary,
Date32, Utf8View (STRING), Decimal128(p, s) with p <= 38 and
List(Int32) (ARRAY<INT>). Negative literals and expressions are
parenthesized because :: binds tighter than unary minus.

arrow_cast is kept for Utf8 (STRING maps to Utf8View), unsigned
integers, timestamps with a precision or time zone, and Arrow types
without a SQL name. Spark has no arrow_cast, so the PySpark runner can
now run these queries instead of skipping them.
…h date_from_unix_date

Spark cannot cast an integer to DATE, so integer day counts now go through
date_from_unix_date, and DATE results go back to day counts with unix_date.
@andygrove
andygrove force-pushed the test-arrow-cast-to-double-colon branch from f0c496c to cc2249f Compare November 18, 2026 22:11

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove, thanks for the follow-up. The cast conversions and DATE input changes look appropriate.

Requesting changes because the new unix_date wrappers break both next_day assertions in DataFusion evaluate mode. Restoring the outer Arrow casts preserves the assertions and remains compatible with Spark.

# the epoch day directly (these dates are past the printable range).
query I
SELECT arrow_cast(next_day(arrow_cast(95026236, 'Date32'), 'Mon'::string), 'Int32');
SELECT unix_date(next_day(date_from_unix_date(95026236), 'Mon'::string));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both unix_date wrappers fail in evaluate mode because physical execution is unsupported. Please restore arrow_cast(next_day(...), 'Int32') on lines 95 and 100. The PySpark runner translates these casts to unix_date.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks. Restored the outer arrow_cast(..., 'Int32') on both queries in c367a7c and kept date_from_unix_date inside. Both pass in evaluate mode now, and the PySpark runner still passes them on Spark 4.2.0 through its unix_date translation.

## NULL with integer format using arrow_cast
query T
SELECT format_string('Integer: %d', arrow_cast(NULL, 'Int32'));
SELECT format_string('Integer: %d', NULL::INT);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove using arrow_cast from the 26 headings above converted queries, retaining it for Arrow-only casts. Also update the notes in math/floor.slt at lines 163, 237, and 244 to explain the unary-minus precedence.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in c367a7c. I removed using arrow_cast from the 26 headings whose queries now use ::. The one left is above the commented-out Float16 query, which still uses arrow_cast. The three floor.slt notes now explain that :: binds tighter than unary minus, so -128::TINYINT would be -(128::TINYINT) and overflow. That's why the boundary literals are in parentheses.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small correction to my last comment. Besides the commented-out Float16 query, the 10 Timestamp(ns) queries at lines 995-1043 also keep "using arrow_cast" in their headings, since they still call arrow_cast (there is no SQL type for a nanosecond timestamp).

…notes

unix_date has no physical implementation, so wrapping next_day in it fails
in evaluate mode. Restore the outer arrow_cast to Int32, which the PySpark
runner already translates to unix_date.

Drop 'using arrow_cast' from format_string headings whose queries now use
:: casts, and explain in floor.slt why the negative boundary literals are
parenthesized.
@andygrove

Copy link
Copy Markdown
Member Author

@kosiew thanks for the review. c367a7c restores the next_day casts so evaluate mode passes. It also fixes the stale headings and notes. Could you take another look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants