Repository navigation
Conversation
Rewrite arrow_cast(x, 'T') as x::SQLTYPE in the sqllogictest files when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name: Int8/16/32/64, Float32/64, Boolean, Binary, Date32, Utf8View (STRING), Decimal128(p, s) with p <= 38 and List(Int32) (ARRAY<INT>). Negative literals and expressions are parenthesized because :: binds tighter than unary minus. arrow_cast is kept for Utf8 (STRING maps to Utf8View), unsigned integers, timestamps with a precision or time zone, and Arrow types without a SQL name. Spark has no arrow_cast, so the PySpark runner can now run these queries instead of skipping them.
…h date_from_unix_date Spark cannot cast an integer to DATE, so integer day counts now go through date_from_unix_date, and DATE results go back to day counts with unix_date.
f0c496c to
cc2249f
Compare
kosiew
left a comment
There was a problem hiding this comment.
@andygrove, thanks for the follow-up. The cast conversions and DATE input changes look appropriate.
Requesting changes because the new unix_date wrappers break both next_day assertions in DataFusion evaluate mode. Restoring the outer Arrow casts preserves the assertions and remains compatible with Spark.
| # the epoch day directly (these dates are past the printable range). | ||
| query I | ||
| SELECT arrow_cast(next_day(arrow_cast(95026236, 'Date32'), 'Mon'::string), 'Int32'); | ||
| SELECT unix_date(next_day(date_from_unix_date(95026236), 'Mon'::string)); |
There was a problem hiding this comment.
Both unix_date wrappers fail in evaluate mode because physical execution is unsupported. Please restore arrow_cast(next_day(...), 'Int32') on lines 95 and 100. The PySpark runner translates these casts to unix_date.
There was a problem hiding this comment.
Good catch, thanks. Restored the outer arrow_cast(..., 'Int32') on both queries in c367a7c and kept date_from_unix_date inside. Both pass in evaluate mode now, and the PySpark runner still passes them on Spark 4.2.0 through its unix_date translation.
| ## NULL with integer format using arrow_cast | ||
| query T | ||
| SELECT format_string('Integer: %d', arrow_cast(NULL, 'Int32')); | ||
| SELECT format_string('Integer: %d', NULL::INT); |
There was a problem hiding this comment.
Please remove using arrow_cast from the 26 headings above converted queries, retaining it for Arrow-only casts. Also update the notes in math/floor.slt at lines 163, 237, and 244 to explain the unary-minus precedence.
There was a problem hiding this comment.
Done in c367a7c. I removed using arrow_cast from the 26 headings whose queries now use ::. The one left is above the commented-out Float16 query, which still uses arrow_cast. The three floor.slt notes now explain that :: binds tighter than unary minus, so -128::TINYINT would be -(128::TINYINT) and overflow. That's why the boundary literals are in parentheses.
There was a problem hiding this comment.
Small correction to my last comment. Besides the commented-out Float16 query, the 10 Timestamp(ns) queries at lines 995-1043 also keep "using arrow_cast" in their headings, since they still call arrow_cast (there is no SQL type for a nanosecond timestamp).
…notes unix_date has no physical implementation, so wrapping next_day in it fails in evaluate mode. Restore the outer arrow_cast to Int32, which the PySpark runner already translates to unix_date. Drop 'using arrow_cast' from format_string headings whose queries now use :: casts, and explain in floor.slt why the negative boundary literals are parenthesized.
Which issue does this PR close?
N/A. Follow-up to #54 (the PySpark runner skips any query that calls
arrow_cast, because Spark has no such function).Rationale for this change
Many
.sltqueries usearrow_castonly to pick a type that has a plain SQL name. Writing those as::casts makes the tests easier to read, and Spark can run them, so the PySpark runner checks these queries instead of skipping them.What changes are included in this PR?
arrow_cast(x, 'T')becomesx::SQLTYPEin 58 files (482 lines) when DataFusion maps the SQL type to exactly the same Arrow type and Spark accepts the type name:Int8/Int16/Int32/Int64TINYINT/SMALLINT/INT/BIGINTFloat32/Float64FLOAT/DOUBLEBoolean,Binary,Date32BOOLEAN,BINARY,DATEUtf8ViewSTRINGDecimal128(p, s), p ≤ 38DECIMAL(p, s)List(Int32)ARRAY<INT>::binds tighter than unary minus, so negative literals and expressions are wrapped in parentheses, for example(-128)::TINYINT. Without them,-128::TINYINTcasts128first and overflows.Spark cannot cast an integer to
DATE, so anarrow_cast(n, 'Date32')of an integer day count becomesdate_from_unix_date(n)rather thann::DATE. A test that reads aDATEresult back as a day count usesunix_date(...), because Spark'sCASTof aDATEtoINTreturns NULL.arrow_castis kept where no SQL type gives the same Arrow type:Utf8, becauseSTRINGandVARCHARmap toUtf8Viewby defaultFloat16,Large*,*View,Date64,Time32,Decimal32/Decimal64/Decimal256,FixedSize*,Dictionary,StructandMapTime64Utf8Viewcasts of a column that is alreadyUtf8View(inbround.sltandpositive.slt), wherecolumn1::STRINGgets the same name as plaincolumn1and DataFusion rejects the duplicate projectionAre these changes tested?
Only test files changed.
cargo test -p datafusion-spark-sqllogictestpasses, apart frommath/expm1.slt, which also fails on main on macOS and is fixed by #359. The PySpark runner now checks the converted queries on Spark 4.2, and none of them adds a new difference from Spark.Are there any user-facing changes?
No.