Repository navigation
fix(datetime): avoid premature overflow in make_dt_interval (#109) - #119
Conversation
Signed-off-by: Subodh Bansode <subodh171011@gmail.com>
andygrove
left a comment
There was a problem hiding this comment.
Note: This is a first-pass automated review generated by Claude Code, posted ahead of human review. A maintainer will follow up with a proper review.
Thanks for the fix. Widening to i64 does fix #109. Every value in the new .slt cases matches PySpark 4.2.0, with ANSI on and off, UTC session: 1,491,309 / ±1,500,000 / ±106,751,991 days return intervals, ±106,751,992 days overflow, and i64::MAX / i64::MIN are exact at the boundary. The 23 unit tests and datetime/make_dt_interval.slt pass locally. I found one correctness problem in how the overflow is detected.
-
crates/spark/src/function/datetime/make_dt_interval.rs:239-262: mixed-sign inputs that overflow in Spark now return a non-NULL valueSpark's
IntervalUtils.makeDayTimeIntervalworks in microseconds. It starts from the seconds and adds each part with an overflow check, in this order:var micros = secs.toUnscaledLong micros = Math.addExact(micros, Math.multiplyExact(days, MICROS_PER_DAY)) micros = Math.addExact(micros, Math.multiplyExact(hours, MICROS_PER_HOUR)) micros = Math.addExact(micros, Math.multiplyExact(mins, MICROS_PER_MINUTE))
Spark raises an error whenever one of those steps overflows, even if a later part with the opposite sign would bring the total back into range. This PR adds days, hours and minutes into a single minute count in
i64before converting to microseconds. An opposite-sign part can therefore cancel the overflow before it is ever checked:Query Spark 4.2.0 (ANSI on and off) This PR make_dt_interval(106751992, -24, 0, 0)INTERVAL_ARITHMETIC_OVERFLOWerror106751991 days 0 hours 0 mins 0.000000 secsmake_dt_interval(106751991, 5, -60, 0)INTERVAL_ARITHMETIC_OVERFLOWerror106751991 days 4 hours 0 mins 0.000000 secsBefore this PR these inputs returned NULL, because
day * 24 * 60overflowedi32. That matched the "overflow returns NULL" behavior tracked in #20. Now they return a value where Spark has none.Suggested fix: follow Spark's order and add up in microseconds, roughly:
let mut micros = secs_to_micros(sec)?; // checked, as today micros = micros.checked_add((day as i64).checked_mul(MICROS_PER_DAY)?)?; micros = micros.checked_add((hour as i64).checked_mul(MICROS_PER_HOUR)?)?; micros = micros.checked_add((min as i64).checked_mul(MICROS_PER_MINUTE)?)?;
This is also simpler than the carry logic between the whole and fractional seconds. Please add the two queries above to
make_dt_interval.slt, expectingNULL, with a comment pointing to #20, next to the existing±106751992cases. It would also help to have the exacti64::MAX/i64::MINboundary in the.slt(make_dt_interval(106751991, 4, 0, 54.775807)andmake_dt_interval(-106751991, -4, 0, -54.775808)). Both are confirmed in PySpark 4.2.0. Today the boundary is only covered by the unit tests.How verified: I ran the Spark results in PySpark 4.2.0 with
spark.sql.ansi.enabledboth true and false. I got the PR results by running the PR branch through a temporary.sltfile.
🤖 Generated with Claude Code
Accumulate components sequentially in microseconds following Spark's exact IntervalUtils.makeDayTimeInterval evaluation order (seconds -> days -> hours -> minutes). Check each multiplication and addition independently using checked operations. Return NULL on overflow consistent with DataFusion Spark's documented contract (apache#20). Add full regression and boundary test coverage including maintainer-requested mixed-sign cases. Signed-off-by: Subodh Bansode <subodh171011@gmail.com>
andygrove
left a comment
There was a problem hiding this comment.
Thanks for the update. My earlier concern is fixed. The result is now built in microseconds in Spark's order (secs, then days, hours, mins), and each multiply and add is checked, so an opposite-sign part can no longer cancel an overflow. The two mixed-sign queries now return NULL with a pointer to #20, and the exact i64::MAX / i64::MIN boundaries are in the .slt.
I checked the new cases against PySpark 4.2.0 with ANSI on and off. The boundary values match. Both one-microsecond-past-boundary queries raise INTERVAL_ARITHMETIC_OVERFLOW in Spark, so NULL is right under #20. On my machine the unit tests, make_dt_interval.slt, clippy and fmt all pass.
Nit, optional: the new unit tests repeat the same array setup and downcast in every test. A helper like fn eval(day, hour, min, sec) -> Option<i64> would make them much shorter, and several of them repeat what the .slt already covers.
LGTM. I'll approve the CI run.
Summary
Fixes premature integer overflow in Spark's
make_dt_intervalexpression (#109).Context & Defect
make_dt_intervalreturns an ArrowDuration(Microsecond)backed by a 64-bit signed integer (i64), supporting durations up toi32). Consequently, any interval withdays >= 1,491,309(ordays <= -1,491,309) overflowedi32during minute calculation (NULLfor valid day-time intervals that Apache Spark evaluates successfully.Key Changes
i64: Promoted intermediate conversion constantsHOURS_PER_DAYandMINS_PER_HOURtoi64and widenedday,hour, andmintoi64before multiplication and addition.checked_mul,checked_add,checked_sub), safely returningNone(NULL) on genuine overflow.const MAX_SECS_ABS: f64 = (1i64 << 53) as f64;rejecting non-finite floats andsec.trunc() as i64without rejecting any representable interval (guard margin is.abs(): Replacedfrac_us.abs()with safe signed comparisons (frac_us >= MICROS_PER_SECandfrac_us <= -MICROS_PER_SEC) and checked carry operations (checked_add(1)?/checked_sub(1)?), eliminating debug panic risks oni64::MIN.NULL, preserving the distinct architectural boundary tracked under Spark make_dt_interval/make_interval return NULL on overflow instead of erroring #20 (error-raising on true overflow).Spark Parity & Compatibility
Verification & Tests
crates/spark/src/function/datetime/make_dt_interval.rs(testing positive/negative boundaries, exacti64::MAX/i64::MIN,crates/sqllogictest/test_files/datetime/make_dt_interval.slt(updating 1,500,000 days from NULL to valid interval and adding positive/negative boundary assertions).cargo test -p datafusion-spark).cargo test -p datafusion-spark-sqllogictest).cargo fmt --all -- --checkclean;cargo clippy -p datafusion-spark --all-targets -- -D warningsclean (0 warnings);git diff --checkclean.Fixes #109.