Skip to content

fix(datetime): avoid premature overflow in make_dt_interval (#109) - #119

Merged
andygrove merged 2 commits into
apache:mainfrom
Subodh-17:fix/make-dt-interval-overflow-109
Nov 16, 2026
Merged

andygrove merged 2 commits into
apache:mainfrom
Subodh-17:fix/make-dt-interval-overflow-109

Conversation

@Subodh-17

Copy link
Copy Markdown
Contributor

Summary

Fixes premature integer overflow in Spark's make_dt_interval expression (#109).

Context & Defect

make_dt_interval returns an Arrow Duration(Microsecond) backed by a 64-bit signed integer (i64), supporting durations up to $\approx \pm 106.75$ million days. However, intermediate conversions previously accumulated hours and minutes using 32-bit signed integers (i32). Consequently, any interval with days >= 1,491,309 (or days <= -1,491,309) overflowed i32 during minute calculation ($1,491,309 \times 24 \times 60 = 2,147,484,960 &gt; \text{i32::MAX}$), prematurely returning NULL for valid day-time intervals that Apache Spark evaluates successfully.

Key Changes

  1. Widening Intermediate Accumulation to i64: Promoted intermediate conversion constants HOURS_PER_DAY and MINS_PER_HOUR to i64 and widened day, hour, and min to i64 before multiplication and addition.
  2. Defensive Arithmetic: All accumulation steps use checked arithmetic (checked_mul, checked_add, checked_sub), safely returning None (NULL) on genuine overflow.
  3. Float Seconds Range Guard: Added float guard const MAX_SECS_ABS: f64 = (1i64 << 53) as f64; rejecting non-finite floats and $|sec| \ge 2^{53}$. This prevents float-to-int cast saturation in sec.trunc() as i64 without rejecting any representable interval (guard margin is $&gt;44\times$ beyond the theoretical maximum seconds that could ever produce a representable duration under extreme opposite-sign cancellation).
  4. Elimination of .abs(): Replaced frac_us.abs() with safe signed comparisons (frac_us >= MICROS_PER_SEC and frac_us <= -MICROS_PER_SEC) and checked carry operations (checked_add(1)? / checked_sub(1)?), eliminating debug panic risks on i64::MIN.
  5. Preserved True Overflow Contract (Spark make_dt_interval/make_interval return NULL on overflow instead of erroring #20): Genuine overflows beyond 64-bit duration limits continue to return NULL, preserving the distinct architectural boundary tracked under Spark make_dt_interval/make_interval return NULL on overflow instead of erroring #20 (error-raising on true overflow).

Spark Parity & Compatibility

  • Verified against live Apache Spark 4.3.0 execution (with Py4J 0.10.9.9 and ANSI enabled). Spark returns valid day-time intervals up to 106,751,991 days, matching this implementation.
  • Note: The original issue description references PySpark 4.2.0; behavior is identical across Spark 4.x releases.

Verification & Tests

  • Focused Unit Tests: 23/23 passed in crates/spark/src/function/datetime/make_dt_interval.rs (testing positive/negative boundaries, exact i64::MAX/i64::MIN, $\pm 1\ \mu\text{s}$ overflows, fractional carry rounding, and extreme floats).
  • Focused SQLLogicTests: 1/1 passed in crates/sqllogictest/test_files/datetime/make_dt_interval.slt (updating 1,500,000 days from NULL to valid interval and adding positive/negative boundary assertions).
  • Full Crate Tests: 345/345 passed (cargo test -p datafusion-spark).
  • Full SQLLogicTest Suite: 245/245 test files passed (cargo test -p datafusion-spark-sqllogictest).
  • Linters: cargo fmt --all -- --check clean; cargo clippy -p datafusion-spark --all-targets -- -D warnings clean (0 warnings); git diff --check clean.

Fixes #109.

Signed-off-by: Subodh Bansode <subodh171011@gmail.com>

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This is a first-pass automated review generated by Claude Code, posted ahead of human review. A maintainer will follow up with a proper review.

Thanks for the fix. Widening to i64 does fix #109. Every value in the new .slt cases matches PySpark 4.2.0, with ANSI on and off, UTC session: 1,491,309 / ±1,500,000 / ±106,751,991 days return intervals, ±106,751,992 days overflow, and i64::MAX / i64::MIN are exact at the boundary. The 23 unit tests and datetime/make_dt_interval.slt pass locally. I found one correctness problem in how the overflow is detected.

  1. crates/spark/src/function/datetime/make_dt_interval.rs:239-262: mixed-sign inputs that overflow in Spark now return a non-NULL value

    Spark's IntervalUtils.makeDayTimeInterval works in microseconds. It starts from the seconds and adds each part with an overflow check, in this order:

    var micros = secs.toUnscaledLong
    micros = Math.addExact(micros, Math.multiplyExact(days, MICROS_PER_DAY))
    micros = Math.addExact(micros, Math.multiplyExact(hours, MICROS_PER_HOUR))
    micros = Math.addExact(micros, Math.multiplyExact(mins, MICROS_PER_MINUTE))

    Spark raises an error whenever one of those steps overflows, even if a later part with the opposite sign would bring the total back into range. This PR adds days, hours and minutes into a single minute count in i64 before converting to microseconds. An opposite-sign part can therefore cancel the overflow before it is ever checked:

    Query Spark 4.2.0 (ANSI on and off) This PR
    make_dt_interval(106751992, -24, 0, 0) INTERVAL_ARITHMETIC_OVERFLOW error 106751991 days 0 hours 0 mins 0.000000 secs
    make_dt_interval(106751991, 5, -60, 0) INTERVAL_ARITHMETIC_OVERFLOW error 106751991 days 4 hours 0 mins 0.000000 secs

    Before this PR these inputs returned NULL, because day * 24 * 60 overflowed i32. That matched the "overflow returns NULL" behavior tracked in #20. Now they return a value where Spark has none.

    Suggested fix: follow Spark's order and add up in microseconds, roughly:

    let mut micros = secs_to_micros(sec)?; // checked, as today
    micros = micros.checked_add((day as i64).checked_mul(MICROS_PER_DAY)?)?;
    micros = micros.checked_add((hour as i64).checked_mul(MICROS_PER_HOUR)?)?;
    micros = micros.checked_add((min as i64).checked_mul(MICROS_PER_MINUTE)?)?;

    This is also simpler than the carry logic between the whole and fractional seconds. Please add the two queries above to make_dt_interval.slt, expecting NULL, with a comment pointing to #20, next to the existing ±106751992 cases. It would also help to have the exact i64::MAX / i64::MIN boundary in the .slt (make_dt_interval(106751991, 4, 0, 54.775807) and make_dt_interval(-106751991, -4, 0, -54.775808)). Both are confirmed in PySpark 4.2.0. Today the boundary is only covered by the unit tests.

    How verified: I ran the Spark results in PySpark 4.2.0 with spark.sql.ansi.enabled both true and false. I got the PR results by running the PR branch through a temporary .slt file.

🤖 Generated with Claude Code

Accumulate components sequentially in microseconds following Spark's exact IntervalUtils.makeDayTimeInterval evaluation order (seconds -> days -> hours -> minutes). Check each multiplication and addition independently using checked operations. Return NULL on overflow consistent with DataFusion Spark's documented contract (apache#20). Add full regression and boundary test coverage including maintainer-requested mixed-sign cases.

Signed-off-by: Subodh Bansode <subodh171011@gmail.com>

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. My earlier concern is fixed. The result is now built in microseconds in Spark's order (secs, then days, hours, mins), and each multiply and add is checked, so an opposite-sign part can no longer cancel an overflow. The two mixed-sign queries now return NULL with a pointer to #20, and the exact i64::MAX / i64::MIN boundaries are in the .slt.

I checked the new cases against PySpark 4.2.0 with ANSI on and off. The boundary values match. Both one-microsecond-past-boundary queries raise INTERVAL_ARITHMETIC_OVERFLOW in Spark, so NULL is right under #20. On my machine the unit tests, make_dt_interval.slt, clippy and fmt all pass.

Nit, optional: the new unit tests repeat the same array setup and downcast in every test. A helper like fn eval(day, hour, min, sec) -> Option<i64> would make them much shorter, and several of them repeat what the .slt already covers.

LGTM. I'll approve the CI run.

@andygrove
andygrove merged commit 7a17ed5 into apache:main Nov 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] make_dt_interval returns NULL for more than about 1.49 million days, Spark returns the interval

2 participants