Repository navigation
support EXTRACT on intervals and durations - #12514
Merged
Merged
Conversation
alexmojaki
reviewed
Sep 19, 2024
alamb
reviewed
Sep 19, 2024
Signed-off-by: Nick Cameron <nrc@ncameron.org>
nrc
force-pushed
the
interval-extract-2
branch
from
September 20, 2024 02:13
2d4732d to
3f91868
Compare
Contributor
Author
|
Null handling and tests addressed. It took a tiny bit more code to make @alamb ready for re-review, thanks! |
alamb
approved these changes
Sep 20, 2024
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks great -- thank you @nrc and @samuelcolvin and @alexmojaki
| })?; | ||
| Ok(Arc::new(r)) | ||
| } else { | ||
| // Nulls in secs are preserved, nulls in subsecs are treated as zero to account for the case |
mbutrovich
added a commit
to mbutrovich/datafusion
that referenced
this pull request
Oct 23, 2026
…6676) ## Which issue does this PR close? - Closes apache#26673. ## Rationale for this change The old code was complicated because it reused a helper written for timestamps. apache#12514 sent `Duration` epochs through [`seconds`](https://github.com/apache/datafusion/blob/c76ed5396ad3f4a4dc9d9ceeddc419a146b871f9/datafusion/functions/src/datetime/date_part.rs#L507-L547) and added `% 1_000_000_000` and a NULL-as-zero branch to recover the fraction, which works only while the total fits in `i32`. apache#19807 fixed the same bug for `Interval` and left `Duration` on that path. ## What changes are included in this PR? The `Duration` arms of `epoch` now divide the `i64` value by the unit's factor with `unary`, like the `Timestamp` arms, and `seconds` is gone. The first commit adds two cases to the `datetime_expressions` bench without the fix, so the benchmark bot can compare against it. It is also 2.5x to 12x faster on the benchmark bot. `main` made three passes over the batch and fell back to a row-by-row iterator on any NULL or large value. The new loop takes the same time whatever the values or NULLs. <img width="1600" height="868" alt="epoch-duration-bot" src="https://github.com/user-attachments/assets/c70a3f6e-81d6-40ba-ab2d-0cbd832643fc" /> <details><summary>Numbers and codegen</summary> Speedups over `353e4cee5` (the bench commit) on 1000 rows, with the bot's thin LTO. "Nulls" means 10% of rows are NULL. The aligned runs add `-align-all-functions=6 -align-all-nofallthru-blocks=6`. | Case | [Neoverse-V2](apache#26676 (comment)) | [aligned](apache#26676 (comment)) | [Xeon 8581C](apache#26676 (comment)) | [aligned](apache#26676 (comment)) | |---|---|---|---|---| | `duration_ns` | 5.60x | 5.57x | 8.09x | 6.95x | | `duration_ns_nulls` | 7.11x | 7.31x | 12.08x | 10.68x | | `duration_ns_small` | 2.50x | 2.45x | 4.50x | 3.53x | | `ns` (unchanged) | 1.00x | 1.01x | 1.09x | 1.03x | In the default Xeon run the unchanged cases are 1.09x to 1.19x, so that run overstates the gain by about 15%. Timing each arrow kernel of `main`'s path separately on 8192 rows on an Apple M5 Max accounts for its totals. | Values | `date_part(Second)` | `date_part(Nanosecond)` | Combine | Sum | |---|---|---|---|---| | Under 2.1 s, `binary` | 2.94 us | 2.32 us | 2.44 us | 7.70 us | | Large, iterator `collect` | 2.92 us | 12.86 us | 16.47 us | 32.25 us | LLVM merges the new loops with the existing `Timestamp` and `Time64` ones, so this PR adds no new loop code. On aarch64 the loop handles 8 rows per iteration with `scvtf.2d` and `fdiv.2d`. On x86_64 the division is packed (`divpd`, `vdivpd` with AVX2) and the `i64` to `f64` conversion is scalar `cvtsi2sd`, because the packed form needs AVX-512DQ. I read this with `cargo asm` (rustc 1.99.0, `release-nonlto`). </details> The code divides rather than multiplying by `1e-9`. The division is most of the loop's time on Neoverse-V2, where `date_part_epoch_s_1000` (no division) takes 412 ns against 1.21 us for `ns`. The multiply would be faster, but it changes 14% of `Duration(ms)` and 41% of `Duration(ns)` results, for example 12.92335 s becomes `12.923350000000001`. ## What is the testing strategy for this PR? New cases in `datafusion/sqllogictest/test_files/datetime/date_part.slt` under `test_extract_duration` cover the `i32::MAX` ns boundary, each `Duration` unit, the issue's reproducer, negative values, NULLs mixed with values, and `i64` extremes. ## Are there any user-facing changes? `EXTRACT(EPOCH)` on `Duration` returns correct results. There are no public API changes. ## AI Disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6327.
Closes #7097
What changes are included in this PR?
Support
extract/date_partby passing through requests to Arrow (where functionality was added in apache/arrow-rs#6071 and apache/arrow-rs#6246). There were also two changes required to thesecondshelper function:A change which would fix both these issues (and I assume is an invariant which previously held) would be to not include whole seconds when returning nanoseconds. However, I think that breaks compat with Postgres.
Are these changes tested?
Yes, sqllogictests. The tests are not exhaustive because these features are properly tested in Arrow.
Are there any user-facing changes?
Users can use
extractordate_partwith intervals and durations.