Skip to content

support EXTRACT on intervals and durations - #12514

Merged
alamb merged 1 commit into
apache:mainfrom
nrc:interval-extract-2
Sep 20, 2024
Merged

alamb merged 1 commit into
apache:mainfrom
nrc:interval-extract-2

Conversation

@nrc

@nrc nrc commented Sep 17, 2024 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #6327.
Closes #7097

What changes are included in this PR?

Support extract/date_part by passing through requests to Arrow (where functionality was added in apache/arrow-rs#6071 and apache/arrow-rs#6246). There were also two changes required to the seconds helper function:

  • If the number of nanoseconds overflows a 32 bit integer, (e.g., a DayTime interval of 5 seconds), then querying the nanoseconds returns null. However, it is reasonable to return a number of seconds in this case. However, if there is a non-round number of seconds, this will give an inaccurate answer, e.g., 5.1 seconds will get returned as 5, but I'm not sure how to solve this - returning null for round numbers of seconds seems bad. We could fall back to milliseconds, but that complicates the code, is still imperfect, and has less predictable results. I also had a question about how I did this (REVIEW comment in the code).
  • The number of seconds was previously counted twice in cases where seconds are reflected in the number of nanoseconds.

A change which would fix both these issues (and I assume is an invariant which previously held) would be to not include whole seconds when returning nanoseconds. However, I think that breaks compat with Postgres.

Are these changes tested?

Yes, sqllogictests. The tests are not exhaustive because these features are properly tested in Arrow.

Are there any user-facing changes?

Users can use extract or date_part with intervals and durations.

@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 17, 2024

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nrc -- this is looking really close to me.

I think we should reasses the null handling (I left a suggestion) as well as some additional testing but otherwise this is ready to go 🚀

Signed-off-by: Nick Cameron <nrc@ncameron.org>
@nrc
nrc force-pushed the interval-extract-2 branch from 2d4732d to 3f91868 Compare September 20, 2024 02:13
@nrc

nrc commented Sep 20, 2024

Copy link
Copy Markdown
Contributor Author

Null handling and tests addressed. It took a tiny bit more code to make epoch work.

@alamb ready for re-review, thanks!

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great -- thank you @nrc and @samuelcolvin and @alexmojaki

})?;
Ok(Arc::new(r))
} else {
// Nulls in secs are preserved, nulls in subsecs are treated as zero to account for the case

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alamb
alamb merged commit 21ec332 into apache:main Sep 20, 2024
mbutrovich added a commit to mbutrovich/datafusion that referenced this pull request Oct 23, 2026
…6676)

## Which issue does this PR close?

- Closes apache#26673.

## Rationale for this change

The old code was complicated because it reused a helper written for
timestamps. apache#12514 sent `Duration` epochs through
[`seconds`](https://github.com/apache/datafusion/blob/c76ed5396ad3f4a4dc9d9ceeddc419a146b871f9/datafusion/functions/src/datetime/date_part.rs#L507-L547)
and added `% 1_000_000_000` and a NULL-as-zero branch to recover the
fraction, which works only while the total fits in `i32`. apache#19807 fixed
the same bug for `Interval` and left `Duration` on that path.

## What changes are included in this PR?

The `Duration` arms of `epoch` now divide the `i64` value by the unit's
factor with `unary`, like the `Timestamp` arms, and `seconds` is gone.
The first commit adds two cases to the `datetime_expressions` bench
without the fix, so the benchmark bot can compare against it.

It is also 2.5x to 12x faster on the benchmark bot. `main` made three
passes over the batch and fell back to a row-by-row iterator on any NULL
or large value. The new loop takes the same time whatever the values or
NULLs.

<img width="1600" height="868" alt="epoch-duration-bot"
src="https://github.com/user-attachments/assets/c70a3f6e-81d6-40ba-ab2d-0cbd832643fc"
/>

<details><summary>Numbers and codegen</summary>

Speedups over `353e4cee5` (the bench commit) on 1000 rows, with the
bot's thin LTO. "Nulls" means 10% of rows are NULL. The aligned runs add
`-align-all-functions=6 -align-all-nofallthru-blocks=6`.

| Case |
[Neoverse-V2](apache#26676 (comment))
|
[aligned](apache#26676 (comment))
| [Xeon
8581C](apache#26676 (comment))
|
[aligned](apache#26676 (comment))
|
|---|---|---|---|---|
| `duration_ns` | 5.60x | 5.57x | 8.09x | 6.95x |
| `duration_ns_nulls` | 7.11x | 7.31x | 12.08x | 10.68x |
| `duration_ns_small` | 2.50x | 2.45x | 4.50x | 3.53x |
| `ns` (unchanged) | 1.00x | 1.01x | 1.09x | 1.03x |

In the default Xeon run the unchanged cases are 1.09x to 1.19x, so that
run overstates the gain by about 15%.

Timing each arrow kernel of `main`'s path separately on 8192 rows on an
Apple M5 Max accounts for its totals.

| Values | `date_part(Second)` | `date_part(Nanosecond)` | Combine | Sum
|
|---|---|---|---|---|
| Under 2.1 s, `binary` | 2.94 us | 2.32 us | 2.44 us | 7.70 us |
| Large, iterator `collect` | 2.92 us | 12.86 us | 16.47 us | 32.25 us |

LLVM merges the new loops with the existing `Timestamp` and `Time64`
ones, so this PR adds no new loop code. On aarch64 the loop handles 8
rows per iteration with `scvtf.2d` and `fdiv.2d`. On x86_64 the division
is packed (`divpd`, `vdivpd` with AVX2) and the `i64` to `f64`
conversion is scalar `cvtsi2sd`, because the packed form needs
AVX-512DQ. I read this with `cargo asm` (rustc 1.99.0,
`release-nonlto`).

</details>

The code divides rather than multiplying by `1e-9`. The division is most
of the loop's time on Neoverse-V2, where `date_part_epoch_s_1000` (no
division) takes 412 ns against 1.21 us for `ns`. The multiply would be
faster, but it changes 14% of `Duration(ms)` and 41% of `Duration(ns)`
results, for example 12.92335 s becomes `12.923350000000001`.

## What is the testing strategy for this PR?

New cases in `datafusion/sqllogictest/test_files/datetime/date_part.slt`
under `test_extract_duration` cover the `i32::MAX` ns boundary, each
`Duration` unit, the issue's reproducer, negative values, NULLs mixed
with values, and `i64` extremes.

## Are there any user-facing changes?

`EXTRACT(EPOCH)` on `Duration` returns correct results. There are no
public API changes.

## AI Disclosure

I wrote this PR with help from Claude but understand and support the
core approach, implementation, and test coverage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

is there a way to get a result similar to datediff function Extract from interval type failed

4 participants