Skip to content

Let scalar functions reach the values of sliced run-end arrays - #11388

Closed
asubiotto wants to merge 1 commit into
vortex-data:developfrom
dash0hq:alfonso/runend-slice-reduce
Closed

asubiotto wants to merge 1 commit into
vortex-data:developfrom
dash0hq:alfonso/runend-slice-reduce

Conversation

@asubiotto

Copy link
Copy Markdown
Contributor

Summary

A reader that reads a row range of a chunk slices the array, and a scalar function such as lower is often applied on top. For a run-end array the optimizer kept lower(slice(runend)), because run-end slicing exists only as an execute kernel that needs an execution context. The existing run-end scalar function rule never saw a run-end child, so the function ran once per row instead of once per run, or once per dictionary value when the run values are a dictionary.

Changes

  • Add SliceReduce for RunEnd, registered with SliceReduceAdaptor in the same way as Dict. When the ends are an unsigned primitive array in host memory, it finds the first and last kept run with a binary search, slices the ends and the values, and keeps the logical offset. In all other cases it returns None, and the existing execute kernel slices as before.
  • Add a trace test: lower over a slice of runend(dict) now optimizes to runend(dict(lower(values))) and returns the expected values. The test fails without the change.

This PR was written with the help of an AI coding agent. I reviewed the change and the tests.

A slice over a run-end array had no reduce rule, so the optimizer kept
lower(slice(runend)) and evaluated the function once per row. Slicing
host-resident primitive ends now produces a run-end array directly, so
the existing run-end scalar function rule pushes the function into the
run values, and through a dictionary into its values.

Co-authored-by: Amp <amp@ampcode.com>
Signed-off-by: Alfonso Subiotto <alfonso.subiotto@dash0.com>
@robert3005

Copy link
Copy Markdown
Contributor

FWIW I think the generic fix for this issue is #11310

@asubiotto

Copy link
Copy Markdown
Contributor Author

Thanks! Our problem is that we would like lower(slice(runend(dict))) to run on the dict values. This draft PR is just one part of an approach suggested by Amp I haven't verified yet. I'll take a look at #11310.

@robert3005

Copy link
Copy Markdown
Contributor

Right so I had the same issue where I have Between(Slice(Runend)) and because for runend slice is an execute rule and between is a reduce rule the between reduce never gets to run because we are already in execution. Should be similar here

@asubiotto

Copy link
Copy Markdown
Contributor Author

Yes, closing this and opening a different stack.

@asubiotto asubiotto closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants