Skip to content

fix(array): between over a constant null array is null, not false - #10744

Merged
robert3005 merged 3 commits into
vortex-data:developfrom
gitRasheed:not-between-all-null
Oct 26, 2026
Merged

robert3005 merged 3 commits into
vortex-data:developfrom
gitRasheed:not-between-all-null

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

In DataFusion, a NOT IN (1, 2, 3, 4, 5) counted null rows that sit in an all-null chunk.

Data: id Int32 0..999. a Int32 nullable, null for rows 0..499 and id % 20 for rows 500..999. Written as ten 100-row chunks (WriteStrategyBuilder with with_row_block_size(100) and with_data_block_target_bytes(None)), so rows 0..499 are five all-null chunks.

SELECT count(*) FROM t WHERE a NOT IN (1, 2, 3, 4, 5)

A DataFusion MemTable with the same data returns 375. On develop Vortex returns 875, so it counts all 500 null rows. With this change it returns 375.

vortex-datafusion turns a NOT IN of consecutive integers into not(between(...)) (in_list_of in vortex-array/src/expr/exprs.rs) and reports the filter as pushed, so DataFusion does not check it again. Each all-null chunk is a ConstantArray. The Constant BetweenReduce rule compared it with the constant bounds using Scalar's ordering, which puts null below every value. So between returned false instead of null, and not turned that into true. In normal dispatch null bounds are handled earlier by short_circuit, so a null array value is the case that got through.

Changes

short_circuit, which runs before every encoding's between, now returns a null Bool constant when the array is a null constant, with the combined nullability of the array and both bounds.

The new test scalar_fn::fns::between::tests::constant_null_array checks that between and not(between) are all null for f64 and i32.

Checked locally: cargo test -p vortex-array --lib between (46 passed, and the new test fails on develop), the DataFusion query above (875 on develop, 375 with this change), clippy with -D warnings, and nightly fmt.

I used AI tools to help write and review this change.

The Constant between reduce rule compared the constant scalar with the
bounds directly. Scalar orders null below every value, so an all-null
chunk (which compression stores as a ConstantArray) gave a non-null
false, and not(between(x, a, b)) over it selected every row instead of
none. Return a null constant when the array constant is null.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@robert3005

Copy link
Copy Markdown
Contributor

You could put this change in the short_circuit method of between adaptor, then it would easily apply everywhere

Move the null check from the Constant between reduce rule into
short_circuit, which runs before every encoding's between.

Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
@gitRasheed

Copy link
Copy Markdown
Contributor Author

You could put this change in the short_circuit method of between adaptor, then it would easily apply everywhere

Sure, moved it into short_circuit now

@robert3005 robert3005 added the changelog/fix A bug fix label Oct 26, 2026
@robert3005
robert3005 enabled auto-merge (squash) October 26, 2026 10:16
@codspeed

codspeed Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 3.84%

4 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ Unable to measure the execution of 2 benchmarks

These benchmarks measured no execution time, so their results are not comparable and count as unchanged. This usually happens when the compiler removes the code under test.

Prevent compiler optimizations

See affected benchmarks

⚠️ 19 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ Base and head ran on different runner types

Some benchmarks with significant performance changes were compared across different runner types, so the changes might not come from the code. Run base and head on the same runner type, or use CodSpeed Macro Runners for more stable results.

Learn more about Macro Runners

Open the report in CodSpeed to investigate

ℹ️ 40 benchmarks have no flamegraph

The flamegraph wasn't generated: the symbols of the profiling trace could not be resolved, because no information about the language frames was collected.

See affected benchmarks

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 2437 untouched benchmarks
⏩ 12 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ CPU Simulation missing_stats[16] 68.8 µs 84.3 µs -18.36%
⚡ CPU Simulation test_scan_filter[int] 3.4 ms 3 ms +13.26%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing gitRasheed:not-between-all-null (e748b8a) with develop (0d4ddfb)

Open in CodSpeed

Footnotes

  1. 12 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005
robert3005 merged commit 33675d9 into vortex-data:develop Oct 26, 2026
79 of 81 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants