Repository navigation
fix(array): between over a constant null array is null, not false - #10744
Conversation
The Constant between reduce rule compared the constant scalar with the bounds directly. Scalar orders null below every value, so an all-null chunk (which compression stores as a ConstantArray) gave a non-null false, and not(between(x, a, b)) over it selected every row instead of none. Return a null constant when the array constant is null. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
|
You could put this change in the short_circuit method of between adaptor, then it would easily apply everywhere |
Move the null check from the Constant between reduce rule into short_circuit, which runs before every encoding's between. Signed-off-by: gitRasheed <99349713+gitRasheed@users.noreply.github.com>
Sure, moved it into short_circuit now |
Merging this PR will degrade performance by 3.84%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Summary
In DataFusion,
a NOT IN (1, 2, 3, 4, 5)counted null rows that sit in an all-null chunk.Data:
idInt32 0..999.aInt32 nullable, null for rows 0..499 andid % 20for rows 500..999. Written as ten 100-row chunks (WriteStrategyBuilderwithwith_row_block_size(100)andwith_data_block_target_bytes(None)), so rows 0..499 are five all-null chunks.A DataFusion
MemTablewith the same data returns 375. On develop Vortex returns 875, so it counts all 500 null rows. With this change it returns 375.vortex-datafusion turns a NOT IN of consecutive integers into
not(between(...))(in_list_ofinvortex-array/src/expr/exprs.rs) and reports the filter as pushed, so DataFusion does not check it again. Each all-null chunk is aConstantArray. TheConstantBetweenReducerule compared it with the constant bounds usingScalar's ordering, which puts null below every value. Sobetweenreturnedfalseinstead of null, andnotturned that intotrue. In normal dispatch null bounds are handled earlier byshort_circuit, so a null array value is the case that got through.Changes
short_circuit, which runs before every encoding'sbetween, now returns a nullBoolconstant when the array is a null constant, with the combined nullability of the array and both bounds.The new test
scalar_fn::fns::between::tests::constant_null_arraychecks thatbetweenandnot(between)are all null forf64andi32.Checked locally:
cargo test -p vortex-array --lib between(46 passed, and the new test fails on develop), the DataFusion query above (875 on develop, 375 with this change), clippy with-D warnings, and nightly fmt.I used AI tools to help write and review this change.