Repository navigation
Conversation
When merging file statistics via try_merge_iter, an input whose column is entirely NULL (null_count == num_rows) or empty (num_rows == 0) now has its Absent min/max skipped instead of poisoning the merged result. Previously, one null-partition file (e.g. d=__HIVE_DEFAULT_PARTITION__) would wipe out the min/max bounds for the entire file group, defeating partition pruning for any table with null partitions. The fix adds is_all_null_or_empty() which checks the pre-computed stats header counts (no row scanning). When counts are inexact or absent, the function conservatively returns false to preserve the existing Absent behavior. Closes apache#26748
5e3c78d to
23728ea
Compare
|
@alamb This is ready for review whenever you have a chance. CI is green and the approach matches the design @asolimando approved on #26993. Happy to address any feedback. |
The accumulator's all-null check compared a null_count that already included the incoming item's nulls against the pre-merge row count, which could adopt the item's bounds and produce wrong min/max. Adds regression tests.
|
Can you please update this Pr / the issue to explain what problem you are fixing from an end user perspective (e.g. a query that is getting wrong results, etc) -- right now it is all in terms of code, and it is not clear to me what we are fixing |
|
Thanks @alamb -- good point. From an end user perspective: Since #26310, DataFusion reads Hive Example: a table partitioned by EXPLAIN SELECT min(d), max(d) FROM t;On main, this scans all files: With this PR, it is answered from statistics with no I/O: The same I'll update the PR description to lead with this, and change the sqllogictest to assert on the |
Which issue does this PR close?
Rationale
After #26310 landed, Hive null partitions (
__HIVE_DEFAULT_PARTITION__) are correctly read as NULL withAbsentmin/max statistics. However, whenStatistics::try_merge_itermerges file statistics into file-group or table-level stats,precision_min/precision_maxreturnAbsentif either input isAbsent. This means a single null-partition file wipes out the min/max bounds for its entire file group, defeating partition pruning for any table that has null partitions.Changes
Added
is_all_null_or_empty()that checks the pre-computed stats counts (no row scanning): it returnstruewhennull_count == num_rows(bothExact) ornum_rows == Exact(0).Modified the merge loop in
try_merge_iter_with_ndv_fallbackto skip min/max from all-null or empty inputs instead of letting theirAbsentbounds poison the result. When counts are inexact or absent, the function conservatively returnsfalseto preserve the existing behavior.Updated the
FileGroupdoc comment to reflect that all-null partitions are now skipped.Tests
Added 5 test functions:
test_is_all_null_or_empty— unit tests for the helpertest_try_merge_iter_skips_all_null_partition— real + null merge in both orders, and null + nulltest_try_merge_iter_inexact_null_count_stays_conservative— inexact counts degrade to Absenttest_try_merge_iter_empty_partition_skipped— empty partition (0 rows) is skippedtest_try_merge_iter_three_files_with_null_partition— three-way merge with a null partition in the middleAll 82 stats tests pass (including existing tests), zero regressions.