Repository navigation
fix: clear error instead of panic on mismatched column statistics - #25694
Conversation
count's statistics fast path indexed column_statistics[col.index()] directly, so a TableProvider reporting fewer column statistics than its schema made SELECT count(col) panic. Use .get(idx)? in both branches, matching the guard already in sum.rs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25694 +/- ##
=======================================
Coverage 82.93% 82.94%
=======================================
Files 1154 1154
Lines 458002 458075 +73
Branches 458002 458075 +73
=======================================
+ Hits 379856 379930 +74
+ Misses 55123 55122 -1
Partials 23023 23023 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kosiew
left a comment
There was a problem hiding this comment.
Thanks for fixing this. Both COUNT and COUNT(DISTINCT) now handle missing column statistics safely, and the new regression tests cover both paths. The change looks good to me. I have one optional suggestion for additional end-to-end coverage.
A statistics provider that drops the trailing column entry keeps the aggregated column's index out of bounds whatever the projection, so `count`'s statistics fast path is reached from SQL. The query panics on main and returns the correct count with the guard in place.
alamb
left a comment
There was a problem hiding this comment.
Thank you for this PR @jackylee-ch and for the review @kosiew
I feel like this is just papering over a bug and by sliently ignoring the problem we will make debugging harder
Changing the code so it doesn't panic seems fine, but if the table provider returns the wrong number of statistics I think that should return a clear error
|
Agreed — it should be an error. Two candidates: the provider/source boundary, or One thing first: planning-time rules compute statistics ( |
I don't understand what you are asking me What is the "broad gate"? |
`Statistics::column_statistics` must hold one entry per schema field, and consumers index it by column. Check that where `StatisticsContext` receives statistics from an operator or a statistics provider, and fail with an error naming the culprit instead of panicking in a consumer.
`value_from_stats` is public, so a direct caller can still pass short statistics; fold the two `count` cases into one loop. Link `ColumnStatistics` in the `StatisticsProvider` docs.
`min` and `max` index `column_statistics` like `count` did, and `value_from_stats` is public, so use `.get(idx)?` there as well. The end-to-end test now also checks that a plain scan fails planning.
Extra entries never indexed out of bounds; only missing ones did.
`FilterExec` indexes its input's column statistics when it is built, so a table provider returning too few entries also panicked on `WHERE`.
A statistics provider returning the statistics of the whole table for a scan of only `b` used to make `min(b)` and `max(b)` return the minimum and maximum of `a`. Check end to end that planning fails instead, and that both statistics sources reject one entry too many as well as too few. Also pin what file scans accept: with a projection, as all built-in sources have, table and file group statistics of any length, missing trailing entries being unknown and extra ones ignored; without one, exactly one entry per table column. Say so in the upgrade guide.
`FileScanConfig` pads table and file group statistics it is given and ignores extra entries when its source has a projection, so a missing or extra entry before the last column shifts the statistics of the columns after it. Describe that as unchecked rather than as accepted.
`FileScanConfig` looks up the table and file group statistics it is given by column position, so an extra entry anywhere but at the end shifted the statistics of every later column by one, and `min`/`max` could be answered from another column's values. Return an internal error naming the statistics instead. Missing trailing entries are still unknown, which virtual columns and some table providers rely on.
|
Sorry, that was unclear. "Broad gate" meant one check in Too few entries can make filters, joins, |
|
I am sorry -- I still don't get it. How about we start with a clear and concise description of what problem you are solving (from the end user API perspective, not a code description) Describe:
It seems like you are trying to add better error messages when a table provider provides incorrect statistics ? But that is not clear |
The error now reads, for example, "MyExec returned 1 column statistics, expected 2 (one per field of its schema)", instead of starting with "Assertion failed: column_statistics == fields (left: 1, right: 2)". Also drop the FileScanConfig check for extra table and file group column statistics. When its FileSource has a projection, FileScanConfig fills in missing statistics as unknown on purpose, for virtual columns and some table providers, so it needs a different check; only the statistics of plans and statistics providers are checked now. Note on Statistics::default() that it does not fit a schema with fields. In the upgrade guide, use self.schema() in the example and mention Statistics::project for plans that drop or reorder their input's columns.
|
I've updated the description; does it answer your questions? Your guess is right: a clear planning error instead of a panic or a wrong answer. In short:
|
alamb
left a comment
There was a problem hiding this comment.
Thanks @jackylee-ch and @kosiew
This is getting closer, but still far too long in my opinion -- can you please read and review all lines and verify you think they are all necessary
| /// Returns an error unless `statistics`, computed for `plan` by `provider` or, | ||
| /// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field | ||
| /// of `plan`'s schema, as [`Statistics::column_statistics`] requires. Consumers | ||
| /// index it by column, so a mismatch could make them go out of bounds or read | ||
| /// the statistics of another column. | ||
| /// | ||
| /// [`ColumnStatistics`]: datafusion_common::ColumnStatistics |
There was a problem hiding this comment.
| /// Returns an error unless `statistics`, computed for `plan` by `provider` or, | |
| /// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field | |
| /// of `plan`'s schema, as [`Statistics::column_statistics`] requires. Consumers | |
| /// index it by column, so a mismatch could make them go out of bounds or read | |
| /// the statistics of another column. | |
| /// | |
| /// [`ColumnStatistics`]: datafusion_common::ColumnStatistics | |
| /// Returns an error if the `statistics` mismatches the plan's schema | |
| /// | |
| /// [`ColumnStatistics`]: datafusion_common::ColumnStatistics |
There was a problem hiding this comment.
Done in cc8d175: the doc is now one line.
| /// If `args.partition()` is `None`, it returns statistics for all partitions. | ||
| /// | ||
| /// The result must hold one [`ColumnStatistics`] per field of | ||
| /// [`Self::schema`]; [`StatisticsContext::compute`] returns an error otherwise. |
There was a problem hiding this comment.
the detail of what returns an erorr is likely irrelevant
[
StatisticsContext::compute] returns an error otherwise.
There was a problem hiding this comment.
Done in cc8d175: dropped the sentence about the error.
| statistics registry can change the result, because the providers are then | ||
| consulted. | ||
|
|
||
| ### Statistics with the wrong number of column statistics are an error |
There was a problem hiding this comment.
this is way too detailed for a new error message. I don't think we need to document that the code detects some errors earlier
There was a problem hiding this comment.
Done in cc8d175: the entry is now 14 lines (was 41), without the "who is affected" list and the FileScanConfig note.
|
Trimmed in cc8d175: the diff against |
kosiew
left a comment
There was a problem hiding this comment.
@jackylee-ch, thanks for addressing the malformed column statistics issue. The approach is sound, and I found no blocking runtime defects. Valid statistics retain their existing behavior, while mismatched statistics now produce actionable errors instead of downstream panics or incorrect aggregate results.
StatisticsContext is the right enforcement boundary because it handles both ExecutionPlan and StatisticsProvider results before caching or parent consumption. The defensive COUNT, MIN, and MAX checks also protect direct callers that bypass this context. This avoids relying on individual producers to validate themselves or changing public APIs.
I'm approving with suggestions. Please move the new upgrade note to the 57.0.0 guide before merging, since the reviewed head targets 57.0.0-dev. The additional boundary, partition, and provider-documentation suggestions are non-blocking.
| operator, as it does above `CoalescePartitionsExec`, and is copied below as | ||
| an early-stop fetch. | ||
|
|
||
| ### Statistics must have one `ColumnStatistics` per schema field |
There was a problem hiding this comment.
Please move the new statistics-contract section to docs/source/library-user-guide/upgrading/57.0.0.md, since the reviewed head declares 57.0.0-dev in Cargo.toml. Keeping it here incorrectly attributes the new planning-time rejection to 56.0.0. Move only the newly added section and leave the existing guidance intact.
There was a problem hiding this comment.
Confirmed, main is 57.0.0-dev; moved to 57.0.0.md in 6e23b65.
| { | ||
| let (statistics, extensions) = computed.into_parts(); | ||
| validate_column_statistics(plan, &statistics, Some(provider.as_ref()))?; | ||
| let extensions = if extensions.is_empty() { |
There was a problem hiding this comment.
Optional, low priority: could we test a partition-aware provider that returns valid overall statistics but malformed statistics for Some(0)? Call compute_arc for both requests using the same context and plan, then assert overall success and a provider/count error for partition 0. This would verify that cached overall statistics do not mask invalid partition results. The provider must handle partition requests explicitly rather than delegate them.
There was a problem hiding this comment.
Not adding: the cache key includes the partition and validation runs before the store, so a cached overall result cannot mask a bad Some(0).
|
@alamb, |
alamb
left a comment
There was a problem hiding this comment.
Looks good to me - -thank you @jackylee-ch and @kosiew
Which issue does this PR close?
analysis.rsbounds check panic #10536, where planning panicked on a user's custom statistics; propagate error instead of panicking on out of bounds in physical-expr/src/analysis.rs #10992 turned only that panic, inExprBoundaries::try_from_column, into an error.Rationale for this change
A custom
ExecutionPlanorStatisticsProvidercan reportcolumn_statisticswith a different length than its schema, for example by starting fromStatistics::default(). Rules and operators index it by column, so the mismatch surfaces far from its cause, as an index out of bounds, as in anvai-labs/proximaDB#76, SkardiLabs/skardi#276 and spiceai/spiceai#14144, or, with too many entries, as another column's values:min(b)returned the minimum ofa.What changes are included in this PR?
StatisticsContextchecks that the statistics of every plan and provider have one entry per schema field, and otherwise returns an error naming the culprit, e.g.MyExec returned 1 column statistics, expected 2 (one per field of its schema). EvenSELECT *over such a plan fails at planning, becauseEnsureRequirementscomputes statistics. Padding with unknown entries would hide the bug.count,minandmaxreturnNoneon a missing entry instead of panicking, assumdoes, because the publicvalue_from_statscannot return an error.Not covered: the table and file group statistics a table provider passes to a file scan, which can legitimately lack entries (virtual columns, delta-rs) and need a different rule.
What is the testing strategy for this PR?
An end-to-end test and unit tests for too few and too many entries, from an operator and from a provider. The check is one length comparison per plan, about 5 ns. Planning TPC-H takes 15.24 ms vs 15.17 ms on
main, TPC-DS 240.3 ms vs 240.1 ms (median of 3 runs).Are there any user-facing changes?
Yes, breaking for extensions: wrong-length statistics now fail planning. The upgrade guide shows the fix. Please add the
api changelabel.