Repository navigation
propagate error instead of panicking on out of bounds in physical-expr/src/analysis.rs - #10992
Merged
Merged
Conversation
jonahgao
reviewed
Jun 20, 2024
| let field = schema.fields() | ||
| .get(col_index) | ||
| .ok_or_else(|| DataFusionError::Internal( | ||
| format!("Could not create `ExprBoundaries`: in `try_from_column` `col_index` has gone out of bounds with a value of {col_index}, the schema has {} columns while.", schema.fields.len()) |
Member
There was a problem hiding this comment.
Using internal_datafusion_err could simplify it.
let field = schema.fields().get(col_index).ok_or_else(|| {
internal_datafusion_err!(
"Could not create `ExprBoundaries`: in `try_from_column` `col_index`
has gone out of bounds with a value of {col_index}, the schema has {} columns while.",
schema.fields.len()
)
})?;
Contributor
Author
There was a problem hiding this comment.
Ah yes, that's much more nicer. Thanks!
jonahgao
approved these changes
Jun 20, 2024
jonahgao
left a comment
Member
There was a problem hiding this comment.
Looks good to me! Thanks @LorrensP-2158466
xinlifoobar
pushed a commit
to xinlifoobar/datafusion
that referenced
this pull request
Jun 22, 2024
…r/src/analysis.rs (apache#10992) * propogate error instead of panicking * use macro for creating internal df error
xinlifoobar
pushed a commit
to xinlifoobar/datafusion
that referenced
this pull request
Jun 22, 2024
…r/src/analysis.rs (apache#10992) * propogate error instead of panicking * use macro for creating internal df error
findepi
pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
…r/src/analysis.rs (apache#10992) * propogate error instead of panicking * use macro for creating internal df error
comphead
pushed a commit
to comphead/arrow-datafusion
that referenced
this pull request
Nov 19, 2026
…ache#25694) ## Which issue does this PR close? - Related to apache#10536, where planning panicked on a user's custom statistics; apache#10992 turned only that panic, in `ExprBoundaries::try_from_column`, into an error. ## Rationale for this change A custom `ExecutionPlan` or `StatisticsProvider` can report `column_statistics` with a different length than its schema, for example by starting from `Statistics::default()`. Rules and operators index it by column, so the mismatch surfaces far from its cause, as an index out of bounds, as in anvai-labs/proximaDB#76, SkardiLabs/skardi#276 and spiceai/spiceai#14144, or, with too many entries, as another column's values: `min(b)` returned the minimum of `a`. ## What changes are included in this PR? `StatisticsContext` checks that the statistics of every plan and provider have one entry per schema field, and otherwise returns an error naming the culprit, e.g. `MyExec returned 1 column statistics, expected 2 (one per field of its schema)`. Even `SELECT *` over such a plan fails at planning, because `EnsureRequirements` computes statistics. Padding with unknown entries would hide the bug. `count`, `min` and `max` return `None` on a missing entry instead of panicking, as `sum` does, because the public `value_from_stats` cannot return an error. Not covered: the table and file group statistics a table provider passes to a file scan, which can legitimately lack entries (virtual columns, delta-rs) and need a different rule. ## What is the testing strategy for this PR? An end-to-end test and unit tests for too few and too many entries, from an operator and from a provider. The check is one length comparison per plan, about 5 ns. Planning TPC-H takes 15.24 ms vs 15.17 ms on `main`, TPC-DS 240.3 ms vs 240.1 ms (median of 3 runs). ## Are there any user-facing changes? Yes, breaking for extensions: wrong-length statistics now fail planning. The upgrade guide shows the fix. Please add the `api change` label. --------- Co-authored-by: kosiew <kosiew@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #10536.
Rationale for this change
In
datafusion/datafusion/physical-expr/src/analysis.rs
Line 95 in cafbc9d
The function panics when
col_indexis out of bounds when returning an internal error with extra context is much more convenient.What changes are included in this PR?
Propagate the error as en
DataFusionError::Internal.Are these changes tested?
Are there any user-facing changes?
Users may need to handle that error case