Skip to content

propagate error instead of panicking on out of bounds in physical-expr/src/analysis.rs - #10992

Merged
jonahgao merged 2 commits into
apache:mainfrom
LorrensP-2158466:analysis-bound-check
Jun 20, 2024
Merged

jonahgao merged 2 commits into
apache:mainfrom
LorrensP-2158466:analysis-bound-check

Conversation

@LorrensP-2158466

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #10536.

Rationale for this change

In

let field = &schema.fields()[col_index];

The function panics when col_index is out of bounds when returning an internal error with extra context is much more convenient.

What changes are included in this PR?

Propagate the error as en DataFusionError::Internal.

Are these changes tested?

Are there any user-facing changes?

Users may need to handle that error case

@LorrensP-2158466 LorrensP-2158466 changed the title propogate error instead of panicking propagate error instead of panicking Jun 19, 2024
@LorrensP-2158466 LorrensP-2158466 changed the title propagate error instead of panicking propagate error instead of panicking on out of bounds in physical-expr/src/analysis.rs Jun 19, 2024
let field = schema.fields()
.get(col_index)
.ok_or_else(|| DataFusionError::Internal(
format!("Could not create `ExprBoundaries`: in `try_from_column` `col_index` has gone out of bounds with a value of {col_index}, the schema has {} columns while.", schema.fields.len())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using internal_datafusion_err could simplify it.

 let field = schema.fields().get(col_index).ok_or_else(|| {
            internal_datafusion_err!(
                "Could not create `ExprBoundaries`: in `try_from_column` `col_index` 
                has gone out of bounds with a value of {col_index}, the schema has {} columns while.",
                schema.fields.len()
            )
        })?;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes, that's much more nicer. Thanks!

@github-actions github-actions Bot added the physical-expr Changes to the physical-expr crates label Jun 20, 2024

@jonahgao jonahgao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Thanks @LorrensP-2158466

@jonahgao
jonahgao merged commit 5316278 into apache:main Jun 20, 2024
@LorrensP-2158466
LorrensP-2158466 deleted the analysis-bound-check branch June 20, 2024 18:22
xinlifoobar pushed a commit to xinlifoobar/datafusion that referenced this pull request Jun 22, 2024
…r/src/analysis.rs (apache#10992)

* propogate error instead of panicking

* use macro for creating internal df error
xinlifoobar pushed a commit to xinlifoobar/datafusion that referenced this pull request Jun 22, 2024
…r/src/analysis.rs (apache#10992)

* propogate error instead of panicking

* use macro for creating internal df error
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
…r/src/analysis.rs (apache#10992)

* propogate error instead of panicking

* use macro for creating internal df error
comphead pushed a commit to comphead/arrow-datafusion that referenced this pull request Nov 19, 2026
…ache#25694)

## Which issue does this PR close?

- Related to apache#10536, where planning panicked on a user's custom
statistics; apache#10992 turned only that panic, in
`ExprBoundaries::try_from_column`, into an error.

## Rationale for this change

A custom `ExecutionPlan` or `StatisticsProvider` can report
`column_statistics` with a different length than its schema, for example
by starting from `Statistics::default()`. Rules and operators index it
by column, so the mismatch surfaces far from its cause, as an index out
of bounds, as in anvai-labs/proximaDB#76, SkardiLabs/skardi#276 and
spiceai/spiceai#14144, or, with too many entries, as another column's
values: `min(b)` returned the minimum of `a`.

## What changes are included in this PR?

`StatisticsContext` checks that the statistics of every plan and
provider have one entry per schema field, and otherwise returns an error
naming the culprit, e.g. `MyExec returned 1 column statistics, expected
2 (one per field of its schema)`. Even `SELECT *` over such a plan fails
at planning, because `EnsureRequirements` computes statistics. Padding
with unknown entries would hide the bug.

`count`, `min` and `max` return `None` on a missing entry instead of
panicking, as `sum` does, because the public `value_from_stats` cannot
return an error.

Not covered: the table and file group statistics a table provider passes
to a file scan, which can legitimately lack entries (virtual columns,
delta-rs) and need a different rule.

## What is the testing strategy for this PR?

An end-to-end test and unit tests for too few and too many entries, from
an operator and from a provider. The check is one length comparison per
plan, about 5 ns. Planning TPC-H takes 15.24 ms vs 15.17 ms on `main`,
TPC-DS 240.3 ms vs 240.1 ms (median of 3 runs).

## Are there any user-facing changes?

Yes, breaking for extensions: wrong-length statistics now fail planning.
The upgrade guide shows the fix. Please add the `api change` label.

---------

Co-authored-by: kosiew <kosiew@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

analysis.rs bounds check panic

2 participants