Skip to content

fix: clear error instead of panic on mismatched column statistics - #25694

Merged
alamb merged 21 commits into
apache:mainfrom
jackylee-ch:fix-count-stats-guard
Nov 19, 2026
Merged

alamb merged 21 commits into
apache:mainfrom
jackylee-ch:fix-count-stats-guard

Conversation

@jackylee-ch

@jackylee-ch jackylee-ch commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

A custom ExecutionPlan or StatisticsProvider can report column_statistics with a different length than its schema, for example by starting from Statistics::default(). Rules and operators index it by column, so the mismatch surfaces far from its cause, as an index out of bounds, as in anvai-labs/proximaDB#76, SkardiLabs/skardi#276 and spiceai/spiceai#14144, or, with too many entries, as another column's values: min(b) returned the minimum of a.

What changes are included in this PR?

StatisticsContext checks that the statistics of every plan and provider have one entry per schema field, and otherwise returns an error naming the culprit, e.g. MyExec returned 1 column statistics, expected 2 (one per field of its schema). Even SELECT * over such a plan fails at planning, because EnsureRequirements computes statistics. Padding with unknown entries would hide the bug.

count, min and max return None on a missing entry instead of panicking, as sum does, because the public value_from_stats cannot return an error.

Not covered: the table and file group statistics a table provider passes to a file scan, which can legitimately lack entries (virtual columns, delta-rs) and need a different rule.

What is the testing strategy for this PR?

An end-to-end test and unit tests for too few and too many entries, from an operator and from a provider. The check is one length comparison per plan, about 5 ns. Planning TPC-H takes 15.24 ms vs 15.17 ms on main, TPC-DS 240.3 ms vs 240.1 ms (median of 3 runs).

Are there any user-facing changes?

Yes, breaking for extensions: wrong-length statistics now fail planning. The upgrade guide shows the fix. Please add the api change label.

count's statistics fast path indexed column_statistics[col.index()]
directly, so a TableProvider reporting fewer column statistics than its
schema made SELECT count(col) panic. Use .get(idx)? in both branches,
matching the guard already in sum.rs.
@github-actions github-actions Bot added the functions Changes to functions implementation label Sep 24, 2026
@codecov-commenter

codecov-commenter commented Sep 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.94%. Comparing base (3ee3e39) to head (9313de8).
⚠️ Report is 131 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main   #25694   +/-   ##
=======================================
  Coverage   82.93%   82.94%           
=======================================
  Files        1154     1154           
  Lines      458002   458075   +73     
  Branches   458002   458075   +73     
=======================================
+ Hits       379856   379930   +74     
+ Misses      55123    55122    -1     
  Partials    23023    23023           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jackylee-ch,

Thanks for fixing this. Both COUNT and COUNT(DISTINCT) now handle missing column statistics safely, and the new regression tests cover both paths. The change looks good to me. I have one optional suggestion for additional end-to-end coverage.

A statistics provider that drops the trailing column entry keeps the
aggregated column's index out of bounds whatever the projection, so
`count`'s statistics fast path is reached from SQL. The query panics
on main and returns the correct count with the guard in place.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR @jackylee-ch and for the review @kosiew

I feel like this is just papering over a bug and by sliently ignoring the problem we will make debugging harder

Changing the code so it doesn't panic seems fine, but if the table provider returns the wrong number of statistics I think that should return a clear error

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Agreed — it should be an error. Two candidates: the provider/source boundary, or StatisticsContext::compute_base, where provider and built-in statistics meet; that one also covers min_max and AggregateExec, which index it the same way.

One thing first: planning-time rules compute statistics (join_selection, limit_pushdown), so the broad gate also fails SELECT * ... LIMIT at planning time, not only count. With it the sqllogictests and core_integration pass; in physical-plan, two StatisticsProvider fixtures reporting no column statistics fail. Which do you prefer?

@alamb

alamb commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Which do you prefer?

I don't understand what you are asking me

What is the "broad gate"?

`Statistics::column_statistics` must hold one entry per schema field,
and consumers index it by column. Check that where `StatisticsContext`
receives statistics from an operator or a statistics provider, and fail
with an error naming the culprit instead of panicking in a consumer.
`value_from_stats` is public, so a direct caller can still pass short
statistics; fold the two `count` cases into one loop. Link
`ColumnStatistics` in the `StatisticsProvider` docs.
`min` and `max` index `column_statistics` like `count` did, and
`value_from_stats` is public, so use `.get(idx)?` there as well. The
end-to-end test now also checks that a plain scan fails planning.
Extra entries never indexed out of bounds; only missing ones did.
`FilterExec` indexes its input's column statistics when it is built, so
a table provider returning too few entries also panicked on `WHERE`.
A statistics provider returning the statistics of the whole table for a
scan of only `b` used to make `min(b)` and `max(b)` return the minimum
and maximum of `a`. Check end to end that planning fails instead, and
that both statistics sources reject one entry too many as well as too
few.

Also pin what file scans accept: with a projection, as all built-in
sources have, table and file group statistics of any length, missing
trailing entries being unknown and extra ones ignored; without one,
exactly one entry per table column. Say so in the upgrade guide.
`FileScanConfig` pads table and file group statistics it is given and
ignores extra entries when its source has a projection, so a missing or
extra entry before the last column shifts the statistics of the columns
after it. Describe that as unchecked rather than as accepted.
`FileScanConfig` looks up the table and file group statistics it is
given by column position, so an extra entry anywhere but at the end
shifted the statistics of every later column by one, and `min`/`max`
could be answered from another column's values. Return an internal
error naming the statistics instead. Missing trailing entries are still
unknown, which virtual columns and some table providers rely on.
@github-actions github-actions Bot added documentation Improvements or additions to documentation core Core DataFusion crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate labels Oct 16, 2026
@jackylee-ch jackylee-ch changed the title fix: guard against out-of-bounds column statistics in count's fast path fix: reject statistics whose column count does not match the schema Oct 16, 2026
@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Sorry, that was unclear. "Broad gate" meant one check in StatisticsContext, which sees what every operator and provider returns. This PR adds it: unless column_statistics has one entry per field, planning fails with an error naming the culprit.

Too few entries can make filters, joins, GROUP BY, count, min/max panic, too many can give min/max another column's values; such plans now fail, plain scans too. File scans also reject extras but still pad short statistics (virtual columns, delta-rs). count/min/max use .get(idx)?: value_from_stats cannot return an error.

@alamb

alamb commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

I am sorry -- I still don't get it.

How about we start with a clear and concise description of what problem you are solving (from the end user API perspective, not a code description)

Describe:

  1. What scenario are you hitting
  2. What happens today
  3. What your proposed behavior ?

It seems like you are trying to add better error messages when a table provider provides incorrect statistics ? But that is not clear

The error now reads, for example, "MyExec returned 1 column statistics,
expected 2 (one per field of its schema)", instead of starting with
"Assertion failed: column_statistics == fields (left: 1, right: 2)".

Also drop the FileScanConfig check for extra table and file group column
statistics. When its FileSource has a projection, FileScanConfig fills in
missing statistics as unknown on purpose, for virtual columns and some
table providers, so it needs a different check; only the statistics of
plans and statistics providers are checked now.

Note on Statistics::default() that it does not fit a schema with fields.
In the upgrade guide, use self.schema() in the example and mention
Statistics::project for plans that drop or reorder their input's columns.
@github-actions github-actions Bot removed the datasource Changes to the datasource crate label Oct 20, 2026
@github-actions github-actions Bot added the common Related to common crate label Oct 20, 2026
@jackylee-ch jackylee-ch changed the title fix: reject statistics whose column count does not match the schema fix: clear error instead of panic on mismatched column statistics Oct 20, 2026
@jackylee-ch

Copy link
Copy Markdown
Contributor Author

I've updated the description; does it answer your questions? Your guess is right: a clear planning error instead of a panic or a wrong answer. In short:

  1. Scenario: a table provider's scan reports statistics for the wrong number of columns. In fix(datafusion): schema-size ProximaScanExec stats (DF54 aggregate panic) anvai-labs/proximaDB#76, the scan fell back to Statistics::default(), which has no column entries; the description has two more cases.
  2. Today: rules and operators read column statistics by position, and nothing checks the count when a plan reports them, so a mismatch surfaces wherever they are first read. DataFusion 54's AggregateExec indexes its input's column statistics, so after upgrading, proximaDB's aggregate queries panicked there (index out of bounds: the len is 0 but the index is 0), with nothing pointing at the scan. With too many entries, min/max can return another column's values.
  3. Proposed: check the count whenever statistics are reported for a plan node, and fail planning with an error naming that node, e.g. MyExec returned 1 column statistics, expected 2 (one per field of its schema). Queries over such a scan then fail at planning, even SELECT *, which works today; the upgrade guide shows the fix.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jackylee-ch and @kosiew

This is getting closer, but still far too long in my opinion -- can you please read and review all lines and verify you think they are all necessary

Comment on lines +716 to +722
/// Returns an error unless `statistics`, computed for `plan` by `provider` or,
/// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field
/// of `plan`'s schema, as [`Statistics::column_statistics`] requires. Consumers
/// index it by column, so a mismatch could make them go out of bounds or read
/// the statistics of another column.
///
/// [`ColumnStatistics`]: datafusion_common::ColumnStatistics

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// Returns an error unless `statistics`, computed for `plan` by `provider` or,
/// if `None`, by the operator itself, hold one [`ColumnStatistics`] per field
/// of `plan`'s schema, as [`Statistics::column_statistics`] requires. Consumers
/// index it by column, so a mismatch could make them go out of bounds or read
/// the statistics of another column.
///
/// [`ColumnStatistics`]: datafusion_common::ColumnStatistics
/// Returns an error if the `statistics` mismatches the plan's schema
///
/// [`ColumnStatistics`]: datafusion_common::ColumnStatistics

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be much clearer

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cc8d175: the doc is now one line.

/// If `args.partition()` is `None`, it returns statistics for all partitions.
///
/// The result must hold one [`ColumnStatistics`] per field of
/// [`Self::schema`]; [`StatisticsContext::compute`] returns an error otherwise.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the detail of what returns an erorr is likely irrelevant

[StatisticsContext::compute] returns an error otherwise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cc8d175: dropped the sentence about the error.

statistics registry can change the result, because the providers are then
consulted.

### Statistics with the wrong number of column statistics are an error

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is way too detailed for a new error message. I don't think we need to document that the code detects some errors earlier

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in cc8d175: the entry is now 14 lines (was 41), without the "who is affected" list and the FileScanConfig note.

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

Trimmed in cc8d175: the diff against main is +188/−23 lines, down from +295/−24. I removed the doc paragraphs that restated the contract, the second end-to-end test (its case is covered by the unit tests) and most of the upgrade guide entry, and shortened the PR description. What remains is the check, the count/min/max guards and their tests.

@github-actions github-actions Bot removed the common Related to common crate label Nov 17, 2026

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jackylee-ch, thanks for addressing the malformed column statistics issue. The approach is sound, and I found no blocking runtime defects. Valid statistics retain their existing behavior, while mismatched statistics now produce actionable errors instead of downstream panics or incorrect aggregate results.

StatisticsContext is the right enforcement boundary because it handles both ExecutionPlan and StatisticsProvider results before caching or parent consumption. The defensive COUNT, MIN, and MAX checks also protect direct callers that bypass this context. This avoids relying on individual producers to validate themselves or changing public APIs.

I'm approving with suggestions. Please move the new upgrade note to the 57.0.0 guide before merging, since the reviewed head targets 57.0.0-dev. The additional boundary, partition, and provider-documentation suggestions are non-blocking.

operator, as it does above `CoalescePartitionsExec`, and is copied below as
an early-stop fetch.

### Statistics must have one `ColumnStatistics` per schema field

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move the new statistics-contract section to docs/source/library-user-guide/upgrading/57.0.0.md, since the reviewed head declares 57.0.0-dev in Cargo.toml. Keeping it here incorrectly attributes the new planning-time rejection to 56.0.0. Move only the newly added section and leave the existing guidance intact.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, main is 57.0.0-dev; moved to 57.0.0.md in 6e23b65.

{
let (statistics, extensions) = computed.into_parts();
validate_column_statistics(plan, &statistics, Some(provider.as_ref()))?;
let extensions = if extensions.is_empty() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional, low priority: could we test a partition-aware provider that returns valid overall statistics but malformed statistics for Some(0)? Call compute_arc for both requests using the same context and plan, then assert overall success and a provider/count error for partition 0. This would verify that cached overall statistics do not mask invalid partition results. The provider must handle partition requests explicitly rather than delegate them.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not adding: the cache key includes the partition and validation runs before the store, so a cached overall result cannot mask a bad Some(0).

@jackylee-ch

Copy link
Copy Markdown
Contributor Author

@alamb, cc8d1755d trims the diff to +188/−23 and answers your three inline comments; 6e23b65f8 only moves the upgrade note to 57.0.0.md. Could you take another look?

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me - -thank you @jackylee-ch and @kosiew

@alamb
alamb added this pull request to the merge queue Nov 19, 2026
Merged via the queue into apache:main with commit 657d2ca Nov 19, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate documentation Improvements or additions to documentation functions Changes to functions implementation physical-plan Changes to the physical-plan crate v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants