Skip to content

fix: keep nested Parquet binary fields out of the binary-to-string coercion - #27067

Open
andygrove wants to merge 6 commits into
apache:mainfrom
andygrove:fix/27065-nested-binary-utf8
Open

andygrove wants to merge 6 commits into
apache:mainfrom
andygrove:fix/27065-nested-binary-utf8

Conversation

@andygrove

@andygrove andygrove commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

Reading a Parquet file with a nested BINARY field (a list element or a struct field) through a table schema that declares that field as a string produces a string array that was never checked for valid UTF-8. When the bytes are not valid UTF-8, the query panics in a debug build. In a release build it returns an invalid StringArray, which is unsound for anything that reads the values as &str. On 55.2.0 the same query fails with Encountered non UTF-8 data.

#25193 made the file-schema coercion recurse into nested types so that the view coercions (Utf8 to Utf8View, Binary to BinaryView) reach nested fields (#25192). The binary-to-string coercion came along with it. That coercion hands the Parquet reader a string type, and the reader only validates UTF-8 for columns annotated UTF8, which a BINARY column is not. #25342 already kept map keys and values out of it for this reason.

#25193 is also on branch-56, so this fix should be backported for 56.1.0.

What changes are included in this PR?

  • coerce_data_type applies the binary-to-string coercion to top-level fields only, as before fix: make Parquet file schema type coercion work on nested schemas #25193. Struct fields and list-like children are coerced with it turned off, so nested binary fields stay binary in the file schema and are converted by the validating cast. The view coercions still apply at every level.
  • coerce_child loses its binary_to_string parameter, since every caller now passes false.
  • The module docs describe the rule for nested fields.

The top-level case is unchanged: a top-level binary column read as a string still goes through the reader without validation, as it has for a long time. Fixing that needs the Parquet reader to validate whenever the target type is a string. That is apache/arrow-rs#11140, with a fix in review in apache/arrow-rs#11144. Skipping the coercion for top-level columns would lose statistics pruning for binary_as_string tables (#27068). PruningPredicate does not prune on nested fields, so this change loses no pruning.

Once the reader validates, nested binary-to-string coercion is safe again, but lifting this restriction would change behavior for Comet. Comet replaces invalid UTF-8 in nested values with U+FFFD, which only works when they arrive as binary (apache/datafusion-comet#7469). So the restriction should stay until the reader can also replace invalid UTF-8 (apache/arrow-rs#12047).

What is the testing strategy for this PR?

  • nested_binary_read_as_string_rejects_invalid_utf8 in datafusion/core/tests/parquet/schema_coercion.rs scans a list<binary> file and a struct<binary> file holding invalid UTF-8 through a table schema that declares those fields as strings, and expects a non UTF-8 data error. Without the fix it panics in the Parquet reader with the error from REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) #27065.
  • nested_coercion_preserves_leaf_conversion_rules now checks that nested view coercions still apply, and that nested binary-to-string coercions do not, for both struct and list children.

Are there any user-facing changes?

A query that reads invalid UTF-8 from a nested binary field as a string returns an error instead of panicking or producing invalid data, which restores the 55.2.0 behavior. Valid data reads as before, through the cast instead of the reader. No public API changes.

…ercion

Since apache#25193 the Parquet file-schema coercion recurses into nested types, so
a nested binary field whose table type is a string was rewritten to that
string type before the reader ran. The Parquet reader only validates UTF-8
for columns annotated UTF8, so invalid bytes produced an invalid string
array: a panic in debug builds and unsound data in release builds, where
DataFusion 55 returned a UTF-8 error from the cast.

Apply the binary-to-string coercion to top-level fields only, as before
apache#25193. Nested binary fields keep going through the validating cast, as map
children already did. The view coercions still apply at every level.
Replace the reader-level unit test with a parquet_integration test that
scans list and struct binary columns holding invalid UTF-8 through a string
table schema and expects an error, and extend the leaf conversion test to
list children.
@adriangb

adriangb commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Is it really worth handling the nested case when the non-nested case has been broken for months (years?)? This also disables functionality (things currently work okay for valid UTF-8, which I'd think is the majority of the use case of this non-default option).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@andygrove

Copy link
Copy Markdown
Member Author

Is it really worth handling the nested case when the non-nested case has been broken for months (years?)?

This is a regression since DF 55 and we had to ignore a test in Comet as a workaround. This is a valid backport candidate according to the release management guide.

Not fixing it leaves a release build able to hand out a StringArray with invalid UTF-8, which is undefined behavior for any safe code that reads it as &str. On 55.2.0 the same query returned an error.

I agree that it is not ideal that this issue already exists for top level columns. We can at least stop this unsound behavior from spreading farther?

This also disables functionality (things currently work okay for valid UTF-8, which I'd think is the majority of the use case of this non-default option).

No, I don't think it does? I added a test to confirm.

@adriangb

Copy link
Copy Markdown
Contributor

OK, I see. I had misunderstood the impact of the change. It looks like it results in moving the transformation from decoding to a cast. and the cast validates.

)])
};
for (table_type, file_types) in [
(

@alamb alamb Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we please keep the old cases too?

Or does this PR change the behavior of the old cases too so it is easier to evaluate the changes?

Will anyone experience this as a regression 🤔

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding is we are just moving where the conversion happens: because the decoded type no longer matches the table's type a cast will be inserted.

But maybe we can move that to SLT tests?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's correct, nested binary is now read as binary and cast to string, like before #25193. I don't expect any regression. I kept the original cases and added SLT tests.

andygrove and others added 2 commits November 19, 2026 15:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) and removed core Core DataFusion crate labels Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

datasource Changes to the datasource crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193)

3 participants