Repository navigation
Conversation
…ercion Since apache#25193 the Parquet file-schema coercion recurses into nested types, so a nested binary field whose table type is a string was rewritten to that string type before the reader ran. The Parquet reader only validates UTF-8 for columns annotated UTF8, so invalid bytes produced an invalid string array: a panic in debug builds and unsound data in release builds, where DataFusion 55 returned a UTF-8 error from the cast. Apply the binary-to-string coercion to top-level fields only, as before apache#25193. Nested binary fields keep going through the validating cast, as map children already did. The view coercions still apply at every level.
Replace the reader-level unit test with a parquet_integration test that scans list and struct binary columns holding invalid UTF-8 through a string table schema and expects an error, and extend the leaf conversion test to list children.
|
Is it really worth handling the nested case when the non-nested case has been broken for months (years?)? This also disables functionality (things currently work okay for valid UTF-8, which I'd think is the majority of the use case of this non-default option). |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This is a regression since DF 55 and we had to ignore a test in Comet as a workaround. This is a valid backport candidate according to the release management guide. Not fixing it leaves a release build able to hand out a StringArray with invalid UTF-8, which is undefined behavior for any safe code that reads it as &str. On 55.2.0 the same query returned an error. I agree that it is not ideal that this issue already exists for top level columns. We can at least stop this unsound behavior from spreading farther?
No, I don't think it does? I added a test to confirm. |
|
OK, I see. I had misunderstood the impact of the change. It looks like it results in moving the transformation from decoding to a cast. and the cast validates. |
| )]) | ||
| }; | ||
| for (table_type, file_types) in [ | ||
| ( |
There was a problem hiding this comment.
can we please keep the old cases too?
Or does this PR change the behavior of the old cases too so it is easier to evaluate the changes?
Will anyone experience this as a regression 🤔
There was a problem hiding this comment.
My understanding is we are just moving where the conversion happens: because the decoded type no longer matches the table's type a cast will be inserted.
But maybe we can move that to SLT tests?
There was a problem hiding this comment.
That's correct, nested binary is now read as binary and cast to string, like before #25193. I don't expect any regression. I kept the original cases and added SLT tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Which issue does this PR close?
Rationale for this change
Reading a Parquet file with a nested BINARY field (a list element or a struct field) through a table schema that declares that field as a string produces a string array that was never checked for valid UTF-8. When the bytes are not valid UTF-8, the query panics in a debug build. In a release build it returns an invalid
StringArray, which is unsound for anything that reads the values as&str. On 55.2.0 the same query fails withEncountered non UTF-8 data.#25193 made the file-schema coercion recurse into nested types so that the view coercions (
Utf8toUtf8View,BinarytoBinaryView) reach nested fields (#25192). The binary-to-string coercion came along with it. That coercion hands the Parquet reader a string type, and the reader only validates UTF-8 for columns annotatedUTF8, which a BINARY column is not. #25342 already kept map keys and values out of it for this reason.#25193 is also on
branch-56, so this fix should be backported for 56.1.0.What changes are included in this PR?
coerce_data_typeapplies the binary-to-string coercion to top-level fields only, as before fix: make Parquet file schema type coercion work on nested schemas #25193. Struct fields and list-like children are coerced with it turned off, so nested binary fields stay binary in the file schema and are converted by the validating cast. The view coercions still apply at every level.coerce_childloses itsbinary_to_stringparameter, since every caller now passesfalse.The top-level case is unchanged: a top-level binary column read as a string still goes through the reader without validation, as it has for a long time. Fixing that needs the Parquet reader to validate whenever the target type is a string. That is apache/arrow-rs#11140, with a fix in review in apache/arrow-rs#11144. Skipping the coercion for top-level columns would lose statistics pruning for
binary_as_stringtables (#27068).PruningPredicatedoes not prune on nested fields, so this change loses no pruning.Once the reader validates, nested binary-to-string coercion is safe again, but lifting this restriction would change behavior for Comet. Comet replaces invalid UTF-8 in nested values with U+FFFD, which only works when they arrive as binary (apache/datafusion-comet#7469). So the restriction should stay until the reader can also replace invalid UTF-8 (apache/arrow-rs#12047).
What is the testing strategy for this PR?
nested_binary_read_as_string_rejects_invalid_utf8indatafusion/core/tests/parquet/schema_coercion.rsscans alist<binary>file and astruct<binary>file holding invalid UTF-8 through a table schema that declares those fields as strings, and expects anon UTF-8 dataerror. Without the fix it panics in the Parquet reader with the error from REGRESSION: Parquet scan returns unvalidated Utf8 for nested BINARY fields read as strings (regression from #25193) #27065.nested_coercion_preserves_leaf_conversion_rulesnow checks that nested view coercions still apply, and that nested binary-to-string coercions do not, for both struct and list children.Are there any user-facing changes?
A query that reads invalid UTF-8 from a nested binary field as a string returns an error instead of panicking or producing invalid data, which restores the 55.2.0 behavior. Valid data reads as before, through the cast instead of the reader. No public API changes.