Repository navigation
fix: reject invalid parquet writer options when they are set - #26501
Closed
tripleaceme wants to merge 1 commit into
Closed
tripleaceme wants to merge 1 commit into
tripleaceme wants to merge 1 commit into
Conversation
Contributor
|
This PR appears to have conflicts |
alamb
marked this pull request as draft
October 19, 2026 20:05
Parquet writer options that the parquet crate cannot accept (a zero write_batch_size, max_row_group_size, data_page_row_count_limit, truncate length or min_chunk_size, or a bloom_filter_fpp outside (0, 1)) were accepted by SET and then panicked or hung on the next write. They are now rejected with a configuration error when set. max_chunk_size > min_chunk_size compares two options, so it is checked when the writer properties are built. Protobuf decoding applies the same checks. A wire value of 0 for a non-optional field is read as absent and replaced by the default. Closes apache#24904
tripleaceme
force-pushed
the
fix/parquet-writer-options-validation
branch
from
October 20, 2026 20:53
d9c952f to
7e19cdb
Compare
Contributor
|
@tripleaceme why did you close this PR? |
tripleaceme
marked this pull request as ready for review
October 21, 2026 21:31
Contributor
|
Thanks for the contribution! I am closing this under DataFusion's AI Dump policy. This has the hallmarks of AI code that was not reviewed by the author: a 500-line revival of a closed PR across 13 files, closed and reopened without explanation, and no answer to the maintainer question about why. If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
max_row_group_size = 0,bloom_filter_fpp = 1.5) panic inCOPYinstead of returning an error #24904.#24905 fixed this before but was closed without review, so this is a new PR. It takes the reviewer feedback from #24905 into account:
data_page_row_count_limit = 0is rejected too, and the valid ranges are in theconfig.rsdocstrings.Rationale for this change
Some Parquet writer options accept values that the
parquetcrate cannot use. TheSETsucceeds, and the nextCOPYpanics (or hangs, forwrite_batch_size = 0):This PR rejects these values when they are set, which is how
batch_size(#23054) andmax_row_group_bytesalready work.What changes are included in this PR?
write_batch_size,max_row_group_size,data_page_row_count_limitandcontent_defined_chunking.min_chunk_sizeare nowConfigNonZeroUsize.column_index_truncate_lengthandstatistics_truncate_lengthare nowOption<ConfigNonZeroUsize>. They get their ownConfigFieldimpl, so a rejected value does not leave the option half-set (same reason as theOption<MaxRowGroupBytes>impl).BloomFilterFpptype for the global and per-columnbloom_filter_fpp. It accepts values strictly between 0 and 1, and rejects NaN.max_chunk_size > min_chunk_sizecompares two options, so it cannot be checked atSETtime. It is checked when the writer properties are built and returns a config error.SETrejects. For the non-optional fields, a wire value of 0 is read as "not set" and replaced by the default. Proto3 cannot tell 0 from absent, and this keeps sparse messages decoding (same approach as the emptywriter_versionin fix: Avoid panic decoding invalid parquet writer version from proto #22467). An explicit 0 in an optional field (*_opt) is still an error.From<protobuf::ParquetColumnOptions> for ParquetColumnOptionsis nowTryFrom, because an out-of-rangebloom_filter_fpphas to be able to fail.What is the testing strategy for this PR?
parquet_writer_options_validation.slt: each zero value and each badbloom_filter_fpp(0, 1, 1.5, -0.1, NaN, text) throughSET, a rejectedSETkeeping the old value, the same checks throughCOPY ... OPTIONS(global and per-column), and the chunk-size check. Onmainthis file panics in the writer.ParquetOptionsdecodes to the defaults, and an explicit zero truncate length is rejected.Are there any user-facing changes?
Yes. Invalid values now fail at
SET(orCOPY ... OPTIONS) with a config error instead of panicking on write. The config docs list the valid ranges.This is an API change: the field types above changed, and the
ParquetColumnOptionsproto conversion is nowTryFrom. Both are documented in the 56.0.0 upgrade guide. It needs theapi changelabel.