Skip to content

fix: reject invalid parquet writer options when they are set - #26501

Closed
tripleaceme wants to merge 1 commit into
apache:mainfrom
tripleaceme:fix/parquet-writer-options-validation
Closed

tripleaceme wants to merge 1 commit into
apache:mainfrom
tripleaceme:fix/parquet-writer-options-validation

Conversation

@tripleaceme

Copy link
Copy Markdown

Which issue does this PR close?

#24905 fixed this before but was closed without review, so this is a new PR. It takes the reviewer feedback from #24905 into account: data_page_row_count_limit = 0 is rejected too, and the valid ranges are in the config.rs docstrings.

Rationale for this change

Some Parquet writer options accept values that the parquet crate cannot use. The SET succeeds, and the next COPY panics (or hangs, for write_batch_size = 0):

SET datafusion.execution.parquet.max_row_group_size = 0;
COPY (SELECT 1) TO 'x.parquet';
-- panicked: Cannot have a 0 max row group row count

This PR rejects these values when they are set, which is how batch_size (#23054) and max_row_group_bytes already work.

What changes are included in this PR?

  • write_batch_size, max_row_group_size, data_page_row_count_limit and content_defined_chunking.min_chunk_size are now ConfigNonZeroUsize.
  • column_index_truncate_length and statistics_truncate_length are now Option<ConfigNonZeroUsize>. They get their own ConfigField impl, so a rejected value does not leave the option half-set (same reason as the Option<MaxRowGroupBytes> impl).
  • New BloomFilterFpp type for the global and per-column bloom_filter_fpp. It accepts values strictly between 0 and 1, and rejects NaN.
  • max_chunk_size > min_chunk_size compares two options, so it cannot be checked at SET time. It is checked when the writer properties are built and returns a config error.
  • Protobuf decoding applies the same checks, so a serialized plan cannot bring back a value SET rejects. For the non-optional fields, a wire value of 0 is read as "not set" and replaced by the default. Proto3 cannot tell 0 from absent, and this keeps sparse messages decoding (same approach as the empty writer_version in fix: Avoid panic decoding invalid parquet writer version from proto #22467). An explicit 0 in an optional field (*_opt) is still an error.
  • From<protobuf::ParquetColumnOptions> for ParquetColumnOptions is now TryFrom, because an out-of-range bloom_filter_fpp has to be able to fail.

What is the testing strategy for this PR?

  • New parquet_writer_options_validation.slt: each zero value and each bad bloom_filter_fpp (0, 1, 1.5, -0.1, NaN, text) through SET, a rejected SET keeping the old value, the same checks through COPY ... OPTIONS (global and per-column), and the chunk-size check. On main this file panics in the writer.
  • Two proto-common unit tests: a sparse ParquetOptions decodes to the defaults, and an explicit zero truncate length is rejected.
  • Existing tests updated for the new types.

Are there any user-facing changes?

Yes. Invalid values now fail at SET (or COPY ... OPTIONS) with a config error instead of panicking on write. The config docs list the valid ranges.

This is an API change: the field types above changed, and the ParquetColumnOptions proto conversion is now TryFrom. Both are documented in the 56.0.0 upgrade guide. It needs the api change label.

@github-actions github-actions Bot added documentation Improvements or additions to documentation core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) common Related to common crate proto Related to proto crate datasource Changes to the datasource crate labels Oct 19, 2026
@alamb

alamb commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

This PR appears to have conflicts

@alamb
alamb marked this pull request as draft October 19, 2026 20:05
Parquet writer options that the parquet crate cannot accept (a zero
write_batch_size, max_row_group_size, data_page_row_count_limit,
truncate length or min_chunk_size, or a bloom_filter_fpp outside
(0, 1)) were accepted by SET and then panicked or hung on the next
write. They are now rejected with a configuration error when set.

max_chunk_size > min_chunk_size compares two options, so it is checked
when the writer properties are built.

Protobuf decoding applies the same checks. A wire value of 0 for a
non-optional field is read as absent and replaced by the default.

Closes apache#24904
@tripleaceme
tripleaceme force-pushed the fix/parquet-writer-options-validation branch from d9c952f to 7e19cdb Compare October 20, 2026 20:53
@alamb

alamb commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

@tripleaceme why did you close this PR?

@tripleaceme tripleaceme reopened this Oct 21, 2026
@tripleaceme
tripleaceme marked this pull request as ready for review October 21, 2026 21:31
@alamb

alamb commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.

https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful

This has the hallmarks of AI code that was not reviewed by the author: a 500-line revival of a closed PR across 13 files, closed and reopened without explanation, and no answer to the maintainer question about why.

If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.

@alamb alamb closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate datasource Changes to the datasource crate documentation Improvements or additions to documentation proto Related to proto crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Invalid parquet writer options (e.g. max_row_group_size = 0, bloom_filter_fpp = 1.5) panic in COPY instead of returning an error

2 participants