Skip to content

GH-43382: [C++][Parquet] min-max Statistics doesn't work well when one of min-max is truncated - #43383

Merged
mapleFU merged 3 commits into
apache:mainfrom
mapleFU:fix-min-max-stats
Aug 5, 2024
Merged

mapleFU merged 3 commits into
apache:mainfrom
mapleFU:fix-min-max-stats

Conversation

@mapleFU

@mapleFU mapleFU commented Jul 23, 2024 •

Copy link
Copy Markdown
Member

Rationale for this change

See #43382

What changes are included in this PR?

Change stats has min-max from min || max to &&

Are these changes tested?

  • TODO

Are there any user-facing changes?

Might affect interface using HasMinMax

This PR includes breaking changes to public APIs.

@mapleFU
mapleFU requested a review from wgtmac as a code owner July 23, 2024 05:22
@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #43382 has been automatically assigned in GitHub to PR creator.

@mapleFU

mapleFU commented Jul 23, 2024

Copy link
Copy Markdown
Member Author

This is just a try, I can also extract has-min and has-max

@mapleFU

mapleFU commented Jul 23, 2024

Copy link
Copy Markdown
Member Author

@pitrou @wgtmac would you mind take a look whether this fixing is ok or there is better api? I'll draft testing once api is stable

@wgtmac

wgtmac commented Jul 23, 2024

Copy link
Copy Markdown
Member

I'm fine to be conservative. So this change looks good. However, the encoded_min and encoded_max are still set. I'd propose to deprecate virtual bool HasMinMax() const = 0; and add virtual bool HasMin() const = 0; and virtual bool HasMax() const = 0;

@mapleFU

mapleFU commented Aug 3, 2024

Copy link
Copy Markdown
Member Author

Updated: I'll separate the implemetation into two separate patch:

  • The first one is test and remove the || in HasMinMax, changing the syntax to both has-min and has-max
  • The second one is add HasMin and HasMax.

Anyway this is a bug rather than feature so we should fix it first

@mapleFU

mapleFU commented Aug 5, 2024

Copy link
Copy Markdown
Member Author

@kou @emkornfield @wgtmac would you mind take a look? This patch only change || -> &&

@kou kou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Could you uncomment <!-- **This PR includes breaking changes to public APIs.** --> in the PR description?

@github-actions github-actions Bot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Aug 5, 2024
@wgtmac wgtmac changed the title GH-43382: [C++][Parquet] min-max Statistics doesn't works well when one of min-max beging truncated GH-43382: [C++][Parquet] min-max Statistics doesn't work well when one of min-max is truncated Aug 5, 2024

@wgtmac wgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@mapleFU
mapleFU merged commit 66cb749 into apache:main Aug 5, 2024
@mapleFU mapleFU removed the awaiting merge Awaiting merge label Aug 5, 2024
@mapleFU
mapleFU deleted the fix-min-max-stats branch August 5, 2024 08:29
@conbench-apache-arrow

Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 66cb749.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants