Skip to content

GH-15216: [C++][Parquet] Parquet writer accepts RecordBatch - #15240

Merged
wjones127 merged 3 commits into
apache:masterfrom
wgtmac:GH-15216
Jan 12, 2023
Merged

wjones127 merged 3 commits into
apache:masterfrom
wgtmac:GH-15216

Conversation

@wgtmac

@wgtmac wgtmac commented Jan 7, 2023 •

Copy link
Copy Markdown
Member
  • Parquet arrow::FileWriter supports buffered row group mode.
  • Parquet arrow::FileWriter accepts arrow::RecordBatch.

@github-actions

github-actions Bot commented Jan 7, 2023

Copy link
Copy Markdown

@github-actions

github-actions Bot commented Jan 7, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #15216 has been automatically assigned in GitHub to PR creator.

@github-actions

github-actions Bot commented Jan 7, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #15216 has no components, please add labels for components.

@wgtmac

wgtmac commented Jan 8, 2023

Copy link
Copy Markdown
Member Author

@emkornfield @pitrou @wjones127 Could you please take a look? Thanks!

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very useful! I have noted a few places where we could use a bit more tests and documentation to clarify some behavior.

Comment thread cpp/src/parquet/arrow/writer.cc Resolved
Comment thread cpp/src/parquet/arrow/writer.h Resolved
@wgtmac
wgtmac requested a review from wjones127 January 10, 2023 15:55

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking very good. I suggest one small change to a test so we use that leaf_count() call within the loop properly.

@wgtmac

wgtmac commented Jan 11, 2023

Copy link
Copy Markdown
Member Author

Looking very good. I suggest one small change to a test so we use that leaf_count() call within the loop properly.

All comments have been addressed. Please take a look again. Thanks!

@wgtmac
wgtmac requested a review from wjones127 January 11, 2023 05:44
@wjones127

Copy link
Copy Markdown
Member

Looks good. @wgtmac could you rebase? Have conflicts since I merged the other PR. Once that's complete, this should be good to merge.

 - Parquet arrow::FileWriter supports buffered row group mode.
 - Parquet arrow::FileWriter accepts arrow::RecordBatch.
@wgtmac

wgtmac commented Jan 12, 2023

Copy link
Copy Markdown
Member Author

Looks good. @wgtmac could you rebase? Have conflicts since I merged the other PR. Once that's complete, this should be good to merge.

@wjones127 Thanks for your review!

I have rebased and updated the PR. Please check again.

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent! Thank you @wgtmac

@wjones127
wjones127 merged commit 427a865 into apache:master Jan 12, 2023
@ursabot

ursabot commented Jan 13, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 3146588 and contender = 427a865. 427a865 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.42% ⬆️0.06%] test-mac-arm
[Finished ⬇️0.51% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.68% ⬆️0.12%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 427a865f ec2-t3-xlarge-us-east-2
[Failed] 427a865f test-mac-arm
[Finished] 427a865f ursa-i9-9960x
[Finished] 427a865f ursa-thinkcentre-m75q
[Finished] 3146588a ec2-t3-xlarge-us-east-2
[Finished] 3146588a test-mac-arm
[Finished] 3146588a ursa-i9-9960x
[Finished] 3146588a ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++][Parquet] Parquet arrow::FileWriter accepts arrow::RecordBatch in the buffered row group mode

3 participants