Skip to content

ARROW-2585: [C++] Add Decimal::FromBigEndian, which was formerly a static method in parquet-cpp/src/parquet/arrow/reader.cc - #2036

Closed
joshuastorck wants to merge 2 commits into
apache:masterfrom
joshuastorck:decimal_from_big_endian
Closed

joshuastorck wants to merge 2 commits into
apache:masterfrom
joshuastorck:decimal_from_big_endian

Conversation

@joshuastorck

Copy link
Copy Markdown
Contributor

No description provided.

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From an API point of view, there's a Decimal128 constructor from little-endian bytes that only takes a uint8_t* (without a length), and a static method factory from big-endian bytes that takes a pointer and a length. Shouldn't we have something more consistent?

Comment thread cpp/src/arrow/util/decimal.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You shouldn't need std:: in front of int32_t.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

Comment thread cpp/src/arrow/util/decimal.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is potentially doing an unaligned access, which is forbidden on some architectures (SPARC for example, perhaps others).

@cpcloud cpcloud May 14, 2018 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like for linux there's get_unaligned/put_unaligned. For MSVC, we could create a template function that creates a temporary struct and accesses a member.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need such complication. Just read the memory byte per byte. Or, if you really insist on doing a single int64_t read, use mempcy into an int64_t local variable. Most compilers will optimize that into an unaligned access if the platform allows for it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in latest version. Using memcpy at particular byte offset.

Comment thread cpp/src/arrow/util/decimal-test.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally you should also test non-trivial values (not one which only has only one bit set), and also negative values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing negative values and complements in the latest version.

@codecov-io

codecov-io commented May 15, 2018 •

Copy link
Copy Markdown

Codecov Report

Merging #2036 into master will increase coverage by 0.04%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #2036      +/-   ##
==========================================
+ Coverage   86.28%   86.33%   +0.04%     
==========================================
  Files         242      242              
  Lines       41041    41091      +50     
==========================================
+ Hits        35412    35474      +62     
+ Misses       5629     5617      -12
Impacted Files Coverage Δ
cpp/src/arrow/util/decimal.h 100% <ø> (ø) ⬆️
cpp/src/arrow/util/decimal.cc 87.97% <100%> (+3.53%) ⬆️
cpp/src/arrow/util/decimal-test.cc 100% <100%> (ø) ⬆️
cpp/src/arrow/util/thread-pool-test.cc 98.87% <0%> (-0.57%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3d41978...e970c87. Read the comment docs.

@pitrou

pitrou commented May 15, 2018

Copy link
Copy Markdown
Member

@joshuastorck, thanks for the test additions. What do you think about the API inconsistency? It would deserve fixing before doing it has a compatibility cost, IMO.

@joshuastorck

Copy link
Copy Markdown
Contributor Author

Since we are using Status codes (I wish we were using exceptions) for errors and this function can return an error, I don't want the logic in the constructor because the only way to handle that error is to do what's being done in the constructor that takes a string, which is to log and abort.

If we were using exceptions, I would advocate for the same approach, primarily because this is a primitive numeric type and users don't want to think about exceptions when they are using primitive numeric types.

@pitrou

pitrou commented May 15, 2018

Copy link
Copy Markdown
Member

That makes sense. I think we can go with this, then.

@pitrou

pitrou commented May 15, 2018

Copy link
Copy Markdown
Member

@joshuastorck, sorry, I forgot to mention: you should prefix this PR's title with the corresponding JIRA ticket number and category, like this: ARROW-1234: [C++] Adding Decimal::FromBigEndian etc..

@joshuastorck joshuastorck changed the title Adding Decimal::FromBigEndian, which was formerly a static method in parquet-cpp/src/parquet/arrow/reader.cc ARROW-2585: Adding Decimal::FromBigEndian, which was formerly a static method in parquet-cpp/src/parquet/arrow/reader.cc May 15, 2018
Comment thread cpp/src/arrow/util/decimal.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way: since this helper is only used by FromBigEndian, it needn't be in the .h file. You can just make it a static non-member function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Joshua Storck added 2 commits May 17, 2018 11:25
@joshuastorck
joshuastorck force-pushed the decimal_from_big_endian branch from 2001769 to e970c87 Compare May 18, 2018 01:18
@joshuastorck

Copy link
Copy Markdown
Contributor Author

@pitrou, I think I've resolved all of your concerns. Could you please merge?

@pitrou

pitrou commented May 18, 2018

Copy link
Copy Markdown
Member

@joshuastorck I'll probably do it in a couple days. If there is any hurry, please tell me.

@joshuastorck

joshuastorck commented May 18, 2018 via email

Copy link
Copy Markdown
Contributor Author

@pitrou pitrou changed the title ARROW-2585: Adding Decimal::FromBigEndian, which was formerly a static method in parquet-cpp/src/parquet/arrow/reader.cc ARROW-2585: [C++] Add Decimal::FromBigEndian, which was formerly a static method in parquet-cpp/src/parquet/arrow/reader.cc May 22, 2018
@pitrou pitrou closed this in fcc13f5 May 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants