Repository navigation
Add JSON single-value serialization for timestamp_ns and timestamptz_ns - #3855
takayoshi-makabe wants to merge 2 commits into
Conversation
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
2c07ba3 to
8d4658f
Compare
|
@ebyhr @kevinjqliu Friendly ping — this is a small fix and CI is green. Would you mind taking a look when you have time? Happy to address any feedback. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Rationale for this change
TimestampNanoTypeandTimestamptzNanoTypeare registered for byte conversion but not forto_json/from_json, so the JSON single-value serialization path raises for them. A nanosecond timestamp field with a default cannot even be constructed:TypeError: Cannot deserialize bytes, type timestamp_ns not supported: 1510871468123456789NestedFieldrunsfrom_jsononinitial-default/write-defaultduring validation, so this fails on construction as well as onmodel_dump_json()andmodel_validate_json().Implementation notes
Per Appendix D both types serialize as ISO-8601 strings with nanosecond precision, matching Java's
SingleValueParser:from_jsonreturns anint(epoch nanoseconds) rather than adatetime, because Python'sdatetimeonly holds microseconds and would silently drop the last three digits.from_bytesalready returns anintfor these two types.to_human_timestamp_ns/to_human_timestamptz_nsare new — no existing helper formats at nanosecond precision.Two things left out of scope: the
TypeErrorabove says "Cannot deserialize bytes" even fromto_json, a pre-existing copy of from_bytes's message that #3856 fixes separately. And this is independent of #3853, which refactors the*_to_nanosparsers while this adds the reverse direction plus the JSON dispatch — they touch different functions.Are these changes tested?
Yes.
tests/test_conversions.py: both types added totest_json_single_serialization(int anddatetimeinputs) andtest_json_serialize_roundtrip.tests/utils/test_datetime.py: the new formatters, covering the epoch, zero-padded sub-second digits, and a pre-epoch value — each asserted to round-trip back throughtimestamp_to_nanos/timestamptz_to_nanos.tests/test_types.py: end-to-endNestedFieldround-trip forinitial-default/write-default, which is the path in the report above.Are there any user-facing changes?
Yes. Nanosecond timestamp fields with an
initial-defaultorwrite-defaultnow serialize and deserialize instead of raisingTypeError.