You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Partition values for timestamp_ns columns are not converted to nanoseconds #3652
While looking at the partitioning code I noticed that _to_partition_representation (pyiceberg/partitioning.py) doesn't handle
the v3 nano timestamp types.
For a normal TimestampType column, a datetime partition value gets converted to microseconds since epoch. For TimestampNanoType / TimestamptzNanoType there is no handler, so the value just passes through unchanged:
That datetime object then ends up in the DataFile.partition record, and the manifest writer expects an int there
(TimestampNanoWriter calls write_int), so writing to a table with an identity partition on a timestamp_ns column would fail.
Users can't hit this today because writing v3 is still blocked (#1551), but it will surface once that lands. It looks like an
oversight #1632 added nano support to transforms, readers/writers and conversions, but partitioning.py was never updated.
Expected: datetime values for nano timestamp columns should convert to nanoseconds since epoch, the same way the microsecond
types convert to micros.
Willingness to contribute
I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time
Activity
fix: Nanosecond partition representation (apache#3695)