Repository navigation
Request: normalization and depythonize #80
Description
When upgrading pythonize from version 0.21.0 to version 0.23.0 in tantivy-py's quickwit-oss/tantivy-py#401, we switched from extract to extract_bound (per PyO3/pyo3#3916).
Now, we are hitting an issue where I64 values are becoming U64 after depythonize.
I came up with a solution in jamesbraza/tantivy-py#1 that involves re-casting U64 back to I64 if it fits, but this is pretty hacky.
Is there some way to preserve integer types across serialization/deserialization?
I64values remainingI64U64values remainingU64
I am wondering if there is some way pythonize can accommodate this, maybe by adding an opt-in flag somewhere?
Also, I am not 100% sure if this is a pythonize vs pyo3 issue, but it seems to be a deserialization issue, so I opened this here.
Reacted by Rene Leonhardt
Activity
davidhewitt commentedon Mar 26, 2025
Thanks for the report & sorry for the slow reply.
I suspect that #69 was the cause here; I see now that it was a behavioural change where positive values previously in
i64range will now be deserialized asu64values.I think it will be hard to guarantee round-tripping of
i64andu64values, because as soon as the data goes from Rust to Python, for positive integers we don't know whether they came from a signed or unsigned value.What we might be able to do is add a config option as to whether you prefer signed or unsigned types, and make
depythonizeselect according to your preference. Does that potentially help?jamesbraza commentedon May 4, 2025
Hi @davidhewitt thank you for the detailed response, #69 was indeed the
pythonizechange, sharing that really helped us.What the
tantivy-pyteam is doing is recursively forcing I64/F64 after deserialization, effectively eliminating the U64 vs I64 ambiguity.I think an option to prefer or force signed/unsigned is useful, it would enable
tantivy-pyto remove the new forcing logic. Up to you though, also feel free to close this out as resolved.Thanks again for the help here btw, much appreciated.