Repository navigation
Make successful Arrow plugin imports final - #11464
Conversation
|
actually I think there is a better way to do this |
05a01cb to
9b3be2e
Compare
9b3be2e to
30c43e2
Compare
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
30c43e2 to
01706a6
Compare
|
So the arrow import and arrow export logic is really quite complicated... I will try to write a consolidated readme for this crate that explains all the different cases and integrations and how we handle them. |
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
01706a6 to
81b8cbb
Compare
Summary
TurboQuant stores its Arrow metadata as base64 without an encoding marker. Vortex's fallback for registered Vortex IDs decodes base64 only when marked, so it reads TurboQuant's metadata as literal text and rejects valid arrays, even after a plugin has imported them successfully. An exporter that declines and delegates to storage conversion can hit the same check.
Changes
Successful plugin imports are final because the existing dtype and length checks already validate their result. Export fallback asks the registered importer to decode the field before comparing types, since the same metadata can use different byte encodings. The complete dtype check, apart from outer nullability, and the physical storage check remain. Without a matching importer, the field must contain the source type's serialized metadata.
Map entries import as a plain struct because the map's dtype comes from its key and value fields. Running a plugin on the entries field could produce a different dtype from schema inference. Key and value plugins still run.
Tests cover native metadata imports and export round trips, invalid metadata, and physical storage mismatches. A byte-collision case checks that equal metadata bytes cannot hide different type parameters. Map tests cover ignored entries plugins and length checks on value imports.
API Changes
Import plugins no longer run on a map's entries field. Extension types on the key and value fields still reach their plugins.