Skip to content

Make successful Arrow plugin imports final - #11464

Merged
connortsui20 merged 1 commit into
developfrom
ct/arrow-plugin-metadata
Nov 13, 2026
Merged

connortsui20 merged 1 commit into
developfrom
ct/arrow-plugin-metadata

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

TurboQuant stores its Arrow metadata as base64 without an encoding marker. Vortex's fallback for registered Vortex IDs decodes base64 only when marked, so it reads TurboQuant's metadata as literal text and rejects valid arrays, even after a plugin has imported them successfully. An exporter that declines and delegates to storage conversion can hit the same check.

Changes

Successful plugin imports are final because the existing dtype and length checks already validate their result. Export fallback asks the registered importer to decode the field before comparing types, since the same metadata can use different byte encodings. The complete dtype check, apart from outer nullability, and the physical storage check remain. Without a matching importer, the field must contain the source type's serialized metadata.

Map entries import as a plain struct because the map's dtype comes from its key and value fields. Running a plugin on the entries field could produce a different dtype from schema inference. Key and value plugins still run.

Tests cover native metadata imports and export round trips, invalid metadata, and physical storage mismatches. A byte-collision case checks that equal metadata bytes cannot hide different type parameters. Map tests cover ignored entries plugins and length checks on value imports.

API Changes

Import plugins no longer run on a map's entries field. Extension types on the key and value fields still reach their plugins.

@connortsui20
connortsui20 marked this pull request as ready for review November 13, 2026 18:26
@connortsui20
connortsui20 marked this pull request as draft November 13, 2026 19:07
@connortsui20

Copy link
Copy Markdown
Member Author

actually I think there is a better way to do this

@connortsui20 connortsui20 changed the title Preserve native metadata after successful Arrow plugin imports Respect native metadata during Arrow import and export Nov 13, 2026
@connortsui20
connortsui20 force-pushed the ct/arrow-plugin-metadata branch from 05a01cb to 9b3be2e Compare November 13, 2026 19:31
@connortsui20 connortsui20 changed the title Respect native metadata during Arrow import and export Make successful Arrow plugin imports final Nov 13, 2026
@connortsui20
connortsui20 marked this pull request as ready for review November 13, 2026 19:31
@connortsui20
connortsui20 force-pushed the ct/arrow-plugin-metadata branch from 9b3be2e to 30c43e2 Compare November 13, 2026 19:35
@codspeed

codspeed Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 1936 untouched benchmarks
⏩ 50 skipped benchmarks1


Comparing ct/arrow-plugin-metadata (81b8cbb) with develop (f5d4d6d)

Open in CodSpeed

Footnotes

  1. 50 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@connortsui20
connortsui20 force-pushed the ct/arrow-plugin-metadata branch from 30c43e2 to 01706a6 Compare November 13, 2026 19:39
@connortsui20 connortsui20 added the changelog/fix A bug fix label Nov 13, 2026
@connortsui20

Copy link
Copy Markdown
Member Author

So the arrow import and arrow export logic is really quite complicated...

I will try to write a consolidated readme for this crate that explains all the different cases and integrations and how we handle them.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/arrow-plugin-metadata branch from 01706a6 to 81b8cbb Compare November 13, 2026 19:57
@connortsui20
connortsui20 merged commit d7aa24e into develop Nov 13, 2026
72 checks passed
@connortsui20
connortsui20 deleted the ct/arrow-plugin-metadata branch November 13, 2026 20:35
@connortsui20 connortsui20 mentioned this pull request Nov 16, 2026
19 of 57 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants