Repository navigation
Drop direct arrow-select dependency - #10826
Conversation
Replace arrow_select::concat with a MutableArrayData-based helper, and replace the arrow_cast fallbacks for run-end and dictionary export with Vortex-native run detection and dict_encode. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FDwK8nfgMsXfEfYoKyoAoB
Rework the replacements for the arrow-cast and arrow-select calls: - Run-end export of a non-run-end array uses RunEnd::encode for primitives, re-wraps encoded storage for extension types, and for every other dtype finds run boundaries with one vectorized Eq comparison of adjacent rows combined with the validity bitmap. - Dictionary export of a non-dictionary array uses dict_encode for primitives and strings/binary, a hash encoder for decimals of every width, and re-wraps encoded storage for extension types, so it covers the value types arrow-cast supported. Nulls live on the keys. - Chunked lists canonicalize to a zero-copy ListView over chunked elements instead of being exported per chunk and concatenated. - Map scalar entries are appended with arrow-data's MutableArrayData, dropping the separate concat module. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FDwK8nfgMsXfEfYoKyoAoB
Keep arrow-cast in vortex-arrow and restore its run-end and dictionary export fallbacks. Only the arrow-select uses are replaced: map scalar entries are appended with arrow-data's MutableArrayData, and chunked lists canonicalize to a zero-copy ListView instead of being exported per chunk and concatenated. Signed-off-by: Robert Kruszewski <github@robertk.io> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FDwK8nfgMsXfEfYoKyoAoB
Merging this PR will degrade performance by 11.39%4 warnings, 1 note
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
This is still transitively picked up through arrow_cast and parquet variant but the logic is simpler without it