Repository navigation
Check extension values when arrays are built and imported - #11601
Draft
connortsui20 wants to merge 3 commits into
Draft
connortsui20 wants to merge 3 commits into
connortsui20 wants to merge 3 commits into
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
This was referenced Nov 18, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ExtensionArray::try_newonly checked the storage dtype, so an extension array could hold values that aren't legal values of its type, such as aTimelonger than one day. Data from Arrow, DuckDB, FFI, and Python also entered Vortex without a value check. The error showed up later, when a row was read as a scalar, or never on the array path. This PR gives each extension type a check over a whole storage array and runs it wherever arrays are built from data that no check has seen.Changes
ExtVTable::validate_arrayapplies the scalar check to each non-null value by default, so every type gets a correct check without new code.Time,Uuid, and WKB override it with faster checks, and the new conformance harness fails when an override and the default disagree on the same storage. It takes the typedExtDType<Self>and anExecutionCtx, because a stateful vtable such asForeignExtDTypeneeds its instance and lazy storage needs a context to read.Kernels that keep their input values, such as slice, take, filter, and the compressor, move to
new_unchecked, so only new data pays for the check. Its contract says that each visible value is legal or comes unchanged from an extension array of the same dtype. A stronger promise with a debug check was the alternative, but it's false in two common cases. A slice of file data keeps whatever the file holds, because file reads are structural by design. Builders also write storage zeros into a non-nullable child under a null parent, and the zero UUID isn't a legal v4 value.An import rebuilds each nested child without the validity of its ancestors, so a value check on the child would reject a placeholder under a null parent. Imports build with the new
try_new_structuraland then callvalidate_extension_valuesonce at their root, which carries the validity of ancestors down and checks only visible rows. Dictionary values and list view elements count as visible only when a visible row references them.The Arrow session keeps a weak reference to the
VortexSessionthat owns it, so the import check runs with the session's registered kernels without a reference cycle. Checking with an empty session was the alternative, but it would hide those kernels from avalidate_arraythat uses them. Import plugins get a view of the session that skips the check, because the root import checks their subtrees later.File reads stay structural, and the strict scan option
with_validate_extension_valuesis off by default, since a check costs a pass over the data and reads already return errors, not panics, for illegal values. For review, read the three commits one at a time: the check and harness, the constructor change, and the import and scan checks.API Changes
ExtensionArray::new,try_new,try_new_from_vtable,try_from_storage, andtry_from_storage_with_vtabletake a&mut ExecutionCtxand reject illegal values.ExtensionArray::new_uncheckedadds the value requirement above.ExtensionBuilder::append_valuerejects a scalar of another extension type, even when the storage matches.ExtVTable::validate_array,ExtensionArray::try_new_structural,validate_extension_values, the scan optionwith_validate_extension_values, andVortexSession::downgradewithWeakVortexSession.