Skip to content

Check extension values when arrays are built and imported - #11601

Draft
connortsui20 wants to merge 3 commits into
developfrom
ct/ext-value-checks
Draft

connortsui20 wants to merge 3 commits into
developfrom
ct/ext-value-checks

Conversation

@connortsui20

Copy link
Copy Markdown
Member

Summary

ExtensionArray::try_new only checked the storage dtype, so an extension array could hold values that aren't legal values of its type, such as a Time longer than one day. Data from Arrow, DuckDB, FFI, and Python also entered Vortex without a value check. The error showed up later, when a row was read as a scalar, or never on the array path. This PR gives each extension type a check over a whole storage array and runs it wherever arrays are built from data that no check has seen.

Changes

ExtVTable::validate_array applies the scalar check to each non-null value by default, so every type gets a correct check without new code. Time, Uuid, and WKB override it with faster checks, and the new conformance harness fails when an override and the default disagree on the same storage. It takes the typed ExtDType<Self> and an ExecutionCtx, because a stateful vtable such as ForeignExtDType needs its instance and lazy storage needs a context to read.

Kernels that keep their input values, such as slice, take, filter, and the compressor, move to new_unchecked, so only new data pays for the check. Its contract says that each visible value is legal or comes unchanged from an extension array of the same dtype. A stronger promise with a debug check was the alternative, but it's false in two common cases. A slice of file data keeps whatever the file holds, because file reads are structural by design. Builders also write storage zeros into a non-nullable child under a null parent, and the zero UUID isn't a legal v4 value.

An import rebuilds each nested child without the validity of its ancestors, so a value check on the child would reject a placeholder under a null parent. Imports build with the new try_new_structural and then call validate_extension_values once at their root, which carries the validity of ancestors down and checks only visible rows. Dictionary values and list view elements count as visible only when a visible row references them.

The Arrow session keeps a weak reference to the VortexSession that owns it, so the import check runs with the session's registered kernels without a reference cycle. Checking with an empty session was the alternative, but it would hide those kernels from a validate_array that uses them. Import plugins get a view of the session that skips the check, because the root import checks their subtrees later.

File reads stay structural, and the strict scan option with_validate_extension_values is off by default, since a check costs a pass over the data and reads already return errors, not panics, for illegal values. For review, read the three commits one at a time: the check and harness, the constructor change, and the import and scan checks.

API Changes

  • ExtensionArray::new, try_new, try_new_from_vtable, try_from_storage, and try_from_storage_with_vtable take a &mut ExecutionCtx and reject illegal values.
  • The contract of ExtensionArray::new_unchecked adds the value requirement above.
  • ExtensionBuilder::append_value rejects a scalar of another extension type, even when the storage matches.
  • Imports from Arrow, DuckDB, FFI, and Python reject visible extension values outside the domain of their type.
  • New ExtVTable::validate_array, ExtensionArray::try_new_structural, validate_extension_values, the scan option with_validate_extension_values, and VortexSession::downgrade with WeakVortexSession.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant