Skip to content

Check extension claims in each function - #11600

Draft
connortsui20 wants to merge 10 commits into
developfrom
ct/ext-claim-checks
Draft

connortsui20 wants to merge 10 commits into
developfrom
ct/ext-claim-checks

Conversation

@connortsui20

@connortsui20 connortsui20 commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Functions ran on the storage of any extension type, so json_a = json_b compared strings and the minimum of a WKB column was its smallest byte sequence. Zone maps, Bloom filters, and engine statistics were then built from those answers. Each function now requires the claim it needs when it binds, so a missing claim is a type error. It checks the claim again before it unwraps storage, because a path that skips binding has to fail instead of answering from storage.

Vector and FixedShapeTensor claim only Eq, because the lexicographic order of their storage isn't an order anyone would sort vectors by. Json, WKB, and the other types claim nothing, because one value can have several storage forms.

DuckDB builds table filters for simple comparisons without asking the pushdown_expression hook, so the scan declines each column whose dtype lacks Ord through supports_pushdown_type. Turning off table filter pushdown for the whole scan was the alternative, but that loses it for every other column too. Moving those filters above the scan exposed two latent bugs, in the multi-file reader's projection order and in join runtime filters, which are fixed here.

Files written before this change can hold statistics that are now rejected, such as the min and max of a Json column. A zoned layout that holds one turns off pruning and reads only its data child, and the footer skips them. A zone map that names an aggregate of the removed storage adapter, such as vortex.is_sorted[vortex.timestamp] or the same form for a third-party type, also turns off pruning instead of failing to open. For review, read the commits one at a time, since each one covers one family of functions.

Breaking change: comparisons, between, list_contains, the aggregates that order or compare values, and Bloom filters reject extension types that lack the claim they need, such as Json and WKB. ExtensionStoragePushdown and AggregateFnSession::register_storage_ordered are removed, because a type that claims Ord gets the order aggregates without a registration. The Scalar and Root variants of BoundExpression are private, so that no code can build a node that skips return_dtype.

Stacked on #11579.

AI assisted with this change.

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20 added a commit that referenced this pull request Nov 20, 2026
- Tracking Issue: #11115

Functions use the storage of any extension type today, so comparing two
`Json` values compares their strings. A type needs a way to promise that
its storage answers equality or order correctly before a function can
trust it. This PR adds only the declaration, and #11600 makes each
function check it.

`ExtVTable::claims` returns an opaque `Claims` value. A `&'static
[Claim]` slice was the alternative, but the slice could disagree with
`DType::holds` about `Eq`, it had several values for the same promise
(`[Ord]` and `[Eq, Ord]`), and it could only name properties. Private
fields leave room for a custom order key later.

AI assisted with this change.

---------

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Base automatically changed from ct/ext-claims to develop November 20, 2026 14:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant