Repository navigation
fix: preserve field metadata through CASE - #27143
Conversation
The logical Expr::Case field and the physical CaseExpr field now merge the metadata of the value branches with coerced_fields_metadata, the same rules COALESCE and NVL2 use: the earlier branch wins for each key, an extension name stays with its parameters, and differing extension names are a planning error. CaseExpr::try_new keeps a typed NULL ELSE that carries metadata, and a NULL-literal ELSE still evaluates like no ELSE.
|
@paleolimbot @alamb @osipovartem This is my alternative solution for preserving metadata in those cases. |
paleolimbot
left a comment
There was a problem hiding this comment.
This approach makes sense to me but I'll defer to those more familiar with the case/when optimizations as to whether the other approach is preferable!
Codex complained to me that this approach requires that logical and physical representations of an expr have to match for this to work and pointed out a few places they don't at the moment which I filed as separate issues. Its noted test gap was "No test compares Expr::Case::to_field with CaseExpr::return_field for the same expression."
One more gap it suggested was to check where the CASE test where branches have different storage types and coercion inserts a cast. This matters because type-only casts strip extension metadata, and logical and physical cast behavior must agree.
Compare Expr::Case::to_field with the physical return_field, including a branch cast to the common type by type coercion. Use the built-in JSON extension type in the CASE metadata.slt cases, add a coercion case, and move the CASE WHEN false case next to the other CASE cases.
|
Thanks Dewey. Integrated all your feedback. Logical vs physical. Agreed, this relies on each branch's physical field matching its logical one, which is the property #27156 describes for every expression. CASE derives its field only from its branches in both layers, so it matches whenever its branches do, and a branch that doesn't (NOT in #27155, Alias in #27156) shows through. #26158 avoids that by carrying the logical field into the physical CASE, at the cost of storing it and a protobuf change. In practice the mismatch doesn't change the result schema, which takes its metadata from the logical plan. Only a function that takes the CASE as an argument sees the physical field. in the NOT case probably this doesnt really matter - its probably just extra metadata. In the Alias case it can be an issue but this is probably exceeding rate to call alias on a subexpression that doesnt persist past the operator.
Coercion casts. Logical and physical agree: the type-only cast that coercion inserts drops the extension type and keeps other keys on both sides. The unit test above covers it, and a new |
jayzhan211
left a comment
There was a problem hiding this comment.
@james-willis thanks for this! In expr_simplifier.rs around line 1485, the other rewrites in this arm still drop the merged metadata: ext_only(CASE WHEN 1 = 0 THEN g ELSE NULL END) (the UDF from the #24860 repro) fails planning with "argument lost its extension type", while CASE WHEN 1 = 0 THEN g END works. SELECT CASE WHEN 1 = 0 THEN g ELSE plain END AS x also reports the extension in df.schema() but not in the batches. Could the arm compare the field metadata before and after and keep the CASE when it changes? I ran this diff: every ext_only probe in the table below passes and the optimizer lib tests still pass. It does make the arm read the ELSE's field, so from reading the code (I didn't run this one) a simplifier without a schema would now error on CASE WHEN true THEN 1 ELSE col END.
- // CASE WHEN false THEN A END --> NULL
- // The NULL keeps the CASE's field metadata, such as an extension type
- if else_expr.is_none()
- && when_then_expr
- .iter()
- .all(|(when, _)| is_false(when.as_ref()))
- {
- let (_, field) = Expr::Case(Case::new(None, when_then_expr, None))
- .to_field(info.schema().as_ref())?;
- let metadata = FieldMetadata::new_from_field(&field);
- let null = Expr::Literal(
- ScalarValue::try_new_null(field.data_type())?,
- (!metadata.is_empty()).then_some(metadata),
- );
- return Ok(Transformed::yes(null));
- }
+ // Dropping a branch must not change the field of the CASE, whose
+ // metadata merges all of its value branches.
+ let original = Expr::Case(Case::new(
+ None,
+ when_then_expr.clone(),
+ else_expr.clone(),
+ ));
+ let schema = info.schema().as_ref();
+ let (_, field) = original.to_field(schema)?;
+ let metadata = FieldMetadata::new_from_field(&field);
@@
// Exclude CASE statement altogether if there are no when-then expressions left
- // CASE WHEN false THEN A ELSE B END --> B
- if new_when_then_expr.is_empty()
- && let Some(else_expr) = else_expr
- {
- return Ok(Transformed::yes(*else_expr));
- }
+ let simplified = match (new_when_then_expr.is_empty(), else_expr) {
+ // CASE WHEN false THEN A END --> NULL
+ (true, None) => {
+ return Ok(Transformed::yes(Expr::Literal(
+ ScalarValue::try_new_null(field.data_type())?,
+ (!metadata.is_empty()).then_some(metadata),
+ )));
+ }
+ // CASE WHEN false THEN A ELSE B END --> B
+ (true, Some(else_expr)) => *else_expr,
+ (false, else_expr) => Expr::Case(Case {
+ expr: None,
+ when_then_expr: new_when_then_expr,
+ else_expr,
+ }),
+ };
- Transformed::yes(Expr::Case(Case {
- expr: None,
- when_then_expr: new_when_then_expr,
- else_expr,
- }))
+ let (_, new_field) = simplified.to_field(schema)?;
+ if FieldMetadata::new_from_field(&new_field) == metadata {
+ Transformed::yes(simplified)
+ } else {
+ Transformed::no(original)
+ }Queries to add to the tests, and the results with this diff
Each arrow_metadata query should return arrow.json, the same as the non-constant CASE (expected results; I ran the same shapes through ext_only below, not these exact queries):
select arrow_metadata(case when 1 = 0 then cast(name as json) else NULL end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when 1 = 0 then cast(name as json) else upper(l_name) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when 1 = 1 then upper(l_name) else cast(name as json) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when id > 1 then upper(l_name) when 1 = 0 then cast(name as json) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(coalesce(nonnull_name, cast(name as json)), 'ARROW:extension:name') from table_with_metadata;With ext_only requiring the extension on its argument (g has one; plain and nn don't, and nn is non-nullable):
| Argument | Now | With the diff |
|---|---|---|
CASE WHEN 1 = 0 THEN g ELSE NULL END |
plan error | ok |
CASE WHEN 1 = 0 THEN g ELSE plain END |
plan error | ok |
CASE WHEN 1 = 1 THEN plain ELSE g END |
plan error | ok |
CASE WHEN k = 1 THEN plain WHEN 1 = 0 THEN g END |
plan error | ok |
COALESCE(nn, g) |
plan error | ok |
NVL2(nn, plain, g) |
plan error | ok |
In expr_simplifier.rs around line 1555, the boolean lowering loses the merged metadata too: for CASE WHEN p THEN f END, with f a Boolean column carrying metadata, df.schema() reports it but the batches don't, and a metadata-checking UDF fails to plan. Could this arm skip the rewrite when a value branch has metadata, with a test on that query? Line 1613 looks like the same rewrite for literal THENs, so it likely needs the same guard; that's from reading the code, not a probe, and I haven't compiled this snippet.
/// Whether a value branch of a CASE has field metadata, which AND/OR cannot keep
fn case_branch_has_metadata(
when_then_expr: &[(Box<Expr>, Box<Expr>)],
else_expr: Option<&Expr>,
info: &SimplifyContext,
) -> Result<bool> {
let branches = when_then_expr
.iter()
.map(|(_, then)| then.as_ref())
.chain(else_expr);
for expr in branches {
let (_, field) = expr.to_field(info.schema().as_ref())?;
if !field.metadata().is_empty() {
return Ok(true);
}
}
Ok(false)
}- && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref()) =>
+ && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref())
+ && !case_branch_has_metadata(&when_then_expr, else_expr.as_deref(), info)? =>Optional cleanup: the "Metadata Handling" list on to_field in expr_schema.rs around line 509 could get a CASE entry now that CASE has its own rule.
/// - **Binary expressions**: field metadata is empty
+ /// - **CASE expressions**: Merges the metadata of the THEN and ELSE branches; the operand and WHEN conditions do not contribute| } | ||
| } | ||
|
|
||
| fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> { |
There was a problem hiding this comment.
The NOT mismatch from #27155 can be a planning error here, not only extra metadata: this query returns rows on main and now fails with "Cannot merge field metadata with differing values". It's fine to handle this in #27155 since the fix belongs in NOT; could you add this query there and adjust the note in the PR description?
-- t(id INT, b BOOLEAN)
select case when id > 1
then not with_metadata(b, 'ARROW:extension:name', 'a.type')
else with_metadata(b, 'ARROW:extension:name', 'b.type') end
from t;There was a problem hiding this comment.
I wish this just wouldn't plan but I didn't make the change in this or my prior PR (#24865) because of some unpleasant implications.
In my prior planning PR we made mismatched "ARROW:extension:name" an error but the pair of a present and a null one goes to the present value.
IMO this should be a planning error because ARROW:extension:name really defines the type and types should match between branches but it leads to some unpleasant situations especially around literals, for example COALESCE(json_col, '{}') would fail.
This pain is just kind of a side effect of ARROW:extension:name being an insufficient type system. Here its limitation is that it cant describe what kinds of up/down casts are allowed
A CASE with a constant WHEN now keeps the CASE when dropping a branch would change its field metadata, and a literal result takes the CASE's metadata. A Boolean CASE whose branch has metadata is no longer rewritten to AND/OR, which cannot carry it.
|
Thanks @jayzhan211, great catches. Addressed in 39697af. Constant WHENs (~L1485): took your diff with one tweak: if the simplified result is a literal, it takes the CASE's metadata, so You're right that a simplifier without a schema now errors on AND/OR lowering (~L1555, L1613): added your Docs: added the CASE entry to |
jayzhan211
left a comment
There was a problem hiding this comment.
@james-willis thanks for the quick turnaround! Every query from my last review now works, and when I remove any of the new simplifier guards, simplify_expr_case_keeps_field_metadata fails. Two cost issues, both inline with fixes I ran locally:
- planning time for a CASE nested inside function calls now grows much faster than on main (about 50× slower at 12 levels);
- the per-batch
return_fieldgrows quadratically with nesting depth.
The logical CASE field reads its type from the branch fields instead of calling get_type, which walked the branches a second time and made planning a CASE nested in function calls grow as 3^depth. The physical CASE computes only the metadata of a nested CASE branch, not its name and nullability, so return_field no longer grows quadratically with nesting depth. Also write the conflicting extension name error in metadata.slt as plain text.
|
thanks for all the help @jayzhan211 I made the tweaks from your review. |
jayzhan211
left a comment
There was a problem hiding this comment.
@james-willis thanks for applying both changes! One more nesting shape is left; details and a fix I ran are inline.
| Ok(Arc::new(metadata.add_to_field(Field::new( | ||
| &schema_name, | ||
| data_type, | ||
| self.nullable(schema)?, |
There was a problem hiding this comment.
One nesting shape still blows up: with the CASE in the ELSE and a non-nullable THEN, 20 levels of upper(CASE WHEN a > k THEN 'x' ELSE <inner> END) take 77 s to EXPLAIN here vs 1.9 s on main (debug build). to_field walks every branch, then self.nullable walks the ELSE again because 'x' can't be null, so the cost is 2^depth, where main walked the ELSE once. Taking each branch's nullability from the fields you already collected makes it linear: 0.07 s at depth 20 with the same plan. The expr, optimizer and physical-expr lib tests and the full sqllogictest suite pass with it.
// to_field, CASE arm
case_nullable(case, schema, |i| Ok(fields[i].is_nullable()))?,
// nullable, CASE arm
Expr::Case(case) => case_nullable(case, input_schema, |i| match case.when_then_expr.get(i) {
Some((_, then)) => then.nullable(input_schema),
None => case
.else_expr
.as_ref()
.map_or(Ok(true), |e| e.nullable(input_schema)),
}),Full diff (moves the existing nullable logic into a helper), query generator and timings
use crate::expr::{
- AggregateFunction, AggregateFunctionParams, Alias, BinaryExpr, Cast, InList,
+ AggregateFunction, AggregateFunctionParams, Alias, BinaryExpr, Case, Cast, InList,
@@ fn nullable
- Expr::Case(case) => {
- let nullable_then = case.when_then_expr.iter().find_map(|(w, t)| {
- let is_nullable = match t.nullable(input_schema) {
- ... (the rest of the current arm, moved unchanged into `case_nullable`)
- }
+ Expr::Case(case) => {
+ case_nullable(case, input_schema, |i| match case.when_then_expr.get(i) {
+ Some((_, then)) => then.nullable(input_schema),
+ None => case
+ .else_expr
+ .as_ref()
+ .map_or(Ok(true), |e| e.nullable(input_schema)),
+ })
+ }
@@ fn to_field, CASE arm
- self.nullable(schema)?,
+ case_nullable(case, schema, |i| Ok(fields[i].is_nullable()))?,
@@
+/// The nullability of `case`, given the nullability of each value branch:
+/// the THEN expressions in order, then the ELSE.
+fn case_nullable(
+ case: &Case,
+ input_schema: &dyn ExprSchema,
+ mut branch_nullable: impl FnMut(usize) -> Result<bool>,
+) -> Result<bool> {
+ let nullable_then = case
+ .when_then_expr
+ .iter()
+ .enumerate()
+ .find_map(|(i, (w, t))| {
+ let is_nullable = match branch_nullable(i) {
+ Err(e) => return Some(Err(e)),
+ Ok(n) => n,
+ };
+ // ... unchanged from the current arm ...
+ });
+
+ if let Some(nullable_then) = nullable_then {
+ nullable_then.map(|_| true)
+ } else if case.else_expr.is_some() {
+ branch_nullable(case.when_then_expr.len())
+ } else {
+ Ok(true)
+ }
+}python3 - 20 > q.sql <<'PY'
import sys
d = int(sys.argv[1]); e = "s"
for k in range(d):
e = f"upper(CASE WHEN a > {k} THEN 'x' ELSE {e} END)"
print("CREATE TABLE t AS VALUES (1, 'x');")
print("CREATE VIEW v AS SELECT column1 AS a, column2 AS s FROM t;")
print(f"EXPLAIN SELECT {e} AS r FROM v;")
PY
time datafusion-cli -q -f q.sql > /dev/null| depth | main | this PR | with the change |
|---|---|---|---|
| 16 | 0.04 s | 4.5–6.0 s | 0.08 s |
| 18 | 0.43 s | 22 s | 0.07 s |
| 20 | 1.9 s | 77 s | 0.07 s |
There was a problem hiding this comment.
Current main is exponential on this shape too. I bisected between this PR’s merge base and main and landed on apache/datafusion#27139, merged Nov 6. It changed CASE get_type to read every branch, and main’s to_field calls get_type and then nullable(), so the ELSE is read twice per level. At 18 levels, the commit before apache/datafusion#27139 plans your query in 0.06 s and apache/datafusion#27139 takes 22 s.
While looking at apache/datafusion#27139, I noticed it also conflicts with this PR. Merged as-is, our to_field would undo it: DESCRIBE SELECT CASE WHEN false THEN 1 ELSE 2.5 END reports Int64 instead of Float64. A test already on main catches this (union_infers_types_before_analysis), but this PR’s CI skips the full test jobs, so it would first have failed in the merge queue.
So I merged main and moved apache/datafusion#27139’s type rule into one helper that get_type and our to_field both call, so each branch is still read only once. Your case_nullable change is in too.
I also timed 17 nesting shapes, including this one. Against current main, 10 take the same time, and 6 that are exponential on main are linear here. The last one, COALESCE nested in its first argument, was about 1.4x slower than main because the constant-WHEN simplification read the branches several times. It now reads each once and is about 1.06x main. That shape is still exponential on both, because the simplifier rewrites COALESCE to a CASE that repeats the argument.
Pushed in afd47cb, and the description’s performance section has the full numbers.
I didn’t appreciate how complex this part of the code base was when I started working on this 😅
to_field already has the field of every branch, so it now derives the nullability from those fields instead of calling nullable, which walked the branches a second time. The shared logic moves into case_nullable.
The constant-WHEN simplification cloned the CASE and computed its field and then the field of the result, walking the branches several times. It now reads each branch's field once, decides from those fields whether the rewrite keeps the CASE's metadata, and only then rewrites the branches. Also check that dropping a branch without metadata still simplifies when the metadata comes from the ELSE.
Since apache#27139, get_type gives a CASE the common type of its THEN and ELSE types. to_field took the first non-NULL branch type, so a CASE field could have a different type than get_type before analysis. Both now share case_result_type, and to_field computes it from the branch fields it already has.
paleolimbot
left a comment
There was a problem hiding this comment.
Thank you for this PR (and reviews)!
I am not an expert in CASE WHEN usage or the work that went in to optimizing this a few releases ago; however, I did review the metadata propagation which flows through the previously added coerced_fields_metadata() in all the cases I see here. I think the change is appropriately scoped and I verified that it can run a CASE WHEN with SedonaDB's extension types.
I also had Codex review for correctness and it didn't find issues.
jayzhan211
left a comment
There was a problem hiding this comment.
@james-willis thanks for the bisect and for merging main! I checked every constant-WHEN CASE with up to three WHENs (21,690 of them), and each keeps the same metadata and type after simplification. One nesting shape is left; details and a fix I ran are inline.
Optional cleanup: the description is about 8,600 characters. Moving "Differences from #26145 + #26158" and "Performance" into <details> would let a maintainer see the problem and the change first.
| } | ||
| } | ||
|
|
||
| fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> { |
There was a problem hiding this comment.
@james-willis thanks for tracking down #27139 and sharing the type rule! A cast between nested CASE levels is still exponential, and this PR makes it grow faster than main: at 8 levels one return_field call takes 842 µs vs 56 µs on main, and ScalarFunctionExpr makes that call for every batch (evaluating this CASE on 8,192 rows takes about 0.15 ms). CastExpr::nullable calls both the child's nullable and return_field, and return_field here calls self.nullable plus each branch's return_field, so each level repeats the one below about 2.6 times (2 times on main). Building the field from one visit of each branch, with the nullability taken from the branch fields as to_field now does, brings it to 7 µs. EXPLAIN at 16 levels drops from 21.9 s (13.3 s on main) to 0.12 s, with the same plan:
fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> {
Ok(Arc::new(
self.unnamed_field(input_schema)?
.with_name(format!("{self}")),
))
}Full diff (replaces return_metadata; nullable_from is the current nullable body, moved unchanged with its comments, except where noted), the shape and timings
-use datafusion_common::metadata::{FieldMetadata, coerced_fields_metadata};
+use datafusion_common::metadata::coerced_fields_metadata;
@@ impl CaseExpr
- /// The metadata of [`PhysicalExpr::return_field`]. ...
- fn return_metadata(&self, input_schema: &Schema) -> Result<FieldMetadata> {
+ /// [`PhysicalExpr::return_field`] without the name, from one visit of each value
+ /// branch. A nested CASE branch is visited the same way, since its name would
+ /// format its whole subtree again at every level.
+ fn unnamed_field(&self, input_schema: &Schema) -> Result<Field> {
let fields = self
...
.map(|expr| match expr.downcast_ref::<CaseExpr>() {
- Some(case) => { ... case.return_metadata(input_schema)? ... }
+ Some(case) => case.unnamed_field(input_schema).map(Arc::new),
None => expr.return_field(input_schema),
})
.collect::<Result<Vec<_>>>()?;
- coerced_fields_metadata(fields.iter().map(|f| f.as_ref()))
+ let (then_fields, else_field) = fields.split_at(self.body.when_then_expr.len());
+ let data_type = then_fields
+ .iter()
+ .map(|f| f.data_type())
+ .find(|t| !t.is_null())
+ .or_else(|| else_field.first().map(|f| f.data_type()))
+ .unwrap_or(&DataType::Null)
+ .clone();
+ let nullable =
+ self.nullable_from(input_schema, |i| Ok(fields[i].is_nullable()))?;
+ let metadata = coerced_fields_metadata(fields.iter().map(|f| f.as_ref()))?;
+ Ok(metadata.add_to_field(Field::new("", data_type, nullable)))
+ }
+
+ /// The nullability of this CASE, given the nullability of each value branch:
+ /// the THEN expressions in order, then the ELSE.
+ fn nullable_from(
+ &self,
+ input_schema: &Schema,
+ mut branch_nullable: impl FnMut(usize) -> Result<bool>,
+ ) -> Result<bool> {
+ // the current `nullable` body, with `.enumerate()` on the THEN iterator,
+ // `t.nullable(input_schema)` -> `branch_nullable(i)`, and the ELSE arm:
+ } else if self.body.else_expr.is_some() {
+ branch_nullable(self.body.when_then_expr.len())
+ } else {
+ Ok(true)
+ }
+ }
@@ impl PhysicalExpr for CaseExpr
fn nullable(&self, input_schema: &Schema) -> Result<bool> {
- (body moved to `nullable_from`)
+ self.nullable_from(input_schema, |i| match self.body.when_then_expr.get(i) {
+ Some((_, then)) => then.nullable(input_schema),
+ None => self
+ .body
+ .else_expr
+ .as_ref()
+ .map_or(Ok(true), |e| e.nullable(input_schema)),
+ })
}The shape, built with case and cast in a release-mode test, alternating Utf8 and LargeUtf8:
CASE WHEN a > k THEN arrow_cast('x', T_k) ELSE arrow_cast(<inner>, T_k) END| depth | main |
this PR | with the change |
|---|---|---|---|
| 4 | 3.4 µs | 12 µs | 2.5 µs |
| 8 | 56 µs | 842 µs | 7.3 µs |
| 16 | 16 ms | 1.8 s | 24 µs |
CASE nested directly in its ELSE stays as fast as now (2.5 µs vs 5.6 µs at depth 8). With the change, the physical-expr lib tests, the full sqllogictest suite and core_integration sql::union_schema pass.
`CaseExpr::return_field` called `nullable` and each branch's `return_field`, so a cast between nested CASE levels, whose `nullable` also calls its child's `return_field`, repeated each level about 2.6 times. The field now takes its type, nullability and metadata from one visit of each branch, using the same rules as `data_type` and `nullable`.
# Conflicts: # datafusion/sqllogictest/test_files/metadata.slt
jayzhan211
left a comment
There was a problem hiding this comment.
Thanks @james-willis and @paleolimbot !
|
Thank you for helping shepherd this through @jayzhan211 |
Which issue does this PR close?
CASEexpressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs #24860.This is a smaller alternative to #26145 + #26158, following the suggestion on #24865 to look for a simpler way to reach the same goal. It also includes the simplifier fix from #26951, which depends on the CASE fix here.
Rationale for this change
CASEdrops the field metadata of its branches, so Arrow extension types are lost. That breaks functions that check the extension type of their argument when the argument is aCASE, or aCOALESCE,NVL,IFNULLorNVL2, since those are rewritten toCASEbefore execution. Examples arevariant_getover a Parquet Variant column, reported on #26145, and geometry and raster functions in SedonaDB.The metadata is dropped in the logical
Expr::Casefield, in the physicalCaseExpr, which has noreturn_fieldoverride, and in simplifier rewrites that drop the branch carrying it.What changes are included in this PR?
Expr::Case::to_fieldmerges the fields of its THEN and ELSE branches withcoerced_fields_metadata, the helperCOALESCE/NVL2use since fix: propagate field metadata through coalesce, nvl, and nvl2 #24865. It takes the type and nullability from the same fields, so it reads each branch once. The type rule from fix: infer UNION output types before analysis #27139 moves into a helper thatget_typealso uses.CaseExpr::return_fielddoes the same with its branches'return_field.CaseExpr::try_new. It is still evaluated like no ELSE.CASE WHEN false THEN x END, takes the CASE's metadata. A CASE is not simplified when that would change its metadata, and a Boolean CASE with metadata on a branch is not rewritten to AND/OR.So CASE follows the same rules as
COALESCE: the earlier branch wins for each key, an extension name is kept together with its parameters, differing extension names are a planning error, and a branch without metadata or an untypedNULLdoes not block another branch's metadata. The operand of a simpleCASEand the WHEN conditions do not contribute.Differences from #26145 + #26158
About 290 lines of non-test code, against about 700 there. That includes the simplifier changes and about 70 lines that move the existing CASE type and nullability logic into helpers.
NOTpasses its input's metadata through while logicalNOTdrops it (Expr::NotandPhysicalExpr::NotExprhave different metadata propagation rules #27155), and an alias's metadata can be lost when the physical expression is created (Alias metadata may be lost when the physical expression is created #27156). WithNOT, this can be a planning error thatmaindoesn't raise, when the other branch has a different extension name. Those mismatches are inNOTandAliasand are not addressed here.Performance
Local runs on a laptop against
main, so treat them as indicative: Criterion runs of the benchmarks from #26145/#26158, andEXPLAINtimings with a debug build ofdatafusion-clifor planning.CASE evaluation and physical planning are unchanged.
return_fieldrebuilds the merged metadata on every call: 954 ns vs 274 ns for a column CASE with 4 metadata keys. A function asks its CASE argument for its field once per batch; field plus evaluation for 8,192 rows took 13.6 µs vs 13.1 µs.Logical
to_fieldfor a wide CASE without any metadata took 2–3x as long.Both were measured before the nesting changes below, which only remove work.
Nested CASE.
to_field,return_fieldand the constant-WHEN simplification visit each branch once. I timed 18 nesting shapes up to 22 levels: the CASE in the THEN, the ELSE or a WHEN; nested directly, in a function call, in||or under a cast; with a nullable or non-nullable other branch; a simple CASE; and COALESCE nested in either argument. The plans are the same as onmain.main.mainand linear here. With the CASE in a function call, 6 shapes take 16–22 s onmainat 18 levels and 0.05 s here. With a cast between levels (found by @jayzhan211),maintakes 13.2 s at 16 levels and this PR 0.06 s.Fast paths can come later: reuse the metadata when only one branch has any, and return early when none has.
What is the testing strategy for this PR?
metadata.sltcases check the metadata at execution witharrow_metadata, mostly with the built-inJSONextension type. They cover precedence, simple CASE, conflicting extension names, a branch without metadata, no ELSE, an untyped and a typed NULL ELSE, coercion to a common type, nested CASE, the CASE rewrites ofcoalesce,nvl,ifnullandnvl2, constant-WHEN simplification, and a Boolean CASE. Each check fails without this change. Each query uses a given CASE only once, because a repeated expression is computed in a lower projection whose field comes from the logical plan.NOT(CASE ...), which SQL can't observe because physicalNOTpasses the metadata through (Expr::NotandPhysicalExpr::NotExprhave different metadata propagation rules #27155).datafusion-expr,datafusion-physical-expranddatafusion-optimizerlib tests pass. The full sqllogictest suite and thecore_integrationsqltests show no new failures; my checkout doesn't have the test-data submodules, so the files that need that data fail the same way onmain.metadata.sltfrom Preserve field metadata through physical CASE expressions #26158, which includes the cases from test: Add sqllogictests for field metadata through coalesce, nvl2 and CASE #26895, also passes with this change, checked locally with the current head of Preserve field metadata through physical CASE expressions #26158.Are there any user-facing changes?
CASEresults keep their field metadata, including extension types, also when the optimizer simplifies the CASE. So doCOALESCE,NVL,IFNULLandNVL2at execution.CASEbranches with differing extension names are now a planning error, as they already are forCOALESCE.No public API changes.