Skip to content

fix: preserve field metadata through CASE - #27143

Merged
jayzhan211 merged 11 commits into
apache:mainfrom
james-willis:case-field-metadata
Nov 21, 2026
Merged

jayzhan211 merged 11 commits into
apache:mainfrom
james-willis:case-field-metadata

Conversation

@james-willis

@james-willis james-willis commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

This is a smaller alternative to #26145 + #26158, following the suggestion on #24865 to look for a simpler way to reach the same goal. It also includes the simplifier fix from #26951, which depends on the CASE fix here.

Rationale for this change

CASE drops the field metadata of its branches, so Arrow extension types are lost. That breaks functions that check the extension type of their argument when the argument is a CASE, or a COALESCE, NVL, IFNULL or NVL2, since those are rewritten to CASE before execution. Examples are variant_get over a Parquet Variant column, reported on #26145, and geometry and raster functions in SedonaDB.

The metadata is dropped in the logical Expr::Case field, in the physical CaseExpr, which has no return_field override, and in simplifier rewrites that drop the branch carrying it.

What changes are included in this PR?

  • Logical Expr::Case::to_field merges the fields of its THEN and ELSE branches with coerced_fields_metadata, the helper COALESCE/NVL2 use since fix: propagate field metadata through coalesce, nvl, and nvl2 #24865. It takes the type and nullability from the same fields, so it reads each branch once. The type rule from fix: infer UNION output types before analysis #27139 moves into a helper that get_type also uses.
  • Physical CaseExpr::return_field does the same with its branches' return_field.
  • A typed NULL ELSE that carries metadata is no longer dropped by CaseExpr::try_new. It is still evaluated like no ELSE.
  • CASE simplifications keep the metadata. A literal left by a constant WHEN, such as the NULL from CASE WHEN false THEN x END, takes the CASE's metadata. A CASE is not simplified when that would change its metadata, and a Boolean CASE with metadata on a branch is not rewritten to AND/OR.

So CASE follows the same rules as COALESCE: the earlier branch wins for each key, an extension name is kept together with its parameters, differing extension names are a planning error, and a branch without metadata or an untyped NULL does not block another branch's metadata. The operand of a simple CASE and the WHEN conditions do not contribute.

Differences from #26145 + #26158

About 290 lines of non-test code, against about 700 there. That includes the simplifier changes and about 70 lines that move the existing CASE type and nullability logic into helpers.

Performance

Local runs on a laptop against main, so treat them as indicative: Criterion runs of the benchmarks from #26145/#26158, and EXPLAIN timings with a debug build of datafusion-cli for planning.

  • CASE evaluation and physical planning are unchanged.

  • return_field rebuilds the merged metadata on every call: 954 ns vs 274 ns for a column CASE with 4 metadata keys. A function asks its CASE argument for its field once per batch; field plus evaluation for 8,192 rows took 13.6 µs vs 13.1 µs.

  • Logical to_field for a wide CASE without any metadata took 2–3x as long.

    Both were measured before the nesting changes below, which only remove work.

  • Nested CASE. to_field, return_field and the constant-WHEN simplification visit each branch once. I timed 18 nesting shapes up to 22 levels: the CASE in the THEN, the ELSE or a WHEN; nested directly, in a function call, in || or under a cast; with a nullable or non-nullable other branch; a simple CASE; and COALESCE nested in either argument. The plans are the same as on main.

    • 10 take the same time as on main.
    • 7 are exponential on main and linear here. With the CASE in a function call, 6 shapes take 16–22 s on main at 18 levels and 0.05 s here. With a cast between levels (found by @jayzhan211), main takes 13.2 s at 16 levels and this PR 0.06 s.
    • COALESCE nested in its first argument is exponential on both, because the simplifier rewrites it to a CASE that repeats that argument: 4.05 s vs 3.82 s at 18 levels.

Fast paths can come later: reuse the metadata when only one branch has any, and return early when none has.

What is the testing strategy for this PR?

  • New metadata.slt cases check the metadata at execution with arrow_metadata, mostly with the built-in JSON extension type. They cover precedence, simple CASE, conflicting extension names, a branch without metadata, no ELSE, an untyped and a typed NULL ELSE, coercion to a common type, nested CASE, the CASE rewrites of coalesce, nvl, ifnull and nvl2, constant-WHEN simplification, and a Boolean CASE. Each check fails without this change. Each query uses a given CASE only once, because a repeated expression is computed in a lower projection whose field comes from the logical plan.
  • Unit tests check that the physical CASE field matches the logical one in metadata, type and nullability, and cover the CASE simplifications. That includes the rewrite of a Boolean CASE to NOT(CASE ...), which SQL can't observe because physical NOT passes the metadata through (Expr::Not and PhysicalExpr::NotExpr have different metadata propagation rules #27155).
  • Existing tests: the datafusion-expr, datafusion-physical-expr and datafusion-optimizer lib tests pass. The full sqllogictest suite and the core_integration sql tests show no new failures; my checkout doesn't have the test-data submodules, so the files that need that data fail the same way on main.
  • The metadata.slt from Preserve field metadata through physical CASE expressions #26158, which includes the cases from test: Add sqllogictests for field metadata through coalesce, nvl2 and CASE #26895, also passes with this change, checked locally with the current head of Preserve field metadata through physical CASE expressions #26158.

Are there any user-facing changes?

  • CASE results keep their field metadata, including extension types, also when the optimizer simplifies the CASE. So do COALESCE, NVL, IFNULL and NVL2 at execution.
  • CASE branches with differing extension names are now a planning error, as they already are for COALESCE.

No public API changes.

The logical Expr::Case field and the physical CaseExpr field now merge the
metadata of the value branches with coerced_fields_metadata, the same rules
COALESCE and NVL2 use: the earlier branch wins for each key, an extension
name stays with its parameters, and differing extension names are a planning
error.

CaseExpr::try_new keeps a typed NULL ELSE that carries metadata, and a
NULL-literal ELSE still evaluates like no ELSE.
@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Nov 5, 2026
@james-willis
james-willis marked this pull request as ready for review November 5, 2026 19:52
@james-willis

Copy link
Copy Markdown
Contributor Author

@paleolimbot @alamb @osipovartem This is my alternative solution for preserving metadata in those cases.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach makes sense to me but I'll defer to those more familiar with the case/when optimizations as to whether the other approach is preferable!

Codex complained to me that this approach requires that logical and physical representations of an expr have to match for this to work and pointed out a few places they don't at the moment which I filed as separate issues. Its noted test gap was "No test compares Expr::Case::to_field with CaseExpr::return_field for the same expression."

One more gap it suggested was to check where the CASE test where branches have different storage types and coercion inserts a cast. This matters because type-only casts strip extension metadata, and logical and physical cast behavior must agree.

Compare Expr::Case::to_field with the physical return_field, including a
branch cast to the common type by type coercion. Use the built-in JSON
extension type in the CASE metadata.slt cases, add a coercion case, and
move the CASE WHEN false case next to the other CASE cases.
@github-actions github-actions Bot added the optimizer Optimizer rules label Nov 6, 2026
@james-willis

Copy link
Copy Markdown
Contributor Author

Thanks Dewey. Integrated all your feedback.

Logical vs physical. Agreed, this relies on each branch's physical field matching its logical one, which is the property #27156 describes for every expression. CASE derives its field only from its branches in both layers, so it matches whenever its branches do, and a branch that doesn't (NOT in #27155, Alias in #27156) shows through. #26158 avoids that by carrying the logical field into the physical CASE, at the cost of storing it and a protobuf change. In practice the mismatch doesn't change the result schema, which takes its metadata from the logical plan. Only a function that takes the CASE as an argument sees the physical field. in the NOT case probably this doesnt really matter - its probably just extra metadata. In the Alias case it can be an issue but this is probably exceeding rate to call alias on a subexpression that doesnt persist past the operator.

test_case_lowering_preserves_field_metadata in physical-expr/src/planner.rs (8ef3402) now compares Expr::Case::to_field with the physical return_field for column, literal, cast and nested CASE branches, including a typed NULL ELSE with metadata.

Coercion casts. Logical and physical agree: the type-only cast that coercion inserts drops the extension type and keeps other keys on both sides. The unit test above covers it, and a new metadata.slt case checks that CASE ... THEN cast(name AS JSON) ELSE arrow_cast(l_name, 'LargeUtf8') END is LargeUtf8 without an extension type.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@james-willis thanks for this! In expr_simplifier.rs around line 1485, the other rewrites in this arm still drop the merged metadata: ext_only(CASE WHEN 1 = 0 THEN g ELSE NULL END) (the UDF from the #24860 repro) fails planning with "argument lost its extension type", while CASE WHEN 1 = 0 THEN g END works. SELECT CASE WHEN 1 = 0 THEN g ELSE plain END AS x also reports the extension in df.schema() but not in the batches. Could the arm compare the field metadata before and after and keep the CASE when it changes? I ran this diff: every ext_only probe in the table below passes and the optimizer lib tests still pass. It does make the arm read the ELSE's field, so from reading the code (I didn't run this one) a simplifier without a schema would now error on CASE WHEN true THEN 1 ELSE col END.

-                // CASE WHEN false THEN A END --> NULL
-                // The NULL keeps the CASE's field metadata, such as an extension type
-                if else_expr.is_none()
-                    && when_then_expr
-                        .iter()
-                        .all(|(when, _)| is_false(when.as_ref()))
-                {
-                    let (_, field) = Expr::Case(Case::new(None, when_then_expr, None))
-                        .to_field(info.schema().as_ref())?;
-                    let metadata = FieldMetadata::new_from_field(&field);
-                    let null = Expr::Literal(
-                        ScalarValue::try_new_null(field.data_type())?,
-                        (!metadata.is_empty()).then_some(metadata),
-                    );
-                    return Ok(Transformed::yes(null));
-                }
+                // Dropping a branch must not change the field of the CASE, whose
+                // metadata merges all of its value branches.
+                let original = Expr::Case(Case::new(
+                    None,
+                    when_then_expr.clone(),
+                    else_expr.clone(),
+                ));
+                let schema = info.schema().as_ref();
+                let (_, field) = original.to_field(schema)?;
+                let metadata = FieldMetadata::new_from_field(&field);
@@
                 // Exclude CASE statement altogether if there are no when-then expressions left
-                // CASE WHEN false THEN A ELSE B END --> B
-                if new_when_then_expr.is_empty()
-                    && let Some(else_expr) = else_expr
-                {
-                    return Ok(Transformed::yes(*else_expr));
-                }
+                let simplified = match (new_when_then_expr.is_empty(), else_expr) {
+                    // CASE WHEN false THEN A END --> NULL
+                    (true, None) => {
+                        return Ok(Transformed::yes(Expr::Literal(
+                            ScalarValue::try_new_null(field.data_type())?,
+                            (!metadata.is_empty()).then_some(metadata),
+                        )));
+                    }
+                    // CASE WHEN false THEN A ELSE B END --> B
+                    (true, Some(else_expr)) => *else_expr,
+                    (false, else_expr) => Expr::Case(Case {
+                        expr: None,
+                        when_then_expr: new_when_then_expr,
+                        else_expr,
+                    }),
+                };
 
-                Transformed::yes(Expr::Case(Case {
-                    expr: None,
-                    when_then_expr: new_when_then_expr,
-                    else_expr,
-                }))
+                let (_, new_field) = simplified.to_field(schema)?;
+                if FieldMetadata::new_from_field(&new_field) == metadata {
+                    Transformed::yes(simplified)
+                } else {
+                    Transformed::no(original)
+                }
Queries to add to the tests, and the results with this diff

Each arrow_metadata query should return arrow.json, the same as the non-constant CASE (expected results; I ran the same shapes through ext_only below, not these exact queries):

select arrow_metadata(case when 1 = 0 then cast(name as json) else NULL end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when 1 = 0 then cast(name as json) else upper(l_name) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when 1 = 1 then upper(l_name) else cast(name as json) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(case when id > 1 then upper(l_name) when 1 = 0 then cast(name as json) end, 'ARROW:extension:name') from table_with_metadata;
select arrow_metadata(coalesce(nonnull_name, cast(name as json)), 'ARROW:extension:name') from table_with_metadata;

With ext_only requiring the extension on its argument (g has one; plain and nn don't, and nn is non-nullable):

Argument Now With the diff
CASE WHEN 1 = 0 THEN g ELSE NULL END plan error ok
CASE WHEN 1 = 0 THEN g ELSE plain END plan error ok
CASE WHEN 1 = 1 THEN plain ELSE g END plan error ok
CASE WHEN k = 1 THEN plain WHEN 1 = 0 THEN g END plan error ok
COALESCE(nn, g) plan error ok
NVL2(nn, plain, g) plan error ok

In expr_simplifier.rs around line 1555, the boolean lowering loses the merged metadata too: for CASE WHEN p THEN f END, with f a Boolean column carrying metadata, df.schema() reports it but the batches don't, and a metadata-checking UDF fails to plan. Could this arm skip the rewrite when a value branch has metadata, with a test on that query? Line 1613 looks like the same rewrite for literal THENs, so it likely needs the same guard; that's from reading the code, not a probe, and I haven't compiled this snippet.

/// Whether a value branch of a CASE has field metadata, which AND/OR cannot keep
fn case_branch_has_metadata(
    when_then_expr: &[(Box<Expr>, Box<Expr>)],
    else_expr: Option<&Expr>,
    info: &SimplifyContext,
) -> Result<bool> {
    let branches = when_then_expr
        .iter()
        .map(|(_, then)| then.as_ref())
        .chain(else_expr);
    for expr in branches {
        let (_, field) = expr.to_field(info.schema().as_ref())?;
        if !field.metadata().is_empty() {
            return Ok(true);
        }
    }
    Ok(false)
}
-                && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref()) =>
+                && can_lower_case_to_boolean(&when_then_expr, else_expr.as_deref())
+                && !case_branch_has_metadata(&when_then_expr, else_expr.as_deref(), info)? =>

Optional cleanup: the "Metadata Handling" list on to_field in expr_schema.rs around line 509 could get a CASE entry now that CASE has its own rule.

     /// - **Binary expressions**: field metadata is empty
+    /// - **CASE expressions**: Merges the metadata of the THEN and ELSE branches; the operand and WHEN conditions do not contribute

}
}

fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The NOT mismatch from #27155 can be a planning error here, not only extra metadata: this query returns rows on main and now fails with "Cannot merge field metadata with differing values". It's fine to handle this in #27155 since the fix belongs in NOT; could you add this query there and adjust the note in the PR description?

-- t(id INT, b BOOLEAN)
select case when id > 1
  then not with_metadata(b, 'ARROW:extension:name', 'a.type')
  else with_metadata(b, 'ARROW:extension:name', 'b.type') end
from t;

@james-willis james-willis Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish this just wouldn't plan but I didn't make the change in this or my prior PR (#24865) because of some unpleasant implications.

In my prior planning PR we made mismatched "ARROW:extension:name" an error but the pair of a present and a null one goes to the present value.

IMO this should be a planning error because ARROW:extension:name really defines the type and types should match between branches but it leads to some unpleasant situations especially around literals, for example COALESCE(json_col, '{}') would fail.

This pain is just kind of a side effect of ARROW:extension:name being an insufficient type system. Here its limitation is that it cant describe what kinds of up/down casts are allowed

A CASE with a constant WHEN now keeps the CASE when dropping a branch
would change its field metadata, and a literal result takes the CASE's
metadata. A Boolean CASE whose branch has metadata is no longer rewritten
to AND/OR, which cannot carry it.
@james-willis

Copy link
Copy Markdown
Contributor Author

Thanks @jayzhan211, great catches. Addressed in 39697af.

Constant WHENs (~L1485): took your diff with one tweak: if the simplified result is a literal, it takes the CASE's metadata, so ELSE NULL and CASE WHEN true THEN 'x' ELSE g END still fold instead of keeping the whole CASE. Anything else only simplifies if its metadata matches the CASE's. Your queries, plus an nvl2 one, are in metadata.slt.

You're right that a simplifier without a schema now errors on CASE WHEN true THEN 1 ELSE col END. I left it since CASE WHEN false THEN col END and CASE WHEN c THEN col END already error without a schema today.

AND/OR lowering (~L1555, L1613): added your case_branch_has_metadata guard to both arms. The NOT(CASE ...) one can't be shown from SQL yet because physical NOT passes the metadata through (#27155), so simplify_expr_case_keeps_field_metadata covers it instead. Your CASE WHEN p THEN f END query is in metadata.slt.

Docs: added the CASE entry to to_field.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@james-willis thanks for the quick turnaround! Every query from my last review now works, and when I remove any of the new simplifier guards, simplify_expr_case_keeps_field_metadata fails. Two cost issues, both inline with fixes I ran locally:

  • planning time for a CASE nested inside function calls now grows much faster than on main (about 50× slower at 12 levels);
  • the per-batch return_field grows quadratically with nesting depth.

Comment thread datafusion/expr/src/expr_schema.rs Resolved
The logical CASE field reads its type from the branch fields instead of
calling get_type, which walked the branches a second time and made planning
a CASE nested in function calls grow as 3^depth.

The physical CASE computes only the metadata of a nested CASE branch, not its
name and nullability, so return_field no longer grows quadratically with
nesting depth.

Also write the conflicting extension name error in metadata.slt as plain text.
@james-willis

Copy link
Copy Markdown
Contributor Author

thanks for all the help @jayzhan211 I made the tweaks from your review.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@james-willis thanks for applying both changes! One more nesting shape is left; details and a fix I ran are inline.

Comment thread datafusion/expr/src/expr_schema.rs Outdated
Ok(Arc::new(metadata.add_to_field(Field::new(
&schema_name,
data_type,
self.nullable(schema)?,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nesting shape still blows up: with the CASE in the ELSE and a non-nullable THEN, 20 levels of upper(CASE WHEN a > k THEN 'x' ELSE <inner> END) take 77 s to EXPLAIN here vs 1.9 s on main (debug build). to_field walks every branch, then self.nullable walks the ELSE again because 'x' can't be null, so the cost is 2^depth, where main walked the ELSE once. Taking each branch's nullability from the fields you already collected makes it linear: 0.07 s at depth 20 with the same plan. The expr, optimizer and physical-expr lib tests and the full sqllogictest suite pass with it.

// to_field, CASE arm
case_nullable(case, schema, |i| Ok(fields[i].is_nullable()))?,

// nullable, CASE arm
Expr::Case(case) => case_nullable(case, input_schema, |i| match case.when_then_expr.get(i) {
    Some((_, then)) => then.nullable(input_schema),
    None => case
        .else_expr
        .as_ref()
        .map_or(Ok(true), |e| e.nullable(input_schema)),
}),
Full diff (moves the existing nullable logic into a helper), query generator and timings
 use crate::expr::{
-    AggregateFunction, AggregateFunctionParams, Alias, BinaryExpr, Cast, InList,
+    AggregateFunction, AggregateFunctionParams, Alias, BinaryExpr, Case, Cast, InList,
@@ fn nullable
-            Expr::Case(case) => {
-                let nullable_then = case.when_then_expr.iter().find_map(|(w, t)| {
-                    let is_nullable = match t.nullable(input_schema) {
-                    ... (the rest of the current arm, moved unchanged into `case_nullable`)
-            }
+            Expr::Case(case) => {
+                case_nullable(case, input_schema, |i| match case.when_then_expr.get(i) {
+                    Some((_, then)) => then.nullable(input_schema),
+                    None => case
+                        .else_expr
+                        .as_ref()
+                        .map_or(Ok(true), |e| e.nullable(input_schema)),
+                })
+            }
@@ fn to_field, CASE arm
-                    self.nullable(schema)?,
+                    case_nullable(case, schema, |i| Ok(fields[i].is_nullable()))?,
@@
+/// The nullability of `case`, given the nullability of each value branch:
+/// the THEN expressions in order, then the ELSE.
+fn case_nullable(
+    case: &Case,
+    input_schema: &dyn ExprSchema,
+    mut branch_nullable: impl FnMut(usize) -> Result<bool>,
+) -> Result<bool> {
+    let nullable_then = case
+        .when_then_expr
+        .iter()
+        .enumerate()
+        .find_map(|(i, (w, t))| {
+            let is_nullable = match branch_nullable(i) {
+                Err(e) => return Some(Err(e)),
+                Ok(n) => n,
+            };
+            // ... unchanged from the current arm ...
+        });
+
+    if let Some(nullable_then) = nullable_then {
+        nullable_then.map(|_| true)
+    } else if case.else_expr.is_some() {
+        branch_nullable(case.when_then_expr.len())
+    } else {
+        Ok(true)
+    }
+}
python3 - 20 > q.sql <<'PY'
import sys
d = int(sys.argv[1]); e = "s"
for k in range(d):
    e = f"upper(CASE WHEN a > {k} THEN 'x' ELSE {e} END)"
print("CREATE TABLE t AS VALUES (1, 'x');")
print("CREATE VIEW v AS SELECT column1 AS a, column2 AS s FROM t;")
print(f"EXPLAIN SELECT {e} AS r FROM v;")
PY
time datafusion-cli -q -f q.sql > /dev/null
depth main this PR with the change
16 0.04 s 4.5–6.0 s 0.08 s
18 0.43 s 22 s 0.07 s
20 1.9 s 77 s 0.07 s

@james-willis james-willis Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main is exponential on this shape too. I bisected between this PR’s merge base and main and landed on apache/datafusion#27139, merged Nov 6. It changed CASE get_type to read every branch, and main’s to_field calls get_type and then nullable(), so the ELSE is read twice per level. At 18 levels, the commit before apache/datafusion#27139 plans your query in 0.06 s and apache/datafusion#27139 takes 22 s.

While looking at apache/datafusion#27139, I noticed it also conflicts with this PR. Merged as-is, our to_field would undo it: DESCRIBE SELECT CASE WHEN false THEN 1 ELSE 2.5 END reports Int64 instead of Float64. A test already on main catches this (union_infers_types_before_analysis), but this PR’s CI skips the full test jobs, so it would first have failed in the merge queue.

So I merged main and moved apache/datafusion#27139’s type rule into one helper that get_type and our to_field both call, so each branch is still read only once. Your case_nullable change is in too.

I also timed 17 nesting shapes, including this one. Against current main, 10 take the same time, and 6 that are exponential on main are linear here. The last one, COALESCE nested in its first argument, was about 1.4x slower than main because the constant-WHEN simplification read the branches several times. It now reads each once and is about 1.06x main. That shape is still exponential on both, because the simplifier rewrites COALESCE to a CASE that repeats the argument.

Pushed in afd47cb, and the description’s performance section has the full numbers.

I didn’t appreciate how complex this part of the code base was when I started working on this 😅

to_field already has the field of every branch, so it now derives the
nullability from those fields instead of calling nullable, which walked the
branches a second time. The shared logic moves into case_nullable.
The constant-WHEN simplification cloned the CASE and computed its field and
then the field of the result, walking the branches several times. It now reads
each branch's field once, decides from those fields whether the rewrite keeps
the CASE's metadata, and only then rewrites the branches.

Also check that dropping a branch without metadata still simplifies when the
metadata comes from the ELSE.
Since apache#27139, get_type gives a CASE the common type of its THEN and ELSE
types. to_field took the first non-NULL branch type, so a CASE field could
have a different type than get_type before analysis. Both now share
case_result_type, and to_field computes it from the branch fields it already
has.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR (and reviews)!

I am not an expert in CASE WHEN usage or the work that went in to optimizing this a few releases ago; however, I did review the metadata propagation which flows through the previously added coerced_fields_metadata() in all the cases I see here. I think the change is appropriately scoped and I verified that it can run a CASE WHEN with SedonaDB's extension types.

I also had Codex review for correctness and it didn't find issues.

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@james-willis thanks for the bisect and for merging main! I checked every constant-WHEN CASE with up to three WHENs (21,690 of them), and each keeps the same metadata and type after simplification. One nesting shape is left; details and a fix I ran are inline.

Optional cleanup: the description is about 8,600 characters. Moving "Differences from #26145 + #26158" and "Performance" into <details> would let a maintainer see the problem and the change first.

}
}

fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@james-willis thanks for tracking down #27139 and sharing the type rule! A cast between nested CASE levels is still exponential, and this PR makes it grow faster than main: at 8 levels one return_field call takes 842 µs vs 56 µs on main, and ScalarFunctionExpr makes that call for every batch (evaluating this CASE on 8,192 rows takes about 0.15 ms). CastExpr::nullable calls both the child's nullable and return_field, and return_field here calls self.nullable plus each branch's return_field, so each level repeats the one below about 2.6 times (2 times on main). Building the field from one visit of each branch, with the nullability taken from the branch fields as to_field now does, brings it to 7 µs. EXPLAIN at 16 levels drops from 21.9 s (13.3 s on main) to 0.12 s, with the same plan:

fn return_field(&self, input_schema: &Schema) -> Result<FieldRef> {
    Ok(Arc::new(
        self.unnamed_field(input_schema)?
            .with_name(format!("{self}")),
    ))
}
Full diff (replaces return_metadata; nullable_from is the current nullable body, moved unchanged with its comments, except where noted), the shape and timings
-use datafusion_common::metadata::{FieldMetadata, coerced_fields_metadata};
+use datafusion_common::metadata::coerced_fields_metadata;
@@ impl CaseExpr
-    /// The metadata of [`PhysicalExpr::return_field`]. ...
-    fn return_metadata(&self, input_schema: &Schema) -> Result<FieldMetadata> {
+    /// [`PhysicalExpr::return_field`] without the name, from one visit of each value
+    /// branch. A nested CASE branch is visited the same way, since its name would
+    /// format its whole subtree again at every level.
+    fn unnamed_field(&self, input_schema: &Schema) -> Result<Field> {
         let fields = self
             ...
             .map(|expr| match expr.downcast_ref::<CaseExpr>() {
-                Some(case) => { ... case.return_metadata(input_schema)? ... }
+                Some(case) => case.unnamed_field(input_schema).map(Arc::new),
                 None => expr.return_field(input_schema),
             })
             .collect::<Result<Vec<_>>>()?;
-        coerced_fields_metadata(fields.iter().map(|f| f.as_ref()))
+        let (then_fields, else_field) = fields.split_at(self.body.when_then_expr.len());
+        let data_type = then_fields
+            .iter()
+            .map(|f| f.data_type())
+            .find(|t| !t.is_null())
+            .or_else(|| else_field.first().map(|f| f.data_type()))
+            .unwrap_or(&DataType::Null)
+            .clone();
+        let nullable =
+            self.nullable_from(input_schema, |i| Ok(fields[i].is_nullable()))?;
+        let metadata = coerced_fields_metadata(fields.iter().map(|f| f.as_ref()))?;
+        Ok(metadata.add_to_field(Field::new("", data_type, nullable)))
+    }
+
+    /// The nullability of this CASE, given the nullability of each value branch:
+    /// the THEN expressions in order, then the ELSE.
+    fn nullable_from(
+        &self,
+        input_schema: &Schema,
+        mut branch_nullable: impl FnMut(usize) -> Result<bool>,
+    ) -> Result<bool> {
+        // the current `nullable` body, with `.enumerate()` on the THEN iterator,
+        // `t.nullable(input_schema)` -> `branch_nullable(i)`, and the ELSE arm:
+        } else if self.body.else_expr.is_some() {
+            branch_nullable(self.body.when_then_expr.len())
+        } else {
+            Ok(true)
+        }
+    }
@@ impl PhysicalExpr for CaseExpr
     fn nullable(&self, input_schema: &Schema) -> Result<bool> {
-        (body moved to `nullable_from`)
+        self.nullable_from(input_schema, |i| match self.body.when_then_expr.get(i) {
+            Some((_, then)) => then.nullable(input_schema),
+            None => self
+                .body
+                .else_expr
+                .as_ref()
+                .map_or(Ok(true), |e| e.nullable(input_schema)),
+        })
     }

The shape, built with case and cast in a release-mode test, alternating Utf8 and LargeUtf8:

CASE WHEN a > k THEN arrow_cast('x', T_k) ELSE arrow_cast(<inner>, T_k) END
depth main this PR with the change
4 3.4 µs 12 µs 2.5 µs
8 56 µs 842 µs 7.3 µs
16 16 ms 1.8 s 24 µs

CASE nested directly in its ELSE stays as fast as now (2.5 µs vs 5.6 µs at depth 8). With the change, the physical-expr lib tests, the full sqllogictest suite and core_integration sql::union_schema pass.

`CaseExpr::return_field` called `nullable` and each branch's
`return_field`, so a cast between nested CASE levels, whose `nullable`
also calls its child's `return_field`, repeated each level about 2.6
times. The field now takes its type, nullability and metadata from one
visit of each branch, using the same rules as `data_type` and
`nullable`.
# Conflicts:
#	datafusion/sqllogictest/test_files/metadata.slt

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jayzhan211
jayzhan211 added this pull request to the merge queue Nov 21, 2026
Merged via the queue into apache:main with commit 95171ad Nov 21, 2026
42 checks passed
@alamb

alamb commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Thank you for helping shepherd this through @jayzhan211

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions optimizer Optimizer rules physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CASE expressions drop field metadata, silently stripping Arrow extension types and breaking metadata-aware UDFs

4 participants