Skip to content

fix: preserve CASE result field metadata - #26145

Open
osipovartem wants to merge 11 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata
Open

osipovartem wants to merge 11 commits into
apache:mainfrom
Embucket:upstream-case-field-metadata

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Part of #24860, complementary to merged #24865 and physical follow-up #26158.

Current qualification checkpoint (2026-11-15)

Published head 2190ce4cb merges Apache main 3d6a5f1a8 and the struct-backed tests from Embucket/datafusion#186. The refresh aligns CASE result-type inference with native common-type coercion, with regressions for mixed THEN/ELSE types and nested CASE metadata.

Current fork CI passes: 13,556 Rust tests with zero failures and 15 ignored, the configured 307-file SQLLogicTest corpus, strict all-target/all-feature workspace Clippy with -D warnings, and formatting. The SLT selection includes metadata.slt; it excludes the optional SQLite/TPCH sets. The separate extended-test workflow and skipped fork macOS job are not claimed as executed. Independent source review found no blocking issue for maintainer review.

The historical test counts and performance measurements below retain their named source/feature boundaries; they are not fresh performance measurements of this head. API and performance acceptance remain maintainer merge decisions.

Rationale

Logical CASE field inference loses result metadata needed by later planning and metadata-aware function signatures. The shared coerced_fields_metadata policy should apply to THEN/ELSE fields, not the CASE operand or WHEN conditions.

What changes are included?

  • Merge result-field metadata with the existing shared helper: earlier keys win, extension name/parameters stay paired, and incompatible extension names are rejected.
  • Use native to_field for ordinary branches. Remove the depth-8 cutoff and whole-CASE metadata erasure for unhandled expression variants. Preserve the documented non-extension cast metadata, including alias-wrapped NULL CAST/TRY_CAST expressions.
  • Retain a bottom-up path for CASE, forwarding wrappers and binary expressions to avoid repeatedly inferring every suffix. Binary expressions reuse the same private coercion/diagnostic helper as native field inference. The counted regression covers wrapper and binary shapes at depths 8/32/128; generic CASE/UDF inference is not claimed to be globally linear.
  • Preserve the plain column/literal fast path and existing nullability/reachability/short-circuit rules. The defaulted, object-safe ExprSchema::field_and_nullable method preserves custom nullable() overrides; DFSchema obtains both results from one field lookup.

What is the testing strategy?

SQL/extension-policy cases are consolidated in metadata.slt. This logical branch now also includes the struct-backed extension regressions contributed in Embucket#186, preserving the contributor's commit: metadata-sensitive planning errors and unchanged struct values. These are not direct executed struct-extension-metadata assertions. The physical acceptance cases in #26158 include the cases proposed in #26895 and are not presented as end-to-end fixes supplied by this logical-only PR.

Focused Rust tests retain API-only field attachments, pre-coercion type guards, custom schema nullability/errors, and inference-count checks. The aliased NULL cast regression was reproduced failing before its fix. The separate binary/CASE negative control required 97 schema lookups at depth 8 against a bound of 56 before bottom-up binary inference; its final coverage is consolidated into the existing counted test rather than duplicating fixtures. A further negative control verifies that NOT(CAST(NULL AS BOOLEAN)) remains a typed Boolean contributor rather than bypassing the pre-coercion storage-type guard.

Local logical-only validation at 66121cee1: 305 expr unit tests; the documented extended workspace run passed 13,351 Rust tests (8 existing ignored) and 545/545 SLT files. Workspace all-target/all-feature Clippy with -D warnings, rustfmt and diff checks passed. Main was subsequently merged into this branch as bd6ab71dd; the two changed expr source/benchmark files are byte-identical. The main-synced combined stack with physical #26158 at 3018e535a additionally passed 13,383 Rust tests (8 ignored), 545/545 SLT files and strict workspace Clippy. That combined run is not a logical-only run. The public SQL integration checks original/optimized logical metadata, returned values and extension-conflict errors; physical CASE metadata remains #26158's scope.

Performance

The Criterion benchmark now includes nested columns, aliases/casts, binary/CASE chains and alternating CASE/UDFs, in addition to the existing flat literal/column cases. Fixtures are constructed outside timing. Measurements use the same host and ci-bench profile (optimization level 1, Rust 1.99.1); these are field-inference microbenchmarks, not query-runtime or production-release performance claims.

The binary-path screen at 66121cee1, before the main merge, used 20 samples and one-second warmup/measurement. To exclude stale artifacts when different source workspaces share a target directory, the entire benchmark profile was cleaned and rebuilt between revisions. Initial runs without that isolation were discarded. Medians in us:

Fixture Main 49c67c296 Previous 1f70d3d90 Corrected 66121cee1
1000 literals 81.83 76.11 77.42
1000 distinct columns 1560.52 1487.24 1503.37
1000 branches, last marked 1504.40 4468.30 4399.79
Binary/CASE depth 8 6.03 40.78 13.55
Binary/CASE depth 32 23.99 525.97 50.36

Bottom-up binary inference removes the previous PR's quadratic schema-inference work, but this is not parity with main: marked-last planning costs about 2.9x main, and the nested-binary fixtures remain about 2.1-2.2x main. Main does not provide the added CASE metadata semantics, including the marked-last output; the table makes that cost explicit rather than presenting correctness fixes as free. The three flat controls differ about -1.5% to +1.7% from the previous PR in this single sequential screen. These are not whole-query or zero-regression claims, and API/performance acceptance remains a maintainer merge decision.

A straightforward recursive prototype that obtains each result branch through to_field repeatedly revisits nested CASE inference. In the earlier screen against logical 1f70d3d90, the narrowed traversal measured about 40 us versus 365 us for aliases and 59 us versus 2,740 us for casts at depth 128. The lookup-count regression rejects that recursive prototype. This historical comparison justifies retaining the traversal; it is not a speedup over the prior worklist or a fresh timing of the final head.

There is a correctness/performance tradeoff in deeply alternating CASE/UDFs: the previous cutoff silently discarded metadata. An earlier short exploratory run compared about 418/1,686 us at depths 6/8 against 910/7,917 us for logical 1f70d3d90 preserving metadata. This is historical evidence, not a fresh timing of the final head. Repeated type/nullability/field inference remains broader follow-up work; the worklist does not solve it. No zero-regression claim is made for that path.

Two earlier sequential flat-path A/B repetitions used 20 samples with one-second warmup and measurement, comparing original PR head adb0656 against logical 1f70d3d90, before the binary follow-up, not Apache main. Representative historical medians in us:

Fixture Original A / B 1f70d3d90 A / B
1000 literals 72.79 / 73.53 77.45 / 76.24
1000 distinct columns 1453.74 / 1501.28 1513.20 / 1609.86
1000 branches, last marked 4325.65 / 4439.19 4435.02 / 4478.32

Across those six 100/1000-branch flat cases, paired medians increased about 0.9-8.0%. Unchanged fast-path source does not justify dismissing measurements or claiming zero regression. These costs accompany correct metadata preservation and removal of unsupported/depth-based erasure; their acceptance remains a maintainer decision. Query-level latency was not measured, and the final single-pair screen does not supersede these repetitions.

Are there user-facing or API changes?

Logical CASE metadata is retained rather than silently discarded. ExprSchema gains a defaulted method without changing the meaning of existing custom nullability implementations. This does not change execution kernels or claim to fix arrow_metadata(CASE ...) without the physical follow-up.

@codecov-commenter

codecov-commenter commented Oct 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 75.40395% with 137 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.76%. Comparing base (4e6e564) to head (7cc7ac7).

Files with missing lines Patch % Lines
datafusion/expr/src/expr_schema.rs 73.93% 39 Missing and 89 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs 86.36% 2 Missing and 7 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26145      +/-   ##
==========================================
- Coverage   82.77%   82.76%   -0.02%     
==========================================
  Files        1148     1148              
  Lines      451075   451632     +557     
  Branches   451075   451632     +557     
==========================================
+ Hits       373397   373810     +413     
- Misses      54930    54976      +46     
- Partials    22748    22846      +98     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osipovartem
osipovartem marked this pull request as draft October 8, 2026 23:00
@osipovartem osipovartem changed the title fix: preserve common field metadata through CASE expressions fix: preserve field metadata for simple CASE branches Oct 8, 2026
@osipovartem
osipovartem marked this pull request as ready for review October 9, 2026 08:58
@osipovartem
osipovartem marked this pull request as draft October 9, 2026 12:37
@github-actions github-actions Bot added the core Core DataFusion crate label Oct 9, 2026
@osipovartem
osipovartem marked this pull request as ready for review October 15, 2026 21:45
@james-willis

Copy link
Copy Markdown
Contributor

PTAL at #24865

I originally had this strict metadata policy like you do here but changed it to be much more permissive on the basis of @paleolimbot 's comments.

@osipovartem

Copy link
Copy Markdown
Contributor Author

Thanks for pointing me to #24865. I agree CASE should use the same policy: merge unrelated metadata keys permissively, keep extension name/parameters paired, and reject conflicting extension names. The current strict full-map equality in this PR is inconsistent with that direction. I will align it with the shared helper and add CASE-specific tests for missing/unrelated metadata and conflicting extension names before this is merged.

@osipovartem
osipovartem marked this pull request as draft October 28, 2026 00:29
@github-actions github-actions Bot added the common Related to common crate label Oct 28, 2026
@paleolimbot

paleolimbot commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

While I don't necessarily mind the approach here, I think it needs some justification for why it's needed over the simpler fix. This is a good workaround for when you don't control the ScalarUDF source code, but the actual bug here is that return_field() for case when and friends are dropping metadata.

I put a note on the other PR as well, but I recently discovered intersect_metadata_for_union(). While I think it is also not perfect, it might be better to use it here to reduce the number of times we define this logic.

@osipovartem

Copy link
Copy Markdown
Contributor Author

Thanks. CASE is a separate logical drop site: direct Expr::Case::to_field does not call the COALESCE/NVL ScalarUDF::return_field_from_args() methods fixed in #24865. I agree the metadata merge policy should be centralized. intersect_metadata_for_union() is not equivalent to the reviewed coerced_fields_metadata in #24865: it intersects ordinary keys and can silently drop a conflicting extension name, whereas #24865 uses earlier-key precedence and rejects incompatible extension names. I will reuse the shared helper once #24865 lands, keep this PR draft until then, and revisit whether the CASE branch-resolution logic can be simplified without losing nested CASE/cast/alias behavior or the plain-CASE fast path.

@osipovartem
osipovartem force-pushed the upstream-case-field-metadata branch from ee2aa10 to d7aebcc Compare October 29, 2026 13:42

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some proposed tests:

/// Return the field and column nullability together.
///
/// The default preserves custom implementations of [`Self::nullable`].
fn field_and_nullable(&self, col: &Column) -> Result<(&FieldRef, bool)> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the field has a nullability column, right? Why can't this just use the existing nullability from Field?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DFSchema does use Field::is_nullable(). The default preserves custom ExprSchema::nullable() overrides, which may disagree with the field or return errors. The paired method avoids a second DFSchema lookup without changing that contract. Added a regression for both disagreement directions in e0753c3.

gafiatulin pushed a commit to gafiatulin/datafusion that referenced this pull request Oct 30, 2026
…#24865)

## Which issue does this PR close?

- Part of apache#24860.

## Rationale for this change

`COALESCE`, `NVL`/`IFNULL`, and `NVL2` drop the field metadata of their
arguments when computing their return field: `return_field_from_args`
builds the output `Field` from the argument *data types* only. For
columns carrying Arrow extension types (`ARROW:extension:name` /
`ARROW:extension:metadata` — e.g. `arrow.uuid`, `geoarrow.wkb`), the
planned schema of any expression containing these functions silently
loses the extension-type identity, and metadata-aware UDFs that dispatch
on their argument `Field`s fail to plan over them (see apache#24860 for a full
reproducer).

This is one of several independent metadata drop sites listed in apache#24860.
It is the plan-time schema fix for the conditional functions. These
functions simplify into `CASE`, whose logical and physical drop sites
are addressed separately in apache#26145 and apache#26158.

## What changes are included in this PR?

- `datafusion_common::metadata::coerced_fields_metadata`: computes the
metadata for an expression whose result may come from any one of several
inputs coerced to a common type. It lives next to the related metadata
utilities so that `CASE` (in `datafusion-expr`) can share it:
- Metadata is merged across the inputs whose data type is not `Null` (an
untyped NULL literal carries no metadata and cannot contribute a typed
value), with earlier inputs taking precedence for repeated keys.
Unrelated keys are not required to agree, so metadata that arrived with
the data (e.g. from an Arrow file) cannot cause an extension type to be
dropped.
- The extension type keys are handled as a unit, as in `Expr::Cast`
output field derivation: differing values of `ARROW:extension:name` are
a planning error, and `ARROW:extension:metadata` always comes from the
input that supplied the extension name (byte-wise differences in it are
tolerated, since extension parameters are often JSON).
- `coalesce`: `return_field_from_args` attaches the merged metadata of
its arguments. Return type and nullability are unchanged. `NVL`/`IFNULL`
delegate to `coalesce` and are fixed by the same change.
- `nvl2`: same, over the second and third arguments only — the first
argument is only tested for NULL and never contributes a value, so its
metadata is excluded.

## What is the testing strategy for this PR?

- Unit tests for `coerced_fields_metadata` in `metadata.rs`: merge
precedence, untyped NULL inputs skipped, empty result, error on
differing extension names, and extension name/metadata kept paired.
- Unit tests in `coalesce.rs` and `nvl2.rs` exercise
`return_field_from_args`: metadata propagated, untyped NULL arguments
not blocking propagation, conflicting extension types rejected, NVL2's
test argument excluded, and nullability unchanged.

There is deliberately no sqllogictest yet: `arrow_metadata()` reads its
argument field at execution time, and by then these functions have been
simplified into `CASE`, so end-to-end coverage depends on the `CASE`
fixes (apache#26145, apache#26158). This ordering is safe: the optimizer's schema
invariant ignores metadata (`assert_expected_schema` /
`logically_equivalent_names_and_types`), so a plan whose `coalesce`
carries metadata that the simplified `CASE` does not still plans.

## Are there any user-facing changes?

- The planned schema (e.g. `DataFrame::schema()`) of expressions
containing `COALESCE`/`NVL`/`IFNULL`/`NVL2` over metadata-bearing
columns now preserves that metadata.
- `COALESCE`/`NVL`/`IFNULL`/`NVL2` over arguments with *differing*
`ARROW:extension:name` values (e.g. an `arrow.uuid` column and an
`arrow.json` column) is now a planning error. Previously these planned
and silently dropped the extension type.
- New public function
`datafusion_common::metadata::coerced_fields_metadata`. No other API
changes.
@osipovartem
osipovartem force-pushed the upstream-case-field-metadata branch from d7aebcc to e0753c3 Compare October 30, 2026 15:47
@osipovartem
osipovartem marked this pull request as ready for review October 30, 2026 15:48
@osipovartem

Copy link
Copy Markdown
Contributor Author

@alamb Thanks for #26895. This now reuses the merged #24865 helper, with public SQL coverage of pre/post-optimization fields. The execution-level arrow_metadata cases need physical #26158; I will use those cases there rather than preserve the documented NULL failures.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @osipovartem

Comment thread datafusion/expr/src/expr_schema.rs Outdated
}
}

// Resolve metadata bottom-up for branches whose metadata can be preserved.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems quite complicated for resolving metadata -- can you explain why it needs a custom worklist?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bottom-up path avoids re-running type/nullability inference for every suffix of a nested CASE chain. I kept it for CASE, forwarding wrappers and binary expressions; other branches use native to_field without the depth cutoff. Added counted regressions and nested benchmarks; CASE/UDF re-inference remains a separate limitation.

assert_eq!(meta, outer_ref.metadata(&schema).unwrap());
}

#[test]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what cases does this cover that the .slt tests do not? Can we conslidate to slt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consolidated the SQL cases in metadata.slt in #26158, including #26895. The remaining Rust tests cover API-only annotations, pre-coercion type guards, custom ExprSchema behavior and inference-count regressions.

@osipovartem osipovartem changed the title fix: preserve field metadata for simple CASE branches fix: preserve CASE result field metadata Oct 30, 2026
@osipovartem
osipovartem requested a review from alamb November 2, 2026 22:43
@raghuvanshraj

raghuvanshraj commented Nov 4, 2026 •

Copy link
Copy Markdown

I tried this stack (#26145 + #26158) with a Parquet Variant column, since the slt tests here and in #26895 only cover extension metadata on Utf8. Variant is an extension type on a struct (arrow.parquet.variant on Struct<metadata, value, typed_value>), and datafusion-variant's variant_get refuses to run if that tag is missing from its input field.

On branch-56, variant_get over any CASE or COALESCE of the column fails with field does not have extension type VariantType. On this stack, the same queries return the right rows:

branch-56 this stack
variant_get(CASE WHEN event:status >= 500 THEN event ELSE event END, 'service') error api, api, db
variant_get(CASE WHEN event:status >= 500 THEN event END, 'service') error NULL, api, db
variant_get(COALESCE(event, event), 'service') error api, api, db
variant_get(CASE WHEN false THEN event END, 'service') error error (that one is #26951)

Would it be worth adding one struct-backed extension case to metadata.slt, so the nested-type path is covered too? Happy to send that as a follow-up if so.

@osipovartem

Copy link
Copy Markdown
Contributor Author

Yes, please. A struct-backed extension case in metadata.slt would be valuable. The execution-level coverage is in #26158, which uses this logical fix as well. Thanks for validating the stack.

@raghuvanshraj

Copy link
Copy Markdown

Yes, please. A struct-backed extension case in metadata.slt would be valuable. The execution-level coverage is in #26158, which uses this logical fix as well. Thanks for validating the stack.

@osipovartem I've opened the struct-backed cases against your branch: Embucket#186.

@osipovartem
osipovartem marked this pull request as draft November 14, 2026 08:50
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Nov 14, 2026
@osipovartem
osipovartem marked this pull request as ready for review November 14, 2026 23:53
bvolpato pushed a commit to bvolpato/datafusion that referenced this pull request Nov 21, 2026
## Which issue does this PR close?

- Closes apache#24860.

This is a smaller alternative to apache#26145 + apache#26158, following the
suggestion on apache#24865 to look for a simpler way to reach the same goal.
It also includes the simplifier fix from apache#26951, which depends on the
CASE fix here.

## Rationale for this change

`CASE` drops the field metadata of its branches, so Arrow extension
types are lost. That breaks functions that check the extension type of
their argument when the argument is a `CASE`, or a `COALESCE`, `NVL`,
`IFNULL` or `NVL2`, since those are rewritten to `CASE` before
execution. Examples are `variant_get` over a Parquet Variant column,
reported on apache#26145, and geometry and raster functions in SedonaDB.

The metadata is dropped in the logical `Expr::Case` field, in the
physical `CaseExpr`, which has no `return_field` override, and in
simplifier rewrites that drop the branch carrying it.

## What changes are included in this PR?

- **Logical `Expr::Case::to_field`** merges the fields of its THEN and
ELSE branches with `coerced_fields_metadata`, the helper
`COALESCE`/`NVL2` use since apache#24865. It takes the type and nullability
from the same fields, so it reads each branch once. The type rule from
apache#27139 moves into a helper that `get_type` also uses.
- **Physical `CaseExpr::return_field`** does the same with its branches'
`return_field`.
- **A typed NULL ELSE that carries metadata** is no longer dropped by
`CaseExpr::try_new`. It is still evaluated like no ELSE.
- **CASE simplifications keep the metadata.** A literal left by a
constant WHEN, such as the NULL from `CASE WHEN false THEN x END`, takes
the CASE's metadata. A CASE is not simplified when that would change its
metadata, and a Boolean CASE with metadata on a branch is not rewritten
to AND/OR.

So CASE follows the same rules as `COALESCE`: the earlier branch wins
for each key, an extension name is kept together with its parameters,
differing extension names are a planning error, and a branch without
metadata or an untyped `NULL` does not block another branch's metadata.
The operand of a simple `CASE` and the WHEN conditions do not
contribute.

<details>
<summary>Differences from apache#26145 + apache#26158</summary>

About 290 lines of non-test code, against about 700 there. That includes
the simplifier changes and about 70 lines that move the existing CASE
type and nullability logic into helpers.

- **No stored copy of the logical field, and no protobuf change.** The
physical CASE derives its field from its physical branches, so it
matches the logical field whenever each branch's field does. A unit test
checks that for column, literal, cast and nested CASE branches. apache#26158
instead carries the logical field into the physical CASE.
- **A branch whose logical and physical fields differ therefore shows
through the CASE.** Physical `NOT` passes its input's metadata through
while logical `NOT` drops it (apache#27155), and an alias's metadata can be
lost when the physical expression is created (apache#27156). With `NOT`, this
can be a planning error that `main` doesn't raise, when the other branch
has a different extension name. Those mismatches are in `NOT` and
`Alias` and are not addressed here.

</details>

<details>
<summary>Performance</summary>

Local runs on a laptop against `main`, so treat them as indicative:
Criterion runs of the benchmarks from apache#26145/apache#26158, and `EXPLAIN`
timings with a debug build of `datafusion-cli` for planning.

- **CASE evaluation and physical planning** are unchanged.
- **`return_field` rebuilds the merged metadata on every call:** 954 ns
vs 274 ns for a column CASE with 4 metadata keys. A function asks its
CASE argument for its field once per batch; field plus evaluation for
8,192 rows took 13.6 µs vs 13.1 µs.
- **Logical `to_field`** for a wide CASE without any metadata took 2–3x
as long.

Both were measured before the nesting changes below, which only remove
work.
- **Nested CASE.** `to_field`, `return_field` and the constant-WHEN
simplification visit each branch once. I timed 18 nesting shapes up to
22 levels: the CASE in the THEN, the ELSE or a WHEN; nested directly, in
a function call, in `||` or under a cast; with a nullable or
non-nullable other branch; a simple CASE; and COALESCE nested in either
argument. The plans are the same as on `main`.
  - 10 take the same time as on `main`.
- 7 are exponential on `main` and linear here. With the CASE in a
function call, 6 shapes take 16–22 s on `main` at 18 levels and 0.05 s
here. With a cast between levels (found by @jayzhan211), `main` takes
13.2 s at 16 levels and this PR 0.06 s.
- COALESCE nested in its first argument is exponential on both, because
the simplifier rewrites it to a CASE that repeats that argument: 4.05 s
vs 3.82 s at 18 levels.

Fast paths can come later: reuse the metadata when only one branch has
any, and return early when none has.

</details>

## What is the testing strategy for this PR?

- **New `metadata.slt` cases** check the metadata at execution with
`arrow_metadata`, mostly with the built-in `JSON` extension type. They
cover precedence, simple CASE, conflicting extension names, a branch
without metadata, no ELSE, an untyped and a typed NULL ELSE, coercion to
a common type, nested CASE, the CASE rewrites of `coalesce`, `nvl`,
`ifnull` and `nvl2`, constant-WHEN simplification, and a Boolean CASE.
Each check fails without this change. Each query uses a given CASE only
once, because a repeated expression is computed in a lower projection
whose field comes from the logical plan.
- **Unit tests** check that the physical CASE field matches the logical
one in metadata, type and nullability, and cover the CASE
simplifications. That includes the rewrite of a Boolean CASE to
`NOT(CASE ...)`, which SQL can't observe because physical `NOT` passes
the metadata through (apache#27155).
- **Existing tests:** the `datafusion-expr`, `datafusion-physical-expr`
and `datafusion-optimizer` lib tests pass. The full sqllogictest suite
and the `core_integration` `sql` tests show no new failures; my checkout
doesn't have the test-data submodules, so the files that need that data
fail the same way on `main`.
- **The `metadata.slt` from apache#26158**, which includes the cases from
apache#26895, also passes with this change, checked locally with the current
head of apache#26158.

## Are there any user-facing changes?

- `CASE` results keep their field metadata, including extension types,
also when the optimizer simplifies the CASE. So do `COALESCE`, `NVL`,
`IFNULL` and `NVL2` at execution.
- `CASE` branches with differing extension names are now a planning
error, as they already are for `COALESCE`.

No public API changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate logical-expr Logical plan and expressions optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants