Skip to content

fix: propagate field metadata through coalesce, nvl, and nvl2 - #24865

Merged
alamb merged 10 commits into
apache:mainfrom
james-willis:coalesce-metadata
Oct 30, 2026
Merged

alamb merged 10 commits into
apache:mainfrom
james-willis:coalesce-metadata

Conversation

@james-willis

@james-willis james-willis commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

COALESCE, NVL/IFNULL, and NVL2 drop the field metadata of their arguments when computing their return field: return_field_from_args builds the output Field from the argument data types only. For columns carrying Arrow extension types (ARROW:extension:name / ARROW:extension:metadata — e.g. arrow.uuid, geoarrow.wkb), the planned schema of any expression containing these functions silently loses the extension-type identity, and metadata-aware UDFs that dispatch on their argument Fields fail to plan over them (see #24860 for a full reproducer).

This is one of several independent metadata drop sites listed in #24860. It is the plan-time schema fix for the conditional functions. These functions simplify into CASE, whose logical and physical drop sites are addressed separately in #26145 and #26158.

What changes are included in this PR?

  • datafusion_common::metadata::coerced_fields_metadata: computes the metadata for an expression whose result may come from any one of several inputs coerced to a common type. It lives next to the related metadata utilities so that CASE (in datafusion-expr) can share it:
    • Metadata is merged across the inputs whose data type is not Null (an untyped NULL literal carries no metadata and cannot contribute a typed value), with earlier inputs taking precedence for repeated keys. Unrelated keys are not required to agree, so metadata that arrived with the data (e.g. from an Arrow file) cannot cause an extension type to be dropped.
    • The extension type keys are handled as a unit, as in Expr::Cast output field derivation: differing values of ARROW:extension:name are a planning error, and ARROW:extension:metadata always comes from the input that supplied the extension name (byte-wise differences in it are tolerated, since extension parameters are often JSON).
  • coalesce: return_field_from_args attaches the merged metadata of its arguments. Return type and nullability are unchanged. NVL/IFNULL delegate to coalesce and are fixed by the same change.
  • nvl2: same, over the second and third arguments only — the first argument is only tested for NULL and never contributes a value, so its metadata is excluded.

What is the testing strategy for this PR?

  • Unit tests for coerced_fields_metadata in metadata.rs: merge precedence, untyped NULL inputs skipped, empty result, error on differing extension names, and extension name/metadata kept paired.
  • Unit tests in coalesce.rs and nvl2.rs exercise return_field_from_args: metadata propagated, untyped NULL arguments not blocking propagation, conflicting extension types rejected, NVL2's test argument excluded, and nullability unchanged.

There is deliberately no sqllogictest yet: arrow_metadata() reads its argument field at execution time, and by then these functions have been simplified into CASE, so end-to-end coverage depends on the CASE fixes (#26145, #26158). This ordering is safe: the optimizer's schema invariant ignores metadata (assert_expected_schema / logically_equivalent_names_and_types), so a plan whose coalesce carries metadata that the simplified CASE does not still plans.

Are there any user-facing changes?

  • The planned schema (e.g. DataFrame::schema()) of expressions containing COALESCE/NVL/IFNULL/NVL2 over metadata-bearing columns now preserves that metadata.
  • COALESCE/NVL/IFNULL/NVL2 over arguments with differing ARROW:extension:name values (e.g. an arrow.uuid column and an arrow.json column) is now a planning error. Previously these planned and silently dropped the extension type.
  • New public function datafusion_common::metadata::coerced_fields_metadata. No other API changes.

@github-actions github-actions Bot added the functions Changes to functions implementation label Sep 1, 2026
@james-willis

james-willis commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor Author

In this PR, if the branches of the coalesce, etc don't agree on the metadata, we drop extension metadata defensively.

My opinion is that this should actually be a planning time error unless all rows will hit only one branch but wanted to take the more conservative approach for now. I want to solicit opinions from the reviewers on what the desried behavior is here.

These functions built their return Field from argument data types only,
dropping field metadata (e.g. Arrow extension types) from the planned
schema. Attach the metadata shared by every value-contributing argument
instead. Part of apache#24860.
@codecov-commenter

codecov-commenter commented Sep 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.70130% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.58%. Comparing base (fb42cf5) to head (e76a9b8).
⚠️ Report is 40 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/common/src/metadata.rs 97.00% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24865      +/-   ##
==========================================
+ Coverage   82.49%   82.58%   +0.09%     
==========================================
  Files        1140     1142       +2     
  Lines      439034   440994    +1960     
  Branches   439034   440994    +1960     
==========================================
+ Hits       362165   364182    +2017     
+ Misses      54962    54813     -149     
- Partials    21907    21999      +92     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@james-willis
james-willis marked this pull request as ready for review September 1, 2026 21:59

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this fix! I'm not a committer so I can't help with the CI but I'll review from a metadata perspective.

I think the family of functions that are affected here are ones that have >1 "compatible" (castable to a common type) inputs. While there are some extension aware type systems that can handle this (notably: vctrs via vec_ptype2() and vec_cast()), datafusion has no mechanism for that and so probably the best we can do for now is to propagate agressively and rely on engines to write analyzer or optimizer rules to enforce other behaviour.

One other place I've seen this pattern is combining values in a values literal clause. I would personally love if the logic were the same there but that might insert too much scope here.

let metadata = value.metadata(&schema)?;
if let Some(ref cm) = common_metadata {
if &metadata != cm {
return plan_err!(
"Inconsistent metadata across values list at row {i} column {j}. Was {:?} but found {:?}",
cm,
metadata
);
}
} else {
common_metadata = Some(metadata.clone());
}

I am curious what @timsaucer would expect to happen to metadata in CASE WHEN since that is a metadata usage I'm less familiar with!

Comment thread datafusion/functions/src/utils.rs Resolved
…on-common

Requiring the arguments to agree on metadata was too strict: extension
types with JSON-encoded parameters can differ byte-wise, and unrelated
metadata that arrived with the data should not drop an extension type
from the result. Propagate the first non-empty metadata among the
value-contributing arguments instead, and move the helper next to the
related utilities in datafusion_common::metadata so the upcoming CASE
fix in datafusion-expr can reuse it.
@github-actions github-actions Bot added the common Related to common crate label Sep 25, 2026
@james-willis
james-willis marked this pull request as draft September 25, 2026 00:21

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know you're still working here, just making sure I didn't forget to review!

Comment thread datafusion/common/src/metadata.rs Resolved
Comment thread datafusion/common/src/metadata.rs Resolved
Comment thread datafusion/common/src/metadata.rs Resolved
Adopt the suggested compromise: merge metadata across the
value-contributing arguments (earlier arguments win for repeated keys)
instead of taking only the first non-empty metadata, but refuse
differing values of 'ARROW:extension:name' with a planning error,
keeping the extension name and its parameters paired. Also accept
impl IntoIterator and inline the field construction in the
datafusion-common tests.
@paleolimbot

Copy link
Copy Markdown
Member

@james-willis Can you address the merge conflicts? I can bring up this PR in the weekly sync tomorrow to get a second review if it's ready.

james-willis and others added 3 commits October 27, 2026 10:53
# Conflicts:
#	datafusion/functions/src/core/nvl2.rs
Drop the field/ext_field/return_field test helpers in favor of
Field::new().with_metadata() and direct return_field_from_args calls,
matching the datafusion-common tests in this PR and the existing tests
in with_metadata.rs.
@james-willis
james-willis marked this pull request as ready for review October 27, 2026 18:20

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

This would benefit from a second review to double check my take on metadata merging here, but in general I think this a standalone improvement and fixes a hard to work around bug.

Comment on lines +143 to +144
/// [`Expr::Cast`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.Expr.html#variant.Cast
pub fn coerced_fields_metadata<'a>(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I recently discovered intersect_metadata_for_union(), which silently discards non-matching metadata keys. I like this version better (I want a louder error when the types don't match), but either is OK with me if we want to keep the existing logic for consistency.

@james-willis james-willis Oct 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't totally understand the UNION plumbing but intersect_metadata_for_union is only used for the first stage and is not what UNION will ultimately give for the query result's type. The ultimate behavior of UNION is also not great here.

I'm inclined to keep this PR as-is because I think probably the different branches need to agree at least on the extension type and it is important to enforce that. Even better we do it at plan time.

I would like to be more strict with the other metadata because often that is part of the type (e.g., the CRS is part of the type of a geometry, randomly picking one of the CRSes will give incorrect query results) but I'm not confident in that being universally the right call to enforce it here (e.g., in the CRS case we would want to convert to item-level CRS when branches diverge).

Comment on lines +90 to +97
// Propagate field metadata (e.g. Arrow extension types) from the
// arguments that can supply a value; see `coerced_fields_metadata`
// for the merge semantics.
let metadata =
coerced_fields_metadata(args.arg_fields.iter().map(|f| f.as_ref()))?;
Ok(metadata
.add_to_field(Field::new(self.name(), return_type, nullable))
.into())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to hear another opinion, but I do think this is a bug worth fixing regardless of whether #26145 is a good idea.

@mprammer mprammer left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Chiming in to say that I ran into the same problem. @paleolimbot's suggested way of merging metadata seems right to me.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this PR @james-willis and for the reviews @paleolimbot and @mprammer

I think we need "end to end" tests of this -- namely to verify that in an actual query the metadata is preserved.

I am pretty sure this PR won't actually fix the problem as it nvl etc are rewritten macro-style to CASE which loses the metadata. In fact I wrote up some tests in a separate PR showing this (this PR doesn't fix the problem):

Here is the kind of tests I think are needed

With those tests I don't think there is any need for the rust unit tests in this PR

coalesce, nvl, ifnull and nvl2 are rewritten to CASE before execution, so
arrow_metadata() cannot observe the field they compute during planning.

- metadata.slt: conflicting extension types are a planning error, and
  nested cases observe the planning-time field through that error (an
  outer coalesce only reports the inner extension type if it was
  propagated).
- core/tests/sql/select.rs: the planned schema (DataFrame::schema())
  carries merged non-extension metadata for each function.
- Remove the coalesce.rs and nvl2.rs unit tests these replace. The
  coerced_fields_metadata unit tests remain.
@github-actions github-actions Bot added core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Oct 29, 2026
@james-willis

Copy link
Copy Markdown
Contributor Author

Thanks for taking a look @alamb.

I agree that this PR is not sufficient to fix the metadata dropping issues in these functions, but it is necessary: COALESCE’s (et al's) output field is computed at initial planning, before that rewrite, and a metadata-aware UDF over COALESCE checks its argument there. Without this change the query fails at that point even once CASE is fixed.

Since this PR fixes something only observable at plan time, I added a test in select.rs to capture that planning went as expected. The style in your linked PR wouldn't let me capture that.

@alamb

alamb commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Since this PR fixes something only observable at plan time, I added a test in select.rs to capture that planning went as expected. The style in your linked PR wouldn't let me capture that.

Thanks -- can you be clear what your usecase is? In other words, how are you hitting the code that this PR solves? Is it like a third-party frontend ?

Why isn't the simplifier being run in your usecase?

@paleolimbot

Copy link
Copy Markdown
Member

Why isn't the simplifier being run in your usecase?

It's being run, but the return type is also being computed before it runs (e.g., asking for the schema of the unoptimized logical plan). For something like st_buffer(coalesce(geom, other_geom)), we get an execution error "st_buffer: no kernel matching arguments" because our extension type aware version of the Signature rejects an unqualified Binary; however, I believe this would also show up as an unexpected/mismatched schema result between an optimized an unoptimized plan.

Agreed that we need to fix more to get this to work end to end, but we do also need to fix the return_field().

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it -- makes sense to me. Thank you

@alamb
alamb added this pull request to the merge queue Oct 30, 2026
Merged via the queue into apache:main with commit be44fab Oct 30, 2026
44 checks passed
@james-willis

Copy link
Copy Markdown
Contributor Author

My intent is to work toward end to end metadata conservation in these branching functions. I just started small with this PR as a new contributor.

@alamb

alamb commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Thank you @james-willis

You have probably seen #26158 from @osipovartem -- perhaps you can help me review / take a look at that and figure out if we can find a simpler way to accomplish the same aims

AdamGS pushed a commit to ianthetechie/datafusion that referenced this pull request Nov 18, 2026
… CASE (apache#26895)

## Which issue does this PR close?

- Related to apache#24860 
- Related to apache#24865

## Rationale for this change

While reviewing apache#24865 I found that `coalesce`, `nvl`, `ifnull` and
`nvl2` are rewritten to `CASE` during simplification, and `CASE` drops
field metadata

This wasn't caught because there was no end user API coverage

These tests document the current behavior end to end so a fix can be
verified against them.

## What changes are included in this PR?

Add New cases in `metadata.slt` for `coalesce`, `nvl`, `ifnull`, `nvl2`
and `CASE` that document existing behavior with some comments explaining
what the expected results are

## What is the testing strategy for this PR?

Tests only.

## Are there any user-facing changes?

No.

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
bvolpato pushed a commit to bvolpato/datafusion that referenced this pull request Nov 21, 2026
## Which issue does this PR close?

- Closes apache#24860.

This is a smaller alternative to apache#26145 + apache#26158, following the
suggestion on apache#24865 to look for a simpler way to reach the same goal.
It also includes the simplifier fix from apache#26951, which depends on the
CASE fix here.

## Rationale for this change

`CASE` drops the field metadata of its branches, so Arrow extension
types are lost. That breaks functions that check the extension type of
their argument when the argument is a `CASE`, or a `COALESCE`, `NVL`,
`IFNULL` or `NVL2`, since those are rewritten to `CASE` before
execution. Examples are `variant_get` over a Parquet Variant column,
reported on apache#26145, and geometry and raster functions in SedonaDB.

The metadata is dropped in the logical `Expr::Case` field, in the
physical `CaseExpr`, which has no `return_field` override, and in
simplifier rewrites that drop the branch carrying it.

## What changes are included in this PR?

- **Logical `Expr::Case::to_field`** merges the fields of its THEN and
ELSE branches with `coerced_fields_metadata`, the helper
`COALESCE`/`NVL2` use since apache#24865. It takes the type and nullability
from the same fields, so it reads each branch once. The type rule from
apache#27139 moves into a helper that `get_type` also uses.
- **Physical `CaseExpr::return_field`** does the same with its branches'
`return_field`.
- **A typed NULL ELSE that carries metadata** is no longer dropped by
`CaseExpr::try_new`. It is still evaluated like no ELSE.
- **CASE simplifications keep the metadata.** A literal left by a
constant WHEN, such as the NULL from `CASE WHEN false THEN x END`, takes
the CASE's metadata. A CASE is not simplified when that would change its
metadata, and a Boolean CASE with metadata on a branch is not rewritten
to AND/OR.

So CASE follows the same rules as `COALESCE`: the earlier branch wins
for each key, an extension name is kept together with its parameters,
differing extension names are a planning error, and a branch without
metadata or an untyped `NULL` does not block another branch's metadata.
The operand of a simple `CASE` and the WHEN conditions do not
contribute.

<details>
<summary>Differences from apache#26145 + apache#26158</summary>

About 290 lines of non-test code, against about 700 there. That includes
the simplifier changes and about 70 lines that move the existing CASE
type and nullability logic into helpers.

- **No stored copy of the logical field, and no protobuf change.** The
physical CASE derives its field from its physical branches, so it
matches the logical field whenever each branch's field does. A unit test
checks that for column, literal, cast and nested CASE branches. apache#26158
instead carries the logical field into the physical CASE.
- **A branch whose logical and physical fields differ therefore shows
through the CASE.** Physical `NOT` passes its input's metadata through
while logical `NOT` drops it (apache#27155), and an alias's metadata can be
lost when the physical expression is created (apache#27156). With `NOT`, this
can be a planning error that `main` doesn't raise, when the other branch
has a different extension name. Those mismatches are in `NOT` and
`Alias` and are not addressed here.

</details>

<details>
<summary>Performance</summary>

Local runs on a laptop against `main`, so treat them as indicative:
Criterion runs of the benchmarks from apache#26145/apache#26158, and `EXPLAIN`
timings with a debug build of `datafusion-cli` for planning.

- **CASE evaluation and physical planning** are unchanged.
- **`return_field` rebuilds the merged metadata on every call:** 954 ns
vs 274 ns for a column CASE with 4 metadata keys. A function asks its
CASE argument for its field once per batch; field plus evaluation for
8,192 rows took 13.6 µs vs 13.1 µs.
- **Logical `to_field`** for a wide CASE without any metadata took 2–3x
as long.

Both were measured before the nesting changes below, which only remove
work.
- **Nested CASE.** `to_field`, `return_field` and the constant-WHEN
simplification visit each branch once. I timed 18 nesting shapes up to
22 levels: the CASE in the THEN, the ELSE or a WHEN; nested directly, in
a function call, in `||` or under a cast; with a nullable or
non-nullable other branch; a simple CASE; and COALESCE nested in either
argument. The plans are the same as on `main`.
  - 10 take the same time as on `main`.
- 7 are exponential on `main` and linear here. With the CASE in a
function call, 6 shapes take 16–22 s on `main` at 18 levels and 0.05 s
here. With a cast between levels (found by @jayzhan211), `main` takes
13.2 s at 16 levels and this PR 0.06 s.
- COALESCE nested in its first argument is exponential on both, because
the simplifier rewrites it to a CASE that repeats that argument: 4.05 s
vs 3.82 s at 18 levels.

Fast paths can come later: reuse the metadata when only one branch has
any, and return early when none has.

</details>

## What is the testing strategy for this PR?

- **New `metadata.slt` cases** check the metadata at execution with
`arrow_metadata`, mostly with the built-in `JSON` extension type. They
cover precedence, simple CASE, conflicting extension names, a branch
without metadata, no ELSE, an untyped and a typed NULL ELSE, coercion to
a common type, nested CASE, the CASE rewrites of `coalesce`, `nvl`,
`ifnull` and `nvl2`, constant-WHEN simplification, and a Boolean CASE.
Each check fails without this change. Each query uses a given CASE only
once, because a repeated expression is computed in a lower projection
whose field comes from the logical plan.
- **Unit tests** check that the physical CASE field matches the logical
one in metadata, type and nullability, and cover the CASE
simplifications. That includes the rewrite of a Boolean CASE to
`NOT(CASE ...)`, which SQL can't observe because physical `NOT` passes
the metadata through (apache#27155).
- **Existing tests:** the `datafusion-expr`, `datafusion-physical-expr`
and `datafusion-optimizer` lib tests pass. The full sqllogictest suite
and the `core_integration` `sql` tests show no new failures; my checkout
doesn't have the test-data submodules, so the files that need that data
fail the same way on `main`.
- **The `metadata.slt` from apache#26158**, which includes the cases from
apache#26895, also passes with this change, checked locally with the current
head of apache#26158.

## Are there any user-facing changes?

- `CASE` results keep their field metadata, including extension types,
also when the optimizer simplifies the CASE. So do `COALESCE`, `NVL`,
`IFNULL` and `NVL2` at execution.
- `CASE` branches with differing extension names are now a planning
error, as they already are for `COALESCE`.

No public API changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate core Core DataFusion crate functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants