[BUG] SQL date_format function returns null for valid inputs on 2.19.0 #1

Activity

1. RyanL1997

   added

   bugSomething isn't workingSomething isn't working

   on May 4, 2026
2. github-actions

   added

   untriaged

   on May 4, 2026
3. RyanL1997

   removed

   untriaged

   on May 4, 2026
4. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
5. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
6. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
7. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
8. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
9. Repository owner deleted a comment from ryan-gh-bot on May 4, 2026
10. RyanL1997 commented on May 4, 2026

    

    RyanL1997

    on May 4, 2026

    OwnerAuthor

    More actions

    @ryan-gh-bot /rca
11. ryan-gh-bot commented on May 4, 2026

    

    ryan-gh-bot

    on May 4, 2026

    More actions

    [Working] on /rca (invoked by @RyanL1997). I'll post the result when done. This usually takes 30 seconds to 5 minutes depending on the command.
12. ryan-gh-bot commented on May 4, 2026

    

    ryan-gh-bot

    on May 4, 2026

    More actions

    [RCA] date_format returns null because the sample accounts dataset has no birthdate field, not a regression in the function.

    An all-null output column from date_format(birthdate, 'yyyy-MM') with no error or stack trace is consistent with nullMissingHandling short-circuiting on a missing/null input on every row. Code-reading across 2.17 and 2.19 shows date_format() is byte-identical between the two branches — there is no regression in the function — and the bundled accounts dataset used as the default sample has no birthdate field.

    Environment

    • Tenant: sql (opensearch-project/sql)
    • Refs checked:
      • upstream/main @ d3bdca82a
      • upstream/2.19 @ 5b5dfee4d
      • upstream/2.17 @ 43738d1aa

    Suspected location

    • core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunctions.java L1106-L1124 — date_format registration; every signature routes through DateTimeFormatterUtil::getFormattedDate wrapped by nullMissingHandling.
    • core/src/main/java/org/opensearch/sql/expression/function/FunctionDSL.java L400-L410 — binary nullMissingHandling: returns missingValue() / nullValue() before invoking the implementation when either input is missing/null.
    • Reporter's sample dataset: integ-test/src/test/resources/accounts.json — has no birthdate field on any document.

    Analysis

    All date_format overloads are wrapped with nullMissingHandling. That wrapper returns null/missing for any row where either input (the date expression or the format string) is null or missing, before the formatting code runs. An invalid format string would instead raise an error, and a type mismatch would fail function resolution — neither matches the reported "null for every row, no error." The only code path that produces silent null-per-row output is a null/missing input value on every row.

    The bundled accounts dataset that ships with this repo does not define a birthdate field. Indexing it as-is produces documents with fields account_number, balance, firstname, lastname, age, gender, address, employer, email, city, state only. Querying birthdate against such an index resolves to missing on every document, which is exactly the input condition that makes nullMissingHandling emit null for every row.

    The "worked in 2.17, broken in 2.19" framing is not supported by the code: between upstream/2.17 @ 43738d1aa and upstream/2.19 @ 5b5dfee4d, the date_format() resolver definition is byte-for-byte identical. The only commit between those refs touching DateTimeFunction(s).java is a str_to_date two-digit year fix, which does not affect date_format.

    Evidence

    • Function registration and null-handling wrapper (read on upstream/main @ d3bdca82a):

      // DateTimeFunctions.java L1106-L1124
      private DefaultFunctionResolver date_format() {
        return define(
            BuiltinFunctionName.DATE_FORMAT.getName(),
            impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate), STRING, STRING, STRING),
            impl(nullMissingHandling(DateTimeFormatterUtil::getFormattedDate), STRING, DATE, STRING),
            // ... TIME and TIMESTAMP overloads, all wrapped with nullMissingHandling
        );
      }

      // FunctionDSL.java L400-L410 (binary nullMissingHandling)
      return (v1, v2) -> {
        if (v1.isMissing() || v2.isMissing()) return ExprValueUtils.missingValue();
        else if (v1.isNull() || v2.isNull())  return ExprValueUtils.nullValue();
        else return function.apply(v1, v2);
      };

    • Regression check — date_format() is unchanged between 2.17 and 2.19:

      $ git log --oneline upstream/2.17..upstream/2.19 -- \
          core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFunction.java \
          core/src/main/java/org/opensearch/sql/expression/datetime/DateTimeFormatterUtil.java
      f412de04c [Backport 2.x] Function `str_to_date` should work with two-digits year (#3227)

      Inspecting f412de04c shows it modifies DateTimeFormatterUtil.java for str_to_date only; date_format() resolver bodies on both branches are identical at line 1253 of DateTimeFunction.java.

    • Sample data inspection — no birthdate in accounts.json:

      $ head -2 integ-test/src/test/resources/accounts.json
      {"index":{"_id":"1"}}
      {"account_number":1,"balance":39225,"firstname":"Amber","lastname":"Duke","age":32,"gender":"M","address":"880 Holmes Lane","employer":"Pyrami","email":"amberduke@pyrami.com","city":"Brogan","state":"IL"}
      $ grep -c 'birthdate' integ-test/src/test/resources/accounts.json
      0

    • Live verification: not performed. Two converging code signals (no regression in date_format between 2.17 and 2.19; nullMissingHandling is the only all-null-per-row path) plus confirmation that the canonical sample dataset lacks birthdate make running a local cluster unnecessary.

    Confidence: High
    Three independent signals agree: (1) the date_format resolver is byte-identical between upstream/2.17 and upstream/2.19 — no regression is possible in this function; (2) nullMissingHandling is the only code path in the function that can produce null-per-row with no error; (3) the bundled accounts dataset used in the reproducer has no birthdate field, which exactly triggers that path.

    Next steps

    • Confirm the index mapping and the actual documents in the reporter's cluster:

      $ curl -s 'localhost:9200/accounts/_mapping' | jq .
      $ curl -s 'localhost:9200/accounts/_search?size=1' | jq '.hits.hits[0]._source | keys'

      If birthdate is absent or present-but-null on every document, that confirms the hypothesis. If birthdate exists and is populated with a date type and date_format still returns null, this is a different issue and should be re-opened with the mapping and a sample document.
    • Unverified suggestion: if the user's index is meant to have birthdate but does not, either re-index with the field or point the query at a dataset that has it (e.g., a custom index, not the canonical accounts fixture).
    • Unverified suggestion: if the user was following a blog/tutorial that assumed birthdate existed in the sample dataset, the tutorial is out of date — accounts.json in this repo has never included birthdate in the checked-out range.
    • If the mapping confirms birthdate is a populated date field and the query still returns all null, escalate with /reproduce so the behavior can be verified against a seeded index locally.

    ---

    Automated response from @ryan-gh-bot (maintainer-triggered agent). About this agent.