Repository navigation
[core] Add Multivalue global index for array membership - #9282
Merged
Merged
Conversation
JingsongLi
force-pushed
the
codex/multivalue-index
branch
from
August 18, 2026 09:28
2f7a6a6 to
2fc9489
Compare
JingsongLi
marked this pull request as ready for review
August 18, 2026 09:55
leaves12138
approved these changes
Aug 18, 2026
leaves12138
left a comment
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Reviewed the Multivalue integration across Core, primary-key maintenance, and the Flink/Spark spill-sort build paths. Source-row coverage remains correct for null and empty arrays, row-id localization is preserved, and the focused Core, Flink, and Spark tests pass locally. LGTM.
5 of 11 tasks
suxiaogang223
added a commit
to apache/doris
that referenced
this pull request
Oct 23, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable release published on October 12, 2026 ([upstream release](https://github.com/apache/paimon/releases/tag/release-2.1.0)). Pin the FE connector and BE JNI scanner to the same shared `paimon.version`, since tables, splits, and commit messages cross the FE/BE boundary using Paimon's serialization. Adapt the connector to the removal of Paimon's file-based `Privilege*` API in 2.1.0, removing its obsolete catalog/table wrappers and reflection-based delegate reconstruction. Preserve catalog query authorization, fallback/chain branch semantics, statement-bound schemas and snapshots, and Doris metadata-cache governance. Update local test fixtures to `FileSystemSchemaManager` and the extended `Snapshot` constructor. Give scanner tests the `java.nio` access already enabled in the BE JVM for Arrow Variant buffer views. Update the oversized-write negative test to the SDK's revised diagnostic while retaining its snapshot/row atomicity checks. Retain the branch/scan API adaptations and grouped format-file split fixture required since 2.0. Verify both the default wrapping and explicitly enabled checked integer-overflow modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast. Keep the existing Rust reader eligibility rules and common/S3 compatibility patches. Fix changelog MERGE target-presence detection for nullable primary keys: matched-only plans use their INNER JOIN semantics, while plans with INSERT branches use a pre-join marker. Keep that private marker outside user expression resolution. Preserve catalog query authorization for the new `$file_indexes` system table and pin both serialized table payloads to the catalog-published snapshot before stripping their catalog loaders. Adapt timestamp precision repair to the SDK's authorization projection validation and four-argument policy callback; use real JSON policy definitions in serialized scanner fixtures. Reject `scan.mode=latest-delta` in Doris `@options`, because the existing full-snapshot pinning would silently widen a delta scan; explicit `@incr` ranges remain available. The FE and BE Java extensions must be deployed with the same SDK version. This change does not establish compatibility between mixed 1.4.2/2.1.0 SDKs during a rolling upgrade. ### Paimon 2.1 feature impact on Doris | Upstream change | Doris impact and scope of this PR | | --- | --- | | [Nullable primary keys](apache/paimon#9094), opt-in with `primary-key.nullable=true` | Existing write-column mapping respects the SDK field nullability. Added a real-table test of nullable key metadata, FE/BE table serialization, and NULL-key deduplication across commits through the Java reader. The cluster regression additionally checks NULL-key deduplication across commits through JNI, and checks a single-file table through both JNI and the actual Rust Reader (verified from the completed query profile). Multi-file primary-key merges retain the existing JNI fallback. Add Doris SQL coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite keys, ordinary equality, INSERT branches, conditional priority, duplicate-match atomicity and private-marker name collisions. | | [`$file_indexes`](apache/paimon#9138) | Automatically discovered through the SDK registry. Preserve deferred catalog query checks and pin both backend payloads. FE scan-properties and actual JNI reconstruction tests cover denied authorization and snapshot 2 on disk while the catalog publishes only snapshot 1. | | [`latest-delta` batch startup mode](apache/paimon#9142) | Not exposed by this upgrade: reject it explicitly in `@options` instead of silently converting it to a full-snapshot read. A real three-snapshot test proves the difference (one delta row versus three full-snapshot rows). | | [Vector indexes](apache/paimon#8930) and [full-text indexes](apache/paimon#9031) on DV tables; [Multivalue array indexes](apache/paimon#9282); BTree TopN | New query acceleration needs Doris expression/search/TopN integration and relevant runtime dependencies and validation. Bumping the SDK does not enable these SQL capabilities. Existing reader selection remains conservative. | | [Arrow bundle writes](apache/paimon#9100) | An optional batching API, not a new `file.format=arrow` storage format. Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle API is a separate performance change requiring memory/lifecycle validation and benchmarks. | | [SchemaManager interface](apache/paimon#9174) and [REST schema history APIs](apache/paimon#9673) | Updated fixtures to `FileSystemSchemaManager`. The REST additions inspect schema history; they do not replace the SDK table's filesystem schema manager. Doris does not call the new display APIs in this PR. | | Manifest/index metadata and Variant handling | Both Java endpoints use SDK 2.1.0; package and serialization tests passed. The scanner tests now have the same `java.nio` access as the BE JVM. The targeted Java, native and cluster checks below cover existing supported paths, but do not establish native/Rust compatibility for every newly introduced table option or file layout. | | BLOB partial updates and first-row/data-evolution improvements | Available upstream; the upgrade preserves Doris' existing write and reader eligibility constraints. New combinations still need external-table regression coverage before being advertised. | | [Removal of file-based privileges](apache/paimon#9186), arithmetic/cast fixes, raw physical-metadata restrictions | Actual behavior changes, described in the release note below; legacy privilege users must migrate before upgrading. | ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE matched UPDATE/DELETE silently omitting rows with NULL primary keys. Paimon 2.1 keeps wrapping integer `sum`/`product` overflow by default and supports per-field `fields.<field>.sum.fail-on-overflow` / `fields.<field>.product.fail-on-overflow` options for checked arithmetic. Timestamp-to-date casts before the Unix epoch now use the preceding calendar date. The SDK rejects raw `$files`, `$file_key_ranges`, `$binlog`, and `$statistics` tables on query-auth tables because column masks cannot be applied to their physical metadata. Doris now preserves query authorization and pins `$file_indexes` to the catalog-visible snapshot and explicitly rejects the unimplemented `latest-delta` OPTIONS mode instead of returning a full snapshot. Paimon's legacy file-based privilege system is no longer supported by the upstream SDK; deployments using it must migrate authorization before upgrading. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test - **Build:** FE/Java extensions and ASAN BE passed after rebasing onto master `5e3ed84abb0`; both Java endpoints package SDK 2.1.0. All 9 JNI plugins passed layout checks. - **Unit tests:** FE connector 844 passed, 1 live-connectivity test skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3 shade 1 passed. The MERGE follow-up passed 38 related FE tests. - **Cluster regression:** 51 Paimon suites passed after rebase; the MERGE follow-up passed 9 targeted suites, including the new nullable-key regression. The opt-in KEY_DYNAMIC resource stress suite was skipped. System-table expected output was regenerated with the runner and replayed successfully. - **Backward reads:** Paimon 1.4.2 append/PK data matched its Spark oracle again; completed profiles confirmed Rust append reads and JNI fallback for multi-file PK merges. Scope: S3/filesystem Paimon catalog. File cache was disabled for a local Btrfs inode-stat issue, and the host swap startup check was skipped. Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not validated. - Behavior changed: - [ ] No. - [x] Yes. Uses Paimon 2.1.0 SDK behavior, including configurable integer overflow checks, corrected pre-epoch timestamp-to-date casts, removal of upstream file-based privileges, snapshot-safe `$file_indexes`, and explicit rejection of unsupported `latest-delta` OPTIONS reads. - Does this need documentation? - [x] No. Dependency upgrade and upstream behavior changes are recorded in the release note. - [ ] Yes. ### Self-review - Validation follow-up scope: fix nullable-key MERGE target-presence detection, then preserve deferred `$file_indexes` authorization and its JNI source snapshot pin, adapt timestamp-policy validation and callbacks to SDK 2.1, and refresh affected system-table expected output through the regression runner. - Goal and scope: shared SDK upgrade plus directly affected API/behavior adaptations. New search and batching features remain separate work; no new connector SPI or native ABI is introduced. - Correctness and parallel paths: FE binding, serialized BE system-table reconstruction, JNI SDK behavior, and reader selection were inspected. `$file_indexes` preserves deferred catalog authorization and carries the snapshot pin through both FE/BE payloads; timestamp policies validate and execute against repaired precision; MERGE uses INNER JOIN semantics or a null-extended target marker rather than testing a nullable key, and user expressions retain their original binding scope; both OPTIONS resolution and statement-fence pinning reject `latest-delta`. - Concurrency and lifecycle: production adaptations add no threads, locks, shared mutable state, native allocations, or resource owners. Java fixtures close their catalog/writer/commit/reader resources; the new cluster suite restores modified session variables in `finally` and preserves tables for debugging. The existing writer-lifecycle regression passed 120 successive inserts and verified compaction writer threads return to zero after each phase. - Performance: reuse the existing row repair and SDK policy evaluation; bind source snapshots during FE planning. MERGE adds one constant target projection only when INSERT branches need unmatched rows. No performance improvement is claimed. - Compatibility and configuration: upstream file privileges are removed, and mixed SDK rolling upgrades remain unverified. No configuration keys, Thrift/Protobuf definitions, new FE/BE variables, or persistence formats are added by Doris. Storage access for `$file_indexes` uses the existing configured catalog endpoint and base-table SELECT privilege; there is no new artifact-download property. - Tests and results: FE scan-properties tests cover denied `$file_indexes` authorization and catalog-published snapshot bounds; an actual serialized JNI reconstruction test covers the second payload. Serialized timestamp-policy tests retain row filters, masking, fallback and nested/hidden-column coverage. MERGE tests cover nullable single/composite keys, ordinary and NULL-safe equality, branch priority, duplicate-match rejection without row changes, and same-named user columns; the new golden is generated and replayed by the regression runner. The system-table golden now includes SDK 2.1's `operation` and `writer_version` fields. Builds, unit tests, cluster replay and legacy-data reads passed after rebase; limits are listed above. - Other CI failures: the P0 view case passed in isolation; it shares and drops `v_distinct_agg_rewrite` with `test_mtmv` in the same database. The Iceberg failure was reproduced as `DESC digest` expecting nullable `Yes` but returning `No`: [master's source-nullability change](c7ffbda3877) predates the [test's old expectation](e83b06cb7b7). Those Iceberg and P0 files are unchanged from the base; no unrelated golden was modified. - Independent optimizer limitation: a one-row constant source with multiple NULL-safe join keys can become an empty relation even when an internal OLAP table has a matching NULL row. This also reproduces in an ordinary SELECT on an internal table; optimizer files are unchanged by this PR. The composite-key DELETE regression uses a multi-row source with an unmatched control. That standalone optimizer defect remains outside this SDK adaptation. - Transactions, writes and observability: no transaction/recovery implementation is changed. The transaction regression confirms an oversized failed insert publishes neither rows nor a snapshot. Completed query profiles prove Rust selection and JNI fallback, rather than relying only on enable flags. Unsupported delta mode fails with an actionable diagnostic; existing system-table planning diagnostics apply to snapshot pinning. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Integrate the Multivalue global-index primitives introduced by #9288 with Paimon's Data Evolution and Primary Key table paths, including spill-sorted index construction in Flink and Spark.
Changes
multivalueindex SPI now that the sorted builders understand normalized index keys.ARRAY_CONTAINS.(element, row-id)entries and spill-sort them in the PK, Flink, and Spark build paths.User impact
Flink and Spark can build a Multivalue index with
create_global_index, and Paimon Core scans can use it to pruneARRAY_CONTAINSpredicates. SQL expression translation to Paimon's predicate remains connector-specific and is outside this PR.Validation
CreateGlobalIndexProcedureTest: 6 tests.git diff --checkand exact tree-equivalence verification after rebasing onto [core] Add Multivalue global index primitives #9288.