[core][flink][spark][python][vector] Support vector indexes on DV tables by XiaoHongbo-Hope · Pull Request #8930 · apache/paimon

XiaoHongbo-Hope · 2026-07-30T09:33:59Z

Purpose

Allow building vector global indexes on deletion-vector (DV) enabled tables. Generic global-index building previously rejected every index type on DV tables, even though the vector read paths can filter DV-deleted rows and safely build over the physical-row superset.

Changes

Add an opt-in GlobalIndexerFactory.supportsDeletionVectors() capability, defaulting to false.
Enable the capability for Lumina and all current paimon-vector native index types: ivf-flat, ivf-pq, ivf-sq, ivf-rq, and diskann.
Relax the Flink builder guard only for opted-in factories and thread the selected index type into the builder.
Pin the Spark vector reader to the snapshot used during planning.
Carry the planned snapshot through PyPaimon vector plans and use it for DV live-row filtering, scalar-index filtering, and raw fallback reads.
Refresh global-index deletions from the latest snapshot on every materialized-deletion compaction commit attempt. This removes indexes committed after compaction preparation and repeats the refresh after optimistic-commit conflicts.

Related merged fixes

[core][flink] Pin DV vector-search read path to the plan snapshot #8946 pins the core and Flink DV vector-search paths to the plan snapshot.
[core] Drop concurrently-built global indexes at materialize-deletion compaction #8947 scans the latest snapshot while preparing global-index deletions for materialized-deletion compaction.
[core][python] Fix global-index live-row filter re-adding DV-deleted rows across overlapping files #8939 contains the union / two-phase live-row filtering fix.

This PR completes snapshot pinning for Spark and PyPaimon and closes the remaining preparation-to-commit race.

Tests

Lumina, its legacy identifier, and all native vector index types are allowed on DV-enabled tables.
Non-opt-in index types remain rejected on DV-enabled tables and continue to work without DV.
Spark and PyPaimon readers retain and propagate the planned snapshot through live-row, scalar-index, and raw-read paths.
A global index committed after materialized-compaction preparation is deleted when that compaction commits.

JingsongLi · 2026-08-03T11:30:16Z

Thanks for working on this. I think this PR needs two changes before merge:

Please complete snapshot pinning for every reader before advertising deletion-vector support. supportsDeletionVectors() is a factory-wide capability, but snapshot pinning is currently incomplete. The core and Flink readers assign planSnapshot = plan.snapshot(), while SparkDataEvolutionVectorRead#read overrides the method without doing so. PyPaimon's VectorSearchScanPlan also only carries splits, and its live-row filter creates a new scan against the latest snapshot. If materialized-deletion compaction commits between planning and reading, an old index plan can be combined with the new row-id/DV state, producing missing or incorrect top-K results. Please thread the planned snapshot through Spark and PyPaimon as well, including the live-row/scalar filters, and add regression coverage for this race.
This PR should also support paimon-vector. paimon-vector is the more important/current native vector-index implementation; Lumina is only the legacy index. NativeVectorGlobalIndexReader already honors includeRowIds for both single and batch search, so NativeVectorGlobalIndexerFactory should opt in to deletion-vector support after the snapshot-pinning issue above is fixed, with tests covering the native index types on DV tables. Limiting this change to Lumina leaves the primary implementation unsupported and makes the feature incomplete.

I consider both items part of this PR rather than follow-ups. The FileStoreCommitImpl refresh still makes sense because it closes the preparation-to-commit/retry race.

Allow deletion-vector tables to build vector global indexes whose readers can filter deleted row IDs. Opt in both Lumina and every paimon-vector native index factory while keeping other generic index types rejected.\n\nPin Spark and PyPaimon vector reads to the snapshot used for planning. Thread that snapshot through live-row filtering, scalar-index filtering, and raw fallback reads so materialized-deletion compaction cannot mix old index plans with new row-id or DV state.\n\nRefresh global-index deletions on every materialized-compaction commit attempt so indexes committed after preparation are still removed.

XiaoHongbo-Hope · 2026-08-03T12:10:36Z

Thanks for working on this. I think this PR needs two changes before merge:

Please complete snapshot pinning for every reader before advertising deletion-vector support. supportsDeletionVectors() is a factory-wide capability, but snapshot pinning is currently incomplete. The core and Flink readers assign planSnapshot = plan.snapshot(), while SparkDataEvolutionVectorRead#read overrides the method without doing so. PyPaimon's VectorSearchScanPlan also only carries splits, and its live-row filter creates a new scan against the latest snapshot. If materialized-deletion compaction commits between planning and reading, an old index plan can be combined with the new row-id/DV state, producing missing or incorrect top-K results. Please thread the planned snapshot through Spark and PyPaimon as well, including the live-row/scalar filters, and add regression coverage for this race.

This PR should also support paimon-vector. paimon-vector is the more important/current native vector-index implementation; Lumina is only the legacy index. NativeVectorGlobalIndexReader already honors includeRowIds for both single and batch search, so NativeVectorGlobalIndexerFactory should opt in to deletion-vector support after the snapshot-pinning issue above is fixed, with tests covering the native index types on DV tables. Limiting this change to Lumina leaves the primary implementation unsupported and makes the feature incomplete.

I consider both items part of this PR rather than follow-ups. The FileStoreCommitImpl refresh still makes sense because it closes the preparation-to-commit/retry race.

Thanks for the detailed review. Both requested changes are addressed in 6363ee8:

JingsongLi · 2026-08-04T03:21:46Z

+    /**
+     * Whether the read path filters DV-deleted rows, so building over all physical rows is safe.
+     */
+    default boolean supportsDeletionVectors() {


Remove this, we have support all vector index.

JingsongLi

I found one PyPaimon snapshot-lifecycle correctness issue that should be addressed before merge.

JingsongLi · 2026-08-04T03:23:05Z


    def read_plan(self, plan):
        # type: (VectorSearchScanPlan) -> GlobalIndexResult
+        self._plan_snapshot = plan.snapshot()


Storing the plan snapshot on the reader makes the result depend on previous calls. read() remains public and does not clear _plan_snapshot, so reusing a reader after read_plan(plan_at_S1) and then calling read(splits_planned_at_S2) makes the S2 live-row, scalar-index, and raw-fallback paths read from S1. The mutable field is also unsafe for concurrent reads, and the batch path has the same issue. After DV materialization this can return deleted rows or omit valid top-K results. Could we keep the reader stateless and pass the snapshot explicitly for each invocation (for example, _read(splits, snapshot), with read_plan forwarding plan.snapshot()), instead of caching it on self?

JingsongLi

The latest update exposes one additional DV snapshot-consistency issue outside the vector readers.

JingsongLi · 2026-08-04T11:49:29Z

-                        + "deleted rows to be indexed.",
-                table.name());
-
        scanSnapshot = table.snapshotManager().latestSnapshot();


Removing the DV guard here enables every generic index type, including full-text and es-index, but those read paths are not snapshot-consistent yet. FullTextRead.read(Plan) discards the plan context, and DataEvolutionFullTextRead computes live rows from the latest snapshot. If materialized-DV compaction commits between planning at S1 and reading at S2, the S1 index row IDs are intersected with S2 live rows, which can produce missing or empty top-K results. Please keep non-vector indexes gated (or restore an explicit vector-capability check) until full-text plans carry their snapshot and both live-row and raw fallback reads are pinned to it. A plan -> materialized compaction -> read regression test would cover this race.

Let‘s add support to full-text too.

JingsongLi · 2026-08-04T11:51:48Z

+1

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable release published on October 12, 2026 ([upstream release](https://github.com/apache/paimon/releases/tag/release-2.1.0)). Pin the FE connector and BE JNI scanner to the same shared `paimon.version`, since tables, splits, and commit messages cross the FE/BE boundary using Paimon's serialization. Adapt the connector to the removal of Paimon's file-based `Privilege*` API in 2.1.0, removing its obsolete catalog/table wrappers and reflection-based delegate reconstruction. Preserve catalog query authorization, fallback/chain branch semantics, statement-bound schemas and snapshots, and Doris metadata-cache governance. Update local test fixtures to `FileSystemSchemaManager` and the extended `Snapshot` constructor. Give scanner tests the `java.nio` access already enabled in the BE JVM for Arrow Variant buffer views. Update the oversized-write negative test to the SDK's revised diagnostic while retaining its snapshot/row atomicity checks. Retain the branch/scan API adaptations and grouped format-file split fixture required since 2.0. Verify both the default wrapping and explicitly enabled checked integer-overflow modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast. Keep the existing Rust reader eligibility rules and common/S3 compatibility patches. Fix changelog MERGE target-presence detection for nullable primary keys: matched-only plans use their INNER JOIN semantics, while plans with INSERT branches use a pre-join marker. Keep that private marker outside user expression resolution. Preserve catalog query authorization for the new `$file_indexes` system table and pin both serialized table payloads to the catalog-published snapshot before stripping their catalog loaders. Adapt timestamp precision repair to the SDK's authorization projection validation and four-argument policy callback; use real JSON policy definitions in serialized scanner fixtures. Reject `scan.mode=latest-delta` in Doris `@options`, because the existing full-snapshot pinning would silently widen a delta scan; explicit `@incr` ranges remain available. The FE and BE Java extensions must be deployed with the same SDK version. This change does not establish compatibility between mixed 1.4.2/2.1.0 SDKs during a rolling upgrade. ### Paimon 2.1 feature impact on Doris | Upstream change | Doris impact and scope of this PR | | --- | --- | | [Nullable primary keys](apache/paimon#9094), opt-in with `primary-key.nullable=true` | Existing write-column mapping respects the SDK field nullability. Added a real-table test of nullable key metadata, FE/BE table serialization, and NULL-key deduplication across commits through the Java reader. The cluster regression additionally checks NULL-key deduplication across commits through JNI, and checks a single-file table through both JNI and the actual Rust Reader (verified from the completed query profile). Multi-file primary-key merges retain the existing JNI fallback. Add Doris SQL coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite keys, ordinary equality, INSERT branches, conditional priority, duplicate-match atomicity and private-marker name collisions. | | [`$file_indexes`](apache/paimon#9138) | Automatically discovered through the SDK registry. Preserve deferred catalog query checks and pin both backend payloads. FE scan-properties and actual JNI reconstruction tests cover denied authorization and snapshot 2 on disk while the catalog publishes only snapshot 1. | | [`latest-delta` batch startup mode](apache/paimon#9142) | Not exposed by this upgrade: reject it explicitly in `@options` instead of silently converting it to a full-snapshot read. A real three-snapshot test proves the difference (one delta row versus three full-snapshot rows). | | [Vector indexes](apache/paimon#8930) and [full-text indexes](apache/paimon#9031) on DV tables; [Multivalue array indexes](apache/paimon#9282); BTree TopN | New query acceleration needs Doris expression/search/TopN integration and relevant runtime dependencies and validation. Bumping the SDK does not enable these SQL capabilities. Existing reader selection remains conservative. | | [Arrow bundle writes](apache/paimon#9100) | An optional batching API, not a new `file.format=arrow` storage format. Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle API is a separate performance change requiring memory/lifecycle validation and benchmarks. | | [SchemaManager interface](apache/paimon#9174) and [REST schema history APIs](apache/paimon#9673) | Updated fixtures to `FileSystemSchemaManager`. The REST additions inspect schema history; they do not replace the SDK table's filesystem schema manager. Doris does not call the new display APIs in this PR. | | Manifest/index metadata and Variant handling | Both Java endpoints use SDK 2.1.0; package and serialization tests passed. The scanner tests now have the same `java.nio` access as the BE JVM. The targeted Java, native and cluster checks below cover existing supported paths, but do not establish native/Rust compatibility for every newly introduced table option or file layout. | | BLOB partial updates and first-row/data-evolution improvements | Available upstream; the upgrade preserves Doris' existing write and reader eligibility constraints. New combinations still need external-table regression coverage before being advertised. | | [Removal of file-based privileges](apache/paimon#9186), arithmetic/cast fixes, raw physical-metadata restrictions | Actual behavior changes, described in the release note below; legacy privilege users must migrate before upgrading. | ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE matched UPDATE/DELETE silently omitting rows with NULL primary keys. Paimon 2.1 keeps wrapping integer `sum`/`product` overflow by default and supports per-field `fields.<field>.sum.fail-on-overflow` / `fields.<field>.product.fail-on-overflow` options for checked arithmetic. Timestamp-to-date casts before the Unix epoch now use the preceding calendar date. The SDK rejects raw `$files`, `$file_key_ranges`, `$binlog`, and `$statistics` tables on query-auth tables because column masks cannot be applied to their physical metadata. Doris now preserves query authorization and pins `$file_indexes` to the catalog-visible snapshot and explicitly rejects the unimplemented `latest-delta` OPTIONS mode instead of returning a full snapshot. Paimon's legacy file-based privilege system is no longer supported by the upstream SDK; deployments using it must migrate authorization before upgrading. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test - **Build:** FE/Java extensions and ASAN BE passed after rebasing onto master `5e3ed84abb0`; both Java endpoints package SDK 2.1.0. All 9 JNI plugins passed layout checks. - **Unit tests:** FE connector 844 passed, 1 live-connectivity test skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3 shade 1 passed. The MERGE follow-up passed 38 related FE tests. - **Cluster regression:** 51 Paimon suites passed after rebase; the MERGE follow-up passed 9 targeted suites, including the new nullable-key regression. The opt-in KEY_DYNAMIC resource stress suite was skipped. System-table expected output was regenerated with the runner and replayed successfully. - **Backward reads:** Paimon 1.4.2 append/PK data matched its Spark oracle again; completed profiles confirmed Rust append reads and JNI fallback for multi-file PK merges. Scope: S3/filesystem Paimon catalog. File cache was disabled for a local Btrfs inode-stat issue, and the host swap startup check was skipped. Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not validated. - Behavior changed: - [ ] No. - [x] Yes. Uses Paimon 2.1.0 SDK behavior, including configurable integer overflow checks, corrected pre-epoch timestamp-to-date casts, removal of upstream file-based privileges, snapshot-safe `$file_indexes`, and explicit rejection of unsupported `latest-delta` OPTIONS reads. - Does this need documentation? - [x] No. Dependency upgrade and upstream behavior changes are recorded in the release note. - [ ] Yes. ### Self-review - Validation follow-up scope: fix nullable-key MERGE target-presence detection, then preserve deferred `$file_indexes` authorization and its JNI source snapshot pin, adapt timestamp-policy validation and callbacks to SDK 2.1, and refresh affected system-table expected output through the regression runner. - Goal and scope: shared SDK upgrade plus directly affected API/behavior adaptations. New search and batching features remain separate work; no new connector SPI or native ABI is introduced. - Correctness and parallel paths: FE binding, serialized BE system-table reconstruction, JNI SDK behavior, and reader selection were inspected. `$file_indexes` preserves deferred catalog authorization and carries the snapshot pin through both FE/BE payloads; timestamp policies validate and execute against repaired precision; MERGE uses INNER JOIN semantics or a null-extended target marker rather than testing a nullable key, and user expressions retain their original binding scope; both OPTIONS resolution and statement-fence pinning reject `latest-delta`. - Concurrency and lifecycle: production adaptations add no threads, locks, shared mutable state, native allocations, or resource owners. Java fixtures close their catalog/writer/commit/reader resources; the new cluster suite restores modified session variables in `finally` and preserves tables for debugging. The existing writer-lifecycle regression passed 120 successive inserts and verified compaction writer threads return to zero after each phase. - Performance: reuse the existing row repair and SDK policy evaluation; bind source snapshots during FE planning. MERGE adds one constant target projection only when INSERT branches need unmatched rows. No performance improvement is claimed. - Compatibility and configuration: upstream file privileges are removed, and mixed SDK rolling upgrades remain unverified. No configuration keys, Thrift/Protobuf definitions, new FE/BE variables, or persistence formats are added by Doris. Storage access for `$file_indexes` uses the existing configured catalog endpoint and base-table SELECT privilege; there is no new artifact-download property. - Tests and results: FE scan-properties tests cover denied `$file_indexes` authorization and catalog-published snapshot bounds; an actual serialized JNI reconstruction test covers the second payload. Serialized timestamp-policy tests retain row filters, masking, fallback and nested/hidden-column coverage. MERGE tests cover nullable single/composite keys, ordinary and NULL-safe equality, branch priority, duplicate-match rejection without row changes, and same-named user columns; the new golden is generated and replayed by the regression runner. The system-table golden now includes SDK 2.1's `operation` and `writer_version` fields. Builds, unit tests, cluster replay and legacy-data reads passed after rebase; limits are listed above. - Other CI failures: the P0 view case passed in isolation; it shares and drops `v_distinct_agg_rewrite` with `test_mtmv` in the same database. The Iceberg failure was reproduced as `DESC digest` expecting nullable `Yes` but returning `No`: [master's source-nullability change](c7ffbda3877) predates the [test's old expectation](e83b06cb7b7). Those Iceberg and P0 files are unchanged from the base; no unrelated golden was modified. - Independent optimizer limitation: a one-row constant source with multiple NULL-safe join keys can become an empty relation even when an internal OLAP table has a matching NULL row. This also reproduces in an ordinary SELECT on an internal table; optimizer files are unchanged by this PR. The composite-key DELETE regression uses a multi-row source with an unmatched control. That standalone optimizer defect remains outside this SDK adaptation. - Transactions, writes and observability: no transaction/recovery implementation is changed. The transaction regression confirms an oversized failed insert publishes neither rows nor a snapshot. Completed query profiles prove Rust selection and JNI fallback, rather than relying only on enable flags. Unsupported delta mode fails with an actionable diagnostic; existing system-table planning diagnostics apply to snapshot pinning. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

XiaoHongbo-Hope force-pushed the feature/lumina-index-support-dv-tables branch 3 times, most recently from ef31d32 to 5b03c02 Compare July 30, 2026 13:59

XiaoHongbo-Hope marked this pull request as draft July 30, 2026 14:08

XiaoHongbo-Hope force-pushed the feature/lumina-index-support-dv-tables branch from 5b03c02 to 5cb2e47 Compare July 30, 2026 14:13

XiaoHongbo-Hope changed the title ~~[core][flink] Support building Lumina global index on deletion-vector tables~~ [core][flink][lumina] Allow building Lumina global index on DV-enabled tables Jul 30, 2026

XiaoHongbo-Hope marked this pull request as ready for review August 1, 2026 09:52

XiaoHongbo-Hope force-pushed the feature/lumina-index-support-dv-tables branch 2 times, most recently from 9ea4129 to 3b1a6ab Compare August 3, 2026 11:46

XiaoHongbo-Hope force-pushed the feature/lumina-index-support-dv-tables branch from 3b1a6ab to 6363ee8 Compare August 3, 2026 12:05

XiaoHongbo-Hope changed the title ~~[core][flink][lumina] Allow building Lumina global index on DV-enabled tables~~ [core][flink][spark][python][vector] Support vector indexes on DV tables Aug 3, 2026

XiaoHongbo-Hope added 2 commits August 3, 2026 22:30

[core][spark][python] Pin vector reads to planned snapshot

cc612b9

[core] Preserve resolved schema for pinned vector reads

7c93d54

JingsongLi reviewed Aug 4, 2026

View reviewed changes

XiaoHongbo-Hope added 2 commits August 4, 2026 11:42

[vector][python] Simplify DV support and snapshot handling

63d7f7c

[vector][python] Fix DV snapshot consistency edge cases

71bcb6a

JingsongLi reviewed Aug 4, 2026

View reviewed changes

JingsongLi merged commit de71827 into apache:master Aug 4, 2026
18 checks passed

XiaoHongbo-Hope mentioned this pull request Aug 4, 2026

[core] Support full-text indexes on DV tables #9031

Merged

TheR1sing3un mentioned this pull request Sep 24, 2026

[python] Support vector index builds on deletion-vector tables #10157

Merged

suxiaogang223 mentioned this pull request Oct 21, 2026

[improvement](fe) Upgrade Paimon to 2.1.0 apache/doris#69343

Merged

5 of 11 tasks

Navigation Menu

Repository navigation

[core][flink][spark][python][vector] Support vector indexes on DV tables - #8930

[core][flink][spark][python][vector] Support vector indexes on DV tables#8930
JingsongLi merged 5 commits into
apache:masterfrom
XiaoHongbo-Hope:feature/lumina-index-support-dv-tables

XiaoHongbo-Hope commented Jul 30, 2026 •

edited

Loading

Uh oh!

JingsongLi commented Aug 3, 2026

Uh oh!

XiaoHongbo-Hope commented Aug 3, 2026

Uh oh!

JingsongLi Aug 4, 2026

Uh oh!

JingsongLi left a comment

Uh oh!

JingsongLi Aug 4, 2026

Uh oh!

JingsongLi left a comment

Uh oh!

JingsongLi Aug 4, 2026

Uh oh!

JingsongLi Aug 4, 2026 •

edited

Loading

Uh oh!

JingsongLi commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

XiaoHongbo-Hope commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

Related merged fixes

Tests

Uh oh!

JingsongLi commented Aug 3, 2026

Uh oh!

XiaoHongbo-Hope commented Aug 3, 2026

Uh oh!

JingsongLi Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

JingsongLi Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

XiaoHongbo-Hope commented Jul 30, 2026 •

edited

Loading

JingsongLi Aug 4, 2026 •

edited

Loading