Repository navigation
feat(core): Support schema management over REST in RESTCatalog - #9673
Merged
JingsongLi merged 3 commits intoSep 8, 2026
Merged
JingsongLi merged 3 commits into
JingsongLi merged 3 commits into
Conversation
Add a Catalog-level API for listing schemas and expose it over the REST
protocol so that RESTCatalog-backed tables can read historical schemas
without direct filesystem access.
* Introduce `Catalog#supportsSchemaManagement` and
`Catalog#listSchemas(Identifier, SchemaFilter)` with a
`SchemaFilter` value object (all / latest / earliest / by id / by
range).
* Add `GET /v1/{prefix}/databases/{db}/tables/{obj}/schemas` with a
`ListSchemaResponse` payload; encode `SchemaFilter` as query
parameters (`latest`, `earliest`, `schemaId`, `maxSchemaId`,
`minSchemaId`).
* Implement `RESTCatalog#supportsSchemaManagement`/`listSchemas` and
map REST errors to catalog exceptions.
* Add `CatalogSchemaManager`, a `SchemaManager` that delegates to the
owning `Catalog` (analogous to `CatalogBranchManager`); reads go
through `listSchemas`, writes reuse existing
`Catalog#createTable`/`alterTable`/`rollbackSchema`.
* Wire `AbstractFileStoreTable#schemaManager` to prefer
`CatalogSchemaManager` whenever `supportsSchemaManagement()` is true.
* Extend the REST mock server and add tests covering the new filter
variants and the catalog-backed schema manager.
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Contributor
Author
|
Hi @JingsongLi , this PR adds schema management support to RESTCatalog. Could you please take a look when you have time? Thanks! |
JingsongLi
requested changes
Sep 8, 2026
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
- API design
Could we follow the existing loadSnapshot and listSnapshotsPaged pattern here, with loadSchema(identifier, version) and listSchemasPaged(identifier, maxResults, pageToken)? The version could support LATEST, EARLIEST, and a schema ID. This would keep the API consistent and avoid introducing SchemaFilter. The responses should carry the complete TableSchema to preserve its metadata.
- Backward compatibility
These new APIs should only serve schema display use cases. Existing functionality must continue using its current implementation so that newer clients remain compatible with older REST servers.
Could we remove CatalogSchemaManager and the related capability flag and table-environment wiring from this PR? Please also add a compatibility test against a server without the new endpoints, verifying that existing operations still work and never call those endpoints.
5 of 11 tasks
suxiaogang223
added a commit
to apache/doris
that referenced
this pull request
Oct 23, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0, the latest stable release published on October 12, 2026 ([upstream release](https://github.com/apache/paimon/releases/tag/release-2.1.0)). Pin the FE connector and BE JNI scanner to the same shared `paimon.version`, since tables, splits, and commit messages cross the FE/BE boundary using Paimon's serialization. Adapt the connector to the removal of Paimon's file-based `Privilege*` API in 2.1.0, removing its obsolete catalog/table wrappers and reflection-based delegate reconstruction. Preserve catalog query authorization, fallback/chain branch semantics, statement-bound schemas and snapshots, and Doris metadata-cache governance. Update local test fixtures to `FileSystemSchemaManager` and the extended `Snapshot` constructor. Give scanner tests the `java.nio` access already enabled in the BE JVM for Arrow Variant buffer views. Update the oversized-write negative test to the SDK's revised diagnostic while retaining its snapshot/row atomicity checks. Retain the branch/scan API adaptations and grouped format-file split fixture required since 2.0. Verify both the default wrapping and explicitly enabled checked integer-overflow modes in 2.1, as well as the corrected pre-epoch timestamp-to-date cast. Keep the existing Rust reader eligibility rules and common/S3 compatibility patches. Fix changelog MERGE target-presence detection for nullable primary keys: matched-only plans use their INNER JOIN semantics, while plans with INSERT branches use a pre-join marker. Keep that private marker outside user expression resolution. Preserve catalog query authorization for the new `$file_indexes` system table and pin both serialized table payloads to the catalog-published snapshot before stripping their catalog loaders. Adapt timestamp precision repair to the SDK's authorization projection validation and four-argument policy callback; use real JSON policy definitions in serialized scanner fixtures. Reject `scan.mode=latest-delta` in Doris `@options`, because the existing full-snapshot pinning would silently widen a delta scan; explicit `@incr` ranges remain available. The FE and BE Java extensions must be deployed with the same SDK version. This change does not establish compatibility between mixed 1.4.2/2.1.0 SDKs during a rolling upgrade. ### Paimon 2.1 feature impact on Doris | Upstream change | Doris impact and scope of this PR | | --- | --- | | [Nullable primary keys](apache/paimon#9094), opt-in with `primary-key.nullable=true` | Existing write-column mapping respects the SDK field nullability. Added a real-table test of nullable key metadata, FE/BE table serialization, and NULL-key deduplication across commits through the Java reader. The cluster regression additionally checks NULL-key deduplication across commits through JNI, and checks a single-file table through both JNI and the actual Rust Reader (verified from the completed query profile). Multi-file primary-key merges retain the existing JNI fallback. Add Doris SQL coverage for NULL-safe MERGE UPDATE/DELETE on nullable single/composite keys, ordinary equality, INSERT branches, conditional priority, duplicate-match atomicity and private-marker name collisions. | | [`$file_indexes`](apache/paimon#9138) | Automatically discovered through the SDK registry. Preserve deferred catalog query checks and pin both backend payloads. FE scan-properties and actual JNI reconstruction tests cover denied authorization and snapshot 2 on disk while the catalog publishes only snapshot 1. | | [`latest-delta` batch startup mode](apache/paimon#9142) | Not exposed by this upgrade: reject it explicitly in `@options` instead of silently converting it to a full-snapshot read. A real three-snapshot test proves the difference (one delta row versus three full-snapshot rows). | | [Vector indexes](apache/paimon#8930) and [full-text indexes](apache/paimon#9031) on DV tables; [Multivalue array indexes](apache/paimon#9282); BTree TopN | New query acceleration needs Doris expression/search/TopN integration and relevant runtime dependencies and validation. Bumping the SDK does not enable these SQL capabilities. Existing reader selection remains conservative. | | [Arrow bundle writes](apache/paimon#9100) | An optional batching API, not a new `file.format=arrow` storage format. Doris keeps its current Arrow-to-Paimon-row writer; using the new bundle API is a separate performance change requiring memory/lifecycle validation and benchmarks. | | [SchemaManager interface](apache/paimon#9174) and [REST schema history APIs](apache/paimon#9673) | Updated fixtures to `FileSystemSchemaManager`. The REST additions inspect schema history; they do not replace the SDK table's filesystem schema manager. Doris does not call the new display APIs in this PR. | | Manifest/index metadata and Variant handling | Both Java endpoints use SDK 2.1.0; package and serialization tests passed. The scanner tests now have the same `java.nio` access as the BE JVM. The targeted Java, native and cluster checks below cover existing supported paths, but do not establish native/Rust compatibility for every newly introduced table option or file layout. | | BLOB partial updates and first-row/data-evolution improvements | Available upstream; the upgrade preserves Doris' existing write and reader eligibility constraints. New combinations still need external-table regression coverage before being advertised. | | [Removal of file-based privileges](apache/paimon#9186), arithmetic/cast fixes, raw physical-metadata restrictions | Actual behavior changes, described in the release note below; legacy privilege users must migrate before upgrading. | ### Release note Upgrade the Paimon Java SDK from 1.4.2 to 2.1.0. Fix NULL-safe MERGE matched UPDATE/DELETE silently omitting rows with NULL primary keys. Paimon 2.1 keeps wrapping integer `sum`/`product` overflow by default and supports per-field `fields.<field>.sum.fail-on-overflow` / `fields.<field>.product.fail-on-overflow` options for checked arithmetic. Timestamp-to-date casts before the Unix epoch now use the preceding calendar date. The SDK rejects raw `$files`, `$file_key_ranges`, `$binlog`, and `$statistics` tables on query-auth tables because column masks cannot be applied to their physical metadata. Doris now preserves query authorization and pins `$file_indexes` to the catalog-visible snapshot and explicitly rejects the unimplemented `latest-delta` OPTIONS mode instead of returning a full snapshot. Paimon's legacy file-based privilege system is no longer supported by the upstream SDK; deployments using it must migrate authorization before upgrading. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Manual test - **Build:** FE/Java extensions and ASAN BE passed after rebasing onto master `5e3ed84abb0`; both Java endpoints package SDK 2.1.0. All 9 JNI plugins passed layout checks. - **Unit tests:** FE connector 844 passed, 1 live-connectivity test skipped; Java scanner 175 passed; native Paimon/JNI/Rust 144 passed; S3 shade 1 passed. The MERGE follow-up passed 38 related FE tests. - **Cluster regression:** 51 Paimon suites passed after rebase; the MERGE follow-up passed 9 targeted suites, including the new nullable-key regression. The opt-in KEY_DYNAMIC resource stress suite was skipped. System-table expected output was regenerated with the runner and replayed successfully. - **Backward reads:** Paimon 1.4.2 append/PK data matched its Spark oracle again; completed profiles confirmed Rust append reads and JNI fallback for multi-file PK merges. Scope: S3/filesystem Paimon catalog. File cache was disabled for a local Btrfs inode-stat issue, and the host swap startup check was skipped. Paimon HMS/REST catalogs and mixed-SDK rolling upgrades were not validated. - Behavior changed: - [ ] No. - [x] Yes. Uses Paimon 2.1.0 SDK behavior, including configurable integer overflow checks, corrected pre-epoch timestamp-to-date casts, removal of upstream file-based privileges, snapshot-safe `$file_indexes`, and explicit rejection of unsupported `latest-delta` OPTIONS reads. - Does this need documentation? - [x] No. Dependency upgrade and upstream behavior changes are recorded in the release note. - [ ] Yes. ### Self-review - Validation follow-up scope: fix nullable-key MERGE target-presence detection, then preserve deferred `$file_indexes` authorization and its JNI source snapshot pin, adapt timestamp-policy validation and callbacks to SDK 2.1, and refresh affected system-table expected output through the regression runner. - Goal and scope: shared SDK upgrade plus directly affected API/behavior adaptations. New search and batching features remain separate work; no new connector SPI or native ABI is introduced. - Correctness and parallel paths: FE binding, serialized BE system-table reconstruction, JNI SDK behavior, and reader selection were inspected. `$file_indexes` preserves deferred catalog authorization and carries the snapshot pin through both FE/BE payloads; timestamp policies validate and execute against repaired precision; MERGE uses INNER JOIN semantics or a null-extended target marker rather than testing a nullable key, and user expressions retain their original binding scope; both OPTIONS resolution and statement-fence pinning reject `latest-delta`. - Concurrency and lifecycle: production adaptations add no threads, locks, shared mutable state, native allocations, or resource owners. Java fixtures close their catalog/writer/commit/reader resources; the new cluster suite restores modified session variables in `finally` and preserves tables for debugging. The existing writer-lifecycle regression passed 120 successive inserts and verified compaction writer threads return to zero after each phase. - Performance: reuse the existing row repair and SDK policy evaluation; bind source snapshots during FE planning. MERGE adds one constant target projection only when INSERT branches need unmatched rows. No performance improvement is claimed. - Compatibility and configuration: upstream file privileges are removed, and mixed SDK rolling upgrades remain unverified. No configuration keys, Thrift/Protobuf definitions, new FE/BE variables, or persistence formats are added by Doris. Storage access for `$file_indexes` uses the existing configured catalog endpoint and base-table SELECT privilege; there is no new artifact-download property. - Tests and results: FE scan-properties tests cover denied `$file_indexes` authorization and catalog-published snapshot bounds; an actual serialized JNI reconstruction test covers the second payload. Serialized timestamp-policy tests retain row filters, masking, fallback and nested/hidden-column coverage. MERGE tests cover nullable single/composite keys, ordinary and NULL-safe equality, branch priority, duplicate-match rejection without row changes, and same-named user columns; the new golden is generated and replayed by the regression runner. The system-table golden now includes SDK 2.1's `operation` and `writer_version` fields. Builds, unit tests, cluster replay and legacy-data reads passed after rebase; limits are listed above. - Other CI failures: the P0 view case passed in isolation; it shares and drops `v_distinct_agg_rewrite` with `test_mtmv` in the same database. The Iceberg failure was reproduced as `DESC digest` expecting nullable `Yes` but returning `No`: [master's source-nullability change](c7ffbda3877) predates the [test's old expectation](e83b06cb7b7). Those Iceberg and P0 files are unchanged from the base; no unrelated golden was modified. - Independent optimizer limitation: a one-row constant source with multiple NULL-safe join keys can become an empty relation even when an internal OLAP table has a matching NULL row. This also reproduces in an ordinary SELECT on an internal table; optimizer files are unchanged by this PR. The composite-key DELETE regression uses a multi-row source with an unmatched control. That standalone optimizer defect remains outside this SDK adaptation. - Transactions, writes and observability: no transaction/recovery implementation is changed. The transaction regression confirms an oversized failed insert publishes neither rows nor a snapshot. Completed query profiles prove Rust selection and JNI fallback, rather than relying only on enable flags. Unsupported delta mode fails with an actionable diagnostic; existing system-table planning diagnostics apply to snapshot pinning. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Follow up on #9174 by adding display-only schema history APIs to
RESTCatalog.The APIs follow the existing snapshot API pattern and allow clients to inspect complete historical
TableSchemametadata through the REST service. Existing table operations and schema-management internals remain unchanged.API
Catalog
loadSchemasupports:LATESTEARLIESTlistSchemasPagedreturns schemas in descending schema ID order.REST protocol
The responses contain complete
TableSchemaobjects so that schema metadata is preserved.Backward compatibility
getTable,alterTable, androllbackSchemabehavior continues to use the existing implementation.SchemaManager, orCatalogEnvironmentwiring is introduced.Tests
Added coverage for:
LATEST,EARLIEST, and a specific schema IDValidation performed:
All targeted tests passed: 4 tests, 0 failures, 0 errors.