[feature](lance) support lance catalog in doris by zhangstar333 · Pull Request #65730 · apache/doris

zhangstar333 · 2026-07-16T14:27:21Z

What problem does this PR solve?

Issue #66340
Problem Summary:
doc: apache/doris-website#4024

This PR introduces read support for the Lance format in Doris, including:

Querying Lance datasets through a Lance Catalog.
Pushing supported predicates down to Lance to reduce the amount of data scanned.
Reading Lance datasets directly through S3 and Local TVFs.
Performing vector similarity search through the vector_search() TVF, with the distance exposed through the _distance column.
Lance is currently read-only and uses File Scanner V2. Writing to Lance is not supported yet.

  -- Create a Lance Catalog
  CREATE CATALOG lance_catalog PROPERTIES (
      "type" = "lance",
      "lance.catalog.type" = "filesystem",
      "warehouse" = "s3://warehouse/lance",
      "s3.endpoint" = "http://127.0.0.1:9000",
      "s3.access_key" = "admin",
      "s3.secret_key" = "password",
      "s3.region" = "us-east-1",
      "use_path_style" = "true"
  );

  -- Regular query; supported WHERE predicates are pushed down to Lance
  SELECT user_id, user_name
  FROM lance_catalog.doris.user_profiles
  WHERE engagement_score >= 90;

  -- Read a Lance dataset directly through the S3 TVF
  SELECT user_id, user_name
  FROM s3(
      "uri" = "s3://warehouse/lance/doris/user_profiles.lance",
      "format" = "lance",
      "s3.endpoint" = "http://127.0.0.1:9000",
      "s3.access_key" = "admin",
      "s3.secret_key" = "password",
      "s3.region" = "us-east-1",
      "use_path_style" = "true"
  );

  -- Perform a vector similarity search
  SELECT row_id, label, _distance
  FROM vector_search(
      "table" = "lance_catalog.doris.vector_search",
      "column" = "embedding",
      "query_vector" = "[0,0,0,0]",
      "top_k" = "5",
      "metric" = "l2"
  )
  ORDER BY _distance, row_id;

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-16T14:27:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-07-31T07:37:40Z

/review

Gabriel39

I found several security and correctness issues that should be addressed before merging.

github-actions · 2026-07-31T09:24:11Z

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30613481239

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

github-actions

This adds a substantial Lance integration, but the current scan path still has blocking correctness, memory-safety, process-safety, security-display, snapshot-consistency, placement, and resource-governance defects. The inline comments identify thirteen distinct issues that remain after deduplicating against the four live comments already on the PR. Review was static-only as required; no build or test commands were run.

Gabriel39 · 2026-07-31T10:11:01Z

在这一版实现里，我们要把Postfilter改成Lance SDK 原生 Postfilter
另外，测试覆盖要把各种索引类型都加上

zhangstar333 · 2026-08-04T08:19:07Z

run buildall

zhangstar333 · 2026-08-04T11:33:21Z

run buildall

hello-stephen · 2026-08-04T13:07:30Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.53% (34057/52774)
Region Coverage	64.49% (17216/26694)
Branch Coverage	54.03% (9220/17064)

zhangstar333 · 2026-08-04T15:45:46Z

run buildall

hello-stephen · 2026-08-04T16:52:21Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.62% (1904/2453)
Line Coverage	64.45% (34012/52774)
Region Coverage	64.46% (17206/26694)
Branch Coverage	53.96% (9207/17064)

hello-stephen · 2026-08-04T19:19:43Z

BE Regression && UT Coverage Report

Increment line coverage 69.09% (847/1226) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.67% (30364/41215)
Line Coverage	57.62% (333867/579401)
Region Coverage	54.49% (277771/509733)
Branch Coverage	55.46% (124428/224358)

hello-stephen · 2026-08-04T19:33:15Z

FE Regression Coverage Report

Increment line coverage 65.66% (1017/1549) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-08-05T00:50:26Z

run buildall

### What problem does this PR solve? Issue Number: apache#66497 Related PR: apache#65730 Problem Summary: Lance external tables can contain logical vector and scalar indexes, but SHOW INDEX currently only handles internal OlapTable metadata and returns an empty result for Lance tables. Read authoritative logical index descriptions from one latest Directory Dataset snapshot, resolve field IDs from the same snapshot schema, and expose deterministic bounded metadata through the existing 13-column SHOW INDEX result without changing internal-table behavior. ### Release note Support SHOW INDEX for Lance filesystem/Directory Catalog tables. Lance REST index inspection remains unsupported in this phase. ### Check List (For Author) - Test: Unit Test and Regression test - Behavior changed: Yes. SHOW INDEX now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs. - Does this need documentation: Yes. Document the supported Directory-only scope and REST limitation.

### What problem does this PR solve? Issue Number: #66497 Related PR: #65730 Lance external tables can contain logical vector and scalar indexes, but `SHOW INDEX` currently only handles internal `OlapTable` metadata and returns an empty result for Lance tables. This change reads authoritative logical index descriptions from one latest Lance Directory Dataset snapshot, resolves field IDs against the schema from that same snapshot, and exposes deterministic, bounded metadata through the existing 13-column `SHOW INDEX` result without changing internal-table behavior. ### What changed? - Add immutable Lance logical-index metadata and a loader backed by one latest Dataset snapshot. - Support `SHOW INDEX`, `SHOW INDEXES`, `SHOW KEY`, and `SHOW KEYS` for Lance filesystem/Directory Catalog tables. - Preserve existing internal-table behavior and permission-check ordering (privilege check runs before any catalog initialization). - Return a fixed unsupported error for Lance REST catalogs in this phase. - Sanitize Dataset URI, storage credential, and REST secret values from index metadata errors. - Defensively reject non-Lance tables resolved from a Lance catalog instead of failing on a raw cast. - Add unit and regression coverage for normalization, formatting, permissions, aliases, empty-index tables, and error handling. ### Design walkthrough (diagrams) `SHOW INDEX` for Lance Directory tables is a small read pipeline whose main concern is object ownership across the JNI boundary: SDK objects (`Dataset`, `IndexDescription`, Arrow allocators) never escape the loader, and the command layer only sees immutable `LanceLogicalIndex` values. Layered view — three new classes (`LanceLogicalIndex`, `LanceIndexMetadataLoader`, `LanceMetadataReadExecutor`) and three touch points (`ShowIndexCommand`, `LanceExternalTable`, `LanceExternalCatalog`): ```mermaid flowchart TB subgraph CMD["Command layer · Nereids"] A["ShowIndexCommand analyze() privilege check (:83) handleShowIndex() dispatch (:111) buildLanceRows() row mapping (:159)"] end subgraph EXT["External table layer"] B["LanceExternalTable loadIndexMetadata() (:69) — thin delegate"] end subgraph CAT["Catalog layer · caller thread"] C["LanceExternalCatalog isRestCatalogConfigured() (:172) early REST rejection resolveTableAccess() (:371) namespace resolution"] C2["sanitizedRootCauseMessage() (:473) URI / credential / token redaction"] end subgraph BND["Execution boundary · lance-metadata-read pool"] D["LanceMetadataReadExecutor (:50) 4 concurrent · 16 queued · deadline ≤ 60s"] E["LanceIndexMetadataLoader load() (:82) → normalize() (:206)"] end subgraph NAT["native · JNI"] F["Dataset.open(uri, latest) schema + describeIndices from one snapshot"] end G["LanceLogicalIndex immutable value object · pure-Java boundary"] A --> B --> C --> D --> E --> F E --> G C -.->|any failure| C2 C2 -.->|redacted exception only| A ``` Request timeline and resource ownership — namespace resolution stays on the caller thread (it owns the catalog's shared namespace/allocator), the JNI read runs on a bounded worker with a task-owned allocator, and a timed-out caller never cancels the in-flight native call: ```mermaid sequenceDiagram autonumber participant U as User participant C as Caller thread ShowIndexCommand participant K as Caller thread LanceExternalCatalog participant X as Executor boundary participant W as Worker thread lance-metadata-read participant J as Lance JNI Dataset U->>C: SHOW INDEX FROM lance_db.t C->>C: analyze() → checkTblPriv(SHOW) Note over C: privilege check runs before any catalog initialization C->>K: isRestCatalogConfigured() Note over K: reads normalized properties only, no namespace init — REST catalogs rejected here with a fixed error C->>K: resolveTableAccess() Note over K: shared namespace / allocator owned by the caller thread, so catalog close stays safe K-->>C: ResolvedTableAccess(uri, options) C->>X: execute(task) X->>W: submit (wrapped task checks deadline first) W->>W: expired in queue? fail before entering JNI W->>J: open(latest) + schema + describeIndices Note over W,J: task-owned RootAllocator (256MB) — catalog close cannot release it early J-->>W: IndexDescription + LanceSchema W-->>X: immutable index list X-->>C: future.get(remaining deadline) Note over C,X: timeout / interrupt never cancels the Future — the worker stays sole owner of native resources C->>C: buildLanceRows() → 13 columns C-->>U: ShowResultSet ``` Loader internals — one snapshot feeds both the schema mapping and the index descriptions; normalization is fail-closed and produces deterministic, bounded output: ```mermaid flowchart LR O["Dataset.open(uri, latest) index cache 0 · metadata cache 64MB schema and indexes from one snapshot"] O --> S["getLanceSchema().fields()"] O --> L["listIndexes() physical entries ≤ 16384"] subgraph SB["schema branch"] S --> DFS["buildFieldNamesById() (:148) DFS depth ≤ 64 · fields ≤ 16384 field id → dotted path, quoted when needed"] end subgraph IB["index branch"] L --> FL["drop system indexes __lance_frag_reuse · __lance_mem_wal"] FL --> DD["dedupe by logical name ≤ 256"] DD --> CR["describeUserIndexes() (:95) per-name describeIndices(hasName) must return exactly 1, name must match"] end DFS --> V CR --> V subgraph NB["normalize() (:206) · fail-closed"] V["known, unique field ids columns ≤ 64 · names ≤ 16KB total"] --> P["details JSON allowlist TreeMap-sorted · ≤ 400B"] P --> R["exact duplicate names rejected sorted by (name, position)"] end R --> OUT["LanceLogicalIndex list immutable · deterministic"] ``` ### Fail-closed semantics (intentional) Three strict behaviors are deliberate and locked by unit tests — the command reports an error rather than showing potentially misleading metadata: - An index referencing an unknown or nested field ID fails the whole `SHOW INDEX` (`testRejectsUnknownNestedDuplicateNullAndEmptyFieldIds`). - One malformed index entry (invalid details JSON, oversize fields, duplicate name) fails the whole result instead of silently dropping rows (`testRejectsMalformedAndNonObjectJsonWithoutEchoingInput`). - Index names differing only by case are preserved as distinct entries; only exact duplicates are rejected (`testRejectsExactDuplicateNameButPreservesCaseOnlyNames`). ### User/developer impact Users can inspect logical Lance indexes through the standard 13-column `SHOW INDEX` interface for filesystem/Directory catalogs. Lance REST index inspection remains unsupported. ### Release note Support `SHOW INDEX` for Lance filesystem/Directory Catalog tables. ### Check List (For Author) - [x] Unit tests: 28 tests passed - [x] Full FE build: all 27 modules passed - [x] Lance regression test: generation and normal verification modes passed - [x] `git diff --check` - [x] Behavior changed: `SHOW INDEX` now returns logical indexes for Lance Directory tables and explicitly rejects Lance REST catalogs - [x] Documentation needed: document the supported Directory-only scope and REST limitation

Issue Number: close apache#66496 Related PR: apache#65730 Problem Summary: Lance schema discovery rejected Blob v2, Arrow JSON, BFloat16, Arrow null, and Arrow duration fields as unsupported. Add validated FE and BE mappings, materialize Blob payloads through lance-c, normalize BFloat16 values to FLOAT, preserve duration counts as BIGINT, and materialize Arrow null values safely. Support reading Lance Blob v2, Arrow JSON, BFloat16, null, and duration columns. - Test: Unit Test (added; not run locally at user request) - Behavior changed: Yes (the newly supported Lance columns are queryable) - Does this need documentation: Yes (type matrix is documented in the regression suite; user documentation follow-up is required)

## Summary - add Apache Doris to the Query Engines and SQL integrations table - link to the Apache Doris Lance Catalog documentation and source repository - describe the current read-only catalog, predicate pushdown, and vector search capabilities ## References - Documentation: https://doris.apache.org/docs/4.x/lakehouse/catalogs/lance-catalog/ - Implementation: apache/doris#65730 ## Verification - `make build` - `make check-links`

Issue Number: close apache#66496 Related PR: apache#65730 Problem Summary: Lance schema discovery rejected Blob v2, Arrow JSON, BFloat16, Arrow null, and Arrow duration fields as unsupported. Add validated FE and BE mappings, materialize Blob payloads through lance-c, normalize BFloat16 values to FLOAT, preserve duration counts as BIGINT, and materialize Arrow null values safely. Support reading Lance Blob v2, Arrow JSON, BFloat16, null, and duration columns. - Test: Unit Test (added; not run locally at user request) - Behavior changed: Yes (the newly supported Lance columns are queryable) - Does this need documentation: Yes (type matrix is documented in the regression suite; user documentation follow-up is required)

…ce managed versioning ### What problem does this PR solve? Issue Number: close apache#66491 Related PR: apache#65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the latest one whose commit time is not later than the timestamp, parsed in the session time zone with second or millisecond precision and compared with commit times at millisecond precision (the precision a namespace reports them in, so storage and namespace resolution agree); a timestamp earlier than the first version is an error. 2. **REST Namespace managed versioning** – the FE now opens such a table through the SDK's namespace client (`Dataset.open().namespaceClient(..).tableId(..)`), so the latest version comes from `ListTableVersions`, an explicit version from `DescribeTableVersion`, and `FOR TIME AS OF` from the commit times (`timestamp_millis`) the namespace reports. The SDK finalizes a still-staged manifest to its canonical `_versions/` path while opening, so the BE keeps opening the dataset by URI and version number through lance-c exactly as before; no BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Iceberg and Paimon, a non-numeric `FOR VERSION AS OF 'v2'` names a tag. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables the SDK derives the branch from that directory and sends it in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or differs from `table_uri` fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `timestamp_millis` is optional in the Lance Namespace spec. If a namespace lists its versions without it, the only commit times available are the ones in the storage manifests, so `FOR TIME AS OF` uses those, restricted to the versions the namespace lists so that the selected version is always one the namespace records. A manifest records when the writer committed the data, which can be earlier than when the namespace registered the version, so the selection follows the writer's clock in that case. - The SDK opens and checks out a namespace-managed dataset with its own native namespace client (the JNI takes the native handle of a REST or Directory namespace instead of calling back into the Java object), so those operations are not serialized on the FE's namespace lock; only the FE's own namespace requests are. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - An expired version (removed by cleanup) is indistinguishable from one that never existed, on storage and in a namespace, so both get the same message. "in the namespace" is only added when the namespace itself reported the miss. - Every selector other than an explicit version on `main` is a checkout from one open of the latest `main`, and a managed `FOR TIME AS OF` whose namespace reports commit times is resolved before anything is opened. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Reading a managed table whose manifest is still staged makes the SDK copy it to its canonical path, so read credentials must allow that write; an access-denied failure on a managed table says so. - The BE reads by URI and version number, which assumes the manifest a namespace records ends up at its canonical path `_versions/<u64::MAX - v>.manifest`. A staged manifest from a normal commit is moved there on read; a namespace that keeps finalized manifests elsewhere cannot be detected by the FE, and the BE then either fails to find the version or reads a different manifest with the same number at the canonical path. The documentation lists this as a limitation. - Known limitations: two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to load Lance table metadata for default.time_travel: IllegalArgumentException: Lance dataset has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, and `time_travel_managed_partial` records 1 and 3: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / staged manifest / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest`, `LanceTableAccessCacheTest`, `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (doris-website PR to follow: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning limitations)

…ce managed versioning (#68453) ### What problem does this PR solve? Issue Number: close #66491 Related PR: #65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the one committed last at or before the timestamp (of versions committed at the same instant, the newer one), with the timestamp parsed in the session time zone with second or millisecond precision and compared with the full commit time, which Lance records below a millisecond, so a commit later within the requested millisecond is not selected; commit times are compared as recorded, without assuming they grow with version numbers. As Iceberg drops its snapshot log before a removed snapshot, only the versions newer than the most recent one removed by cleanup are candidates, so a timestamp earlier than all of them (before the first version, or in removed history) is an error. 2. **REST Namespace managed versioning** – the namespace decides which versions of such a table exist and which is the newest: the latest version comes from `ListTableVersions`, an explicit version is checked with `DescribeTableVersion`, and `FOR TIME AS OF` selects among the versions the namespace lists. The FE then reads the chosen version the way it reads any Lance dataset, and the way the BE does through lance-c: by the location and storage options of the table's `DescribeTable` and the version number. No BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables, and only that tag's file is read (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Paimon, and for tag names in Iceberg, a non-numeric `FOR VERSION AS OF 'v2'` names a tag; a tag name is never combined with `@branch`, `@branch(main)` included, because the tag already determines its branch. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables Doris sends the branch in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or names another place than `table_uri` (a query string aside), or an `s3+ddb` URI, whose DynamoDB commit handler would record and finalize versions on read, fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `FOR TIME AS OF` uses the commit times the manifests record, for storage-versioned and managed tables alike, as Lance's `asof` does; a managed table only selects among the versions its namespace lists. The namespace's `timestamp_millis` is not used: it is optional in the spec and its meaning varies (Lance's Directory namespace reports the manifest object's modification time, second-granular on S3 and reset when a dataset is copied), so the same dataset would otherwise resolve differently depending on how it is catalogued. A removed version takes its commit time with it, which is why history before it is not used. On a managed chain, a version number the namespace no longer records between versions it records is a removed version too (Lance numbers a chain's commits consecutively, and Lance's Directory namespace only accepts the next version), and so is a recorded version whose manifest is not in storage. - For a managed table every read of the latest version (a plain query, the search functions, a branch head) is pinned to the newest version the namespace records, asked for the way Lance's own reader asks (`ListTableVersions`, newest first, one entry); an empty history is an error. Opening the storage latest would expose a version the namespace never published. - The FE and the BE read a managed table the same way, by URI and version number, rather than the FE opening it through the SDK's namespace client (`Dataset.open().namespaceClient(..)`). The BE's lance-c can only open by URI and version, and with the FE in namespace mode the two readers resolved the store, the credentials and the manifest differently; earlier rounds of this PR kept adding code to align them (a wrapper around the namespace client, store identities, reconciliation of a second describe). Reading both sides by URI removes that class of mismatch. The namespace still decides the versions: Doris asks it directly for every managed read, and a managed table's access is never cached, so each read uses the location and options of its own `DescribeTable`, which the BE receives unchanged. The table describe and the version requests run outside the catalog's namespace lock, which keeps only the namespace and table listings: the REST and Directory namespaces are safe to call concurrently, a managed table makes these requests on every read, and Lance's REST client sets no request timeout, so a stalled request must not hold up every other table. - The BE reads a version by number at its canonical manifest, `<table root>[/tree/<branch>]/_versions/<u64::MAX - v>.manifest` (or the V1 `<v>.manifest`). Doris checks that the manifest the namespace records for the selected version is that path, or a staged manifest beside it (`<canonical>-<id>`, which a namespace that records the path Lance's client sends in `CreateTableVersion` keeps recording after the commit is finalized, since Lance's namespace store cannot update a record). Any other path means the namespace names a manifest neither reader would open (a manifest recorded elsewhere, or a table moved between the describe and the version lookup), and the read fails with both paths in the message and a hint to retry. `FOR TIME AS OF` checks every version the namespace records on the chain, since the commit times of those manifests decide the selection. The table root is derived from the location as lance-io derives it. - A version the namespace records only at a staged manifest whose canonical manifest does not exist (a commit that was never finalized, or whose manifest cleanup removed) cannot be read: the query fails and says so, instead of falling back to an older version, and Doris writes nothing on read. This is how Doris's Iceberg and Paimon connectors treat a snapshot their metadata names but they cannot load. Lance's own namespaces finalize a commit inside `CreateTableVersion`, before they list it, so they do not produce this state for committed versions. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. FE packaging replaces the Linux x86_64 JNI library in that JAR with a glibc 2.17 rebuild, so the pins in `docker/thirdparties/lance-jni-helpers.sh` move to the 12.0.0 rebuild published as [lance-jni-12.0.0-glibc2.17-r1](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-12.0.0-glibc2.17-r1) (built from unmodified Lance v12.0.0 with the procedure of the 11.0.0 asset; its 288 JNI exports match the upstream library, and the Lance v12 Java test suite passes with it on CentOS 7). The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - On a storage-versioned table, an expired version (removed by cleanup) is indistinguishable from one that never existed, so both get the same message. On a managed table the namespace's record tells them apart: a version it does not record is "not found in the namespace", one it records at its canonical path but storage lacks is "not found", and one recorded only at a staged manifest "cannot be read". - A managed table's branch is read from its own directory (`<table>/tree/<branch>`, joined before any query string as Lance joins it) by URI and version, as the BE reads it, so it needs neither a recorded `main` version nor a readable `main` manifest. Every other selector except an explicit version on `main` and the latest `main` is a checkout from one open of the latest `main` in storage; for a managed table that open only supplies the tag files and the manifest listing, and the version finally read is always one the namespace records, checked as above. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Known limitations: a version recorded at a staged manifest is read from its canonical manifest when one exists; a namespace that lets one version number be committed twice (deleting records while the earlier commit's canonical manifest stays in storage) is not detected. While the newest version of a managed table is not finalized, reading the table without `FOR VERSION AS OF` fails, and so does loading its schema. Two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. On a storage-versioned table, a branch whose metadata was deleted, or whose creation stopped partway, stays readable through `@branch` until its directory is cleaned up, because the Lance SDK checks a branch out by its directory; on a managed table the namespace decides whether the branch exists (a missing table reported by Lance's Directory namespace for a branch request counts as the missing branch). - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance table default.time_travel has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, `time_travel_managed_partial` records 1 and 3, and `time_travel_managed_pending` records a fourth version only at a staged manifest that was never finalized: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance cannot resolve FOR TIME AS OF '2026-09-19 13:06:10' on default.time_travel_managed_partial: version 2, which may hold the state at that time, no longer exists mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_pending; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 4 of default.time_travel_managed_pending cannot be read: the namespace records it at a staged manifest, and its canonical manifest, which Doris reads, does not exist (its commit was not finalized, or cleanup removed it) mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_pending FOR VERSION AS OF 3; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 9 | 9 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs, and can record a version at a staged manifest (`"staged": true`) or at any path (`"manifest_path"`), which the `_pending`, `_staged` and `_misrecorded` fixture tables use. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list / relocated table / table moved after it was described / version recorded at a staged manifest, finalized or not, on `main` and on a branch, newest or older / finalized manifest outside its canonical path / the BE receiving the options of the read's describe / history gaps / namespace times that differ from manifest times), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest` (including sub-millisecond commit times, and a check of time selection against its rule on random histories with microsecond commit times and versions a namespace records but storage lacks), `LanceManifestPathsTest` (canonical and staged paths in both naming schemes, branches, rejected paths, a non-ASCII-digit locale, location paths as lance-io derives them), `LanceTableAccessCacheTest` (including a managed table described on every read), `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (apache/doris-website#4172: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning rules and limitations)

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…69575) ### What problem does this PR solve? Issue Number: None Related PR: #65730, #67235, #68781 Problem Summary: **In short.** This PR makes master read the Lance catalogs that branch-4.1 persists (#65730, released in 4.1.4): an FE image written by 4.1.4 or later with a Lance catalog in it now loads. Before this PR the image failed to parse, so a cluster with a Lance catalog could not be upgraded from any released 4.1.4 / 4.1.4.1 / 4.1.5 build (or from 4.2) to a master-based build: the FE did not start. The Lance catalog's GSON labels are mapped to the plugin-driven classes, as the Iceberg, Paimon and LakeSoul labels already are, and the catalog stays registered, unusable until a Lance connector plugin is installed. **Background.** branch-4.1 added a Lance catalog as fe-core classes in #65730 (released in 4.1.4). Like every external catalog, it is persisted in the image's `datasource` module as GSON JSON whose `clazz` field names the class, `LanceExternalCatalog`, with `logType` `LANCE`. master has no Lance classes: external catalogs are served by connector plugins, and the labels of the catalogs that used to be fe-core classes are redirected to `PluginDrivenExternalCatalog` / `PluginDrivenExternalDatabase` / `PluginDrivenMvccExternalTable` by `registerCompatibleSubtype` in `GsonUtils`. A Lance connector plugin does not exist on master yet; it will be ported later. branch-4.1 also had, for a while, a durable Lance index job framework (#67235) that wrote edit-log op 500 (`OP_LANCE_INDEX_JOB_UPSERT`) and an image module named `lanceIndexJobManager`. #68781 reverted it on branch-4.1 and branch-4.2 before any release: among the release tags only 4.1.5-rc01 has the writer, while 4.1.5 and 4.2.0-rc01 keep just a tombstone that reads and discards both. master never had the framework, and this PR does not make it read that metadata either. The upgrade fixtures on master (`upgrade/413`) were generated from 4.1.3, before Lance existed, so no test caught this. **The problem, and what it cost.** Compared on master against fixtures emitted by real 4.1.5 bytecode: | Metadata from a 4.1.4+ FE | master before this PR | after this PR | |---|---|---| | image whose `datasource` module has a Lance catalog (every 4.1.4+ cluster with one) | image fails to parse: `cannot deserialize interface org.apache.doris.datasource.CatalogIf subtype named LanceExternalCatalog; did you forget to register a subtype?`; the FE cannot start | loads; the catalog is a `PluginDrivenExternalCatalog` of type `lance` with all its properties, comment and auto-analyze policies; using it fails with `No ConnectorProvider found for plugin-driven catalog: <name>, type: lance. Ensure the connector plugin is installed.` | | `OP_CREATE_CATALOG` of a Lance catalog | replays as a degraded catalog (the entry carries only properties) | unchanged | The only labels in the GSON registries of 4.1.4.1, 4.1.5 and 4.2.0-rc01 that master does not know are `LanceExternalCatalog`, `LanceExternalDatabase` and `LanceExternalTable`. **How this PR fixes it.** - `GsonUtils`: `LanceExternalCatalog` → `PluginDrivenExternalCatalog`, `LanceExternalDatabase` → `PluginDrivenExternalDatabase`, `LanceExternalTable` → `PluginDrivenMvccExternalTable` (`LanceExternalTable` implemented `MvccTable`). 4.1.x never persists an external database or table, so only the catalog label can actually appear; the other two complete the registry, as for the other migrated connectors. - `InitCatalogLog.Type` keeps `LANCE`, so the `logType` 4.1.4+ wrote still deserializes to itself. Nothing depends on it: a Lance catalog always persists its `type` property (no resource type creates one), and `PluginDrivenExternalCatalog#gsonPostProcess` rewrites the logType to `PLUGIN`. - `BdbjeJournalDataProcNode`, which renders `SHOW PROC '/bdbje/<db>/<journalId>'` in BDBJE debug mode, prints a record read without a payload as `null`, as `BDBTool` already does, instead of failing with a `NullPointerException`. This is an existing master issue that came up in review, not a Lance one: the readers of the retired operation types already on master (the PL/SQL ones of #58700 and the external-catalog ones) keep no payload. Until a Lance connector plugin is installed, an upgraded cluster lists its Lance catalogs and fails only when one is used; installing the plugin makes them work without any metadata change, since the catalog properties are handed to the connector unchanged. **Results.** - New golden fixtures `fe/fe-core/src/test/resources/upgrade/415/`, emitted by 4.1.5-release bytecode like `upgrade/413` (generator and `PROVENANCE.txt` included, byte-deterministic): the `datasource` module with three Lance catalogs (filesystem on S3, filesystem on `file://`, REST with bearer auth), their `OP_CREATE_CATALOG` entries, and 4.1.5's label registries. - `Legacy415LanceUpgradeTest` (7 tests): the image loads and every Lance catalog becomes a `PluginDrivenExternalCatalog` of type `lance` with the `PLUGIN` logType; every persisted property, the comment, `lastUpdateTime` and the auto-analyze policy survive; the persisted logType names still deserialize; without a plugin the catalog fails only when used, with the message above; a provider claiming `lance` receives exactly the persisted properties, from the image and from the journal; the journal entries replay with and without a plugin. - `LanceGsonCompatReplayTest` (4 tests): the three labels replay as the expected classes (the table as the MVCC variant), and they are the only labels 4.1.5 registered beyond 4.1.3. - Each of these changes, undone on its own, turns tests red: dropping the catalog label (5 tests, with the parse error above), the database label, mapping the table label to the non-MVCC class, or removing `LANCE`. - `BdbjeJournalDataProcNodeTest` (2 tests): a retired PL/SQL record, in its written form, shows its journal ID, operation name, size and `null` data in the debug view; a live `OP_SAVE_NEXTID` record still shows its payload. With `getData().toString()` back, the first test fails with a `NullPointerException`. - The three classes above with 13 related upgrade, GSON and journal suites (`Legacy413ImageMigrationTest`, `*GsonCompatReplayTest`, `LegacyExternalTableTypeReplayTest`, `CatalogMgrTest`, `ExternalCatalogTest`, `ConstraintPersistTest`, `BDBJEJournalTest`, `BDBToolTest`, `BDBDebuggerTest`, ...): 103 tests pass; fe-core checkstyle passes. Co-authored-by: kid <lucian1412@outlook.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@zhangstar333

### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>

…udit text ### Background SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit text of CREATE CATALOG replace a property value with `*XXX` when `DatasourcePrintableMap.isSensitiveKey` matches its key. Until now the keys came from a list in fe-core and from the keys filesystem plugins report at startup. A connector plugin marks its secret properties `sensitive = true`, but that mark only masks the connector's own log lines. A connector tells the engine about its security-relevant properties through `ConnectorProvider#securitySurface()`. ### What problem does this PR solve? Issue Number: None Related PR: apache#69588 (added the connector security surface this extends), apache#45164 (introduced the unmasked ALTER CATALOG audit text), apache#27115 (masked it for the statement apache#45164 replaced), apache#59502 (added `iceberg.jdbc.password`), apache#54523 (added the Paimon REST DLF keys), apache#65730 (branch-4.1's Lance catalog, whose REST secrets the Lance port will declare on this surface) PR Type: Bugfix Problem Summary: Catalog secrets that only a connector knows are on no masking list, so SHOW CREATE CATALOG and the other renderings print them in clear: `iceberg.jdbc.password`, `paimon.jdbc.password`, the Paimon REST DLF key pair, `mc.access_key`, and credentials passed through under `trino.` or `adbc.`. Separately, `ALTER CATALOG ... SET PROPERTIES` is written to the audit log unmasked, whatever the key. Each connector now declares its secrets on its security surface, taken from its `sensitive` marks, and the surface can also declare a namespace of secret keys for options passed through unread; Fluss's lake options were hard-coded in fe-core for that reason. The engine masks the union, with fe-core's own list kept as the floor. ALTER CATALOG is masked like CREATE CATALOG. ### Details - **Change map:** 42 files, +1035 −76 over two commits: production code +376 −48, tests +645 −24, the connector README +13 −3, the plugin API version in `fe/fe-connector/pom.xml`. The first commit adds secret keys to the surface (reviewed alone before). The second does the rest: 1. Key namespaces on the SPI: `withSensitiveKeyPrefix` and `SensitiveKeyPrefix.covers`, which holds the matching rule; a collection form of `withSensitiveKeys` (`ConnectorSecuritySurface`, +130). 2. Engine: `isSensitiveKey` matches registered namespaces, the Fluss rule becomes one of them; `ConnectorPluginManager` registers them and reads every surface before it publishes a provider (+70). 3. Connectors: every provider declares its secrets; `MetaStoreProviders.sensitivePropertyKeys()` gives the union of a plugin's metastore backends; `IcebergJdbcMetaStoreProvider` reports its keys like its siblings; the keytab paths lose their `sensitive` mark (+140). 4. ALTER CATALOG audit masking (+16). - **Frequency:** Always. Create or alter a catalog that sets one of the keys above, then run SHOW CREATE CATALOG, SHOW CATALOG `<name>` or `select * from catalogs()`, or read the audit log after an ALTER CATALOG ... SET PROPERTIES. - **Root cause:** `isSensitiveKey` consulted only fe-core's list and the filesystem providers' keys. A connector's `sensitive` marks fed only `toMaskedString`; `MetaStoreProvider.sensitivePropertyKeys()` existed for this, but nothing called it, and the Iceberg JDBC backend did not implement it. For ALTER CATALOG, `AlterCatalogPropertiesCommand` does not implement `NeedAuditEncryption` and `LogicalPlanBuilderForEncryption` has no `visitAlterCatalogProperties`, so `AuditLogHelper` records the original text; the statement it replaced in apache#45164 was masked by apache#27115. - **Why existing tests missed it:** Unit tests: `DatasourcePrintableMapTest` checks fe-core's own keys, the provider surface tests checked only the SSRF and driver_url keys, and `EncryptSQLTest` and `RepositoryAuditEncryptionTest` cover CREATE CATALOG, repositories and resources but not ALTER CATALOG. Regression: `audit/test_external_catalog_credential_masking` creates only Iceberg REST catalogs, whose keys are on fe-core's list, and never alters one. - **Fix:** The engine must not name a connector's properties, so the connectors declare their secrets, from the same `sensitive` marks that mask their logs; jdbc and es declare both spellings their legacy prefix allows, and trino and adbc declare the namespace they pass through, a key there being a secret when its rest names `key`, `secret`, `password`, `token` or `credential` (also `authorization` for ADBC). fe-core keeps the connector keys it lists, so a catalog whose plugin is not loaded masks what it masked before. A provider whose `securitySurface()` throws at startup is skipped, as a filesystem provider is, instead of stopping FE. - **Other paths and branches:** Fixed here: every rendering above and FE's statement logging, which share `isSensitiveKey` and `NeedAuditEncryption`. Not covered, since a key name cannot express them safely: raw Iceberg SDK keys copied through (`token`, `credential`), Paimon options re-keyed from `paimon.*`, Hadoop `fs.*` keys, and secrets inside a URL. The ALTER CATALOG gap is on branch-4.0, 4.1 and 4.2 (from apache#45164); branch-3.1 still uses the masked statement. branch-4.1 and 4.2 also print the connector keys above in clear; without the connector SPI their fix differs and is tracked separately. - **Behavior change:** The namespaces also mask non-secrets whose names carry those words, such as `trino.hive.s3.aws-access-key` or a keystore path. Keytab paths, which no release masks, are no longer masked in connector log lines: marked `sensitive`, `hadoop.kerberos.keytab` would be masked in every storage vault and resource. ### Testing All runs were local (macOS), on master `f5c993f2c94` plus this change: - Regression `audit/test_external_catalog_credential_masking`, extended with an Iceberg JDBC catalog: SHOW CREATE CATALOG masks `iceberg.jdbc.password`, and the audit texts of its CREATE CATALOG and of an ALTER CATALOG that sets the password and `s3.secret_key` hold only `*XXX`. Against the local cluster's FE without this change it fails at the first new check, SHOW CREATE CATALOG printing the password; with a FE built from this change it passes. - fe-connector-spi: all 160 tests, with the refreshed surface baseline (+4 lines); the metastore modules: all tests; ten connector modules: 29 classes, 511 tests, including each connector's surface test, which checks real key names. - fe-core: 26 classes, 341 tests, including `ConnectorPluginManagerTest`, `DatasourcePrintableMapTest`, `EncryptSQLTest` and the audit and plugin gate tests; 14 of them again after the last change to registration. - Twenty mutations, each caught by its test: for example, reverting either half of the ALTER CATALOG fix (`EncryptSQLTest` sees the statement in clear), matching a marker anywhere in the key, an empty metastore union, or re-marking a keytab. ### Notes for Reviewers - `SensitiveKeyPrefix.covers` lives on the SPI so that a connector's test can check its declaration against real key names and the engine applies the same rule; the Fluss floor in fe-core uses it too. - The entries both fe-core and a connector declare are deliberate: the floor is frozen, as the comment in `DatasourcePrintableMap` says. - The plugin API stays at 31.0, the first commit's bump. Hudi declares nothing: it owns no catalog properties. ### Release note SHOW CREATE CATALOG, SHOW CATALOG, the `catalogs()` table function and the audit log now mask `iceberg.jdbc.password`, `paimon.jdbc.password`, `paimon.rest.dlf.access-key-id`, `paimon.rest.dlf.access-key-secret`, `mc.access_key`, and the `trino.*` and `adbc.*` properties whose names carry `key`, `secret`, `password`, `token` or `credential` (`authorization` too for `adbc.*`). ALTER CATALOG ... SET PROPERTIES statements are masked in the audit log like CREATE CATALOG. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the code, the tests, and this description - Human review: not stated - Prompts: The author first asked for a generic connector SPI to mask a connector's secret catalog properties instead of adding the Lance keys to fe-core's list. Then, in this PR: let the surface express prefix rules such as Fluss's, mask the union of the engine's and the connectors' keys, wire every connector, keep fe-core's existing entries as the floor, mask whatever is marked `sensitive` but stop marking keytab paths, and fix the unmasked ALTER CATALOG audit text on master only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zhangstar333 requested a review from yiguolei as a code owner July 16, 2026 14:27

zhangstar333 force-pushed the branch-4.1-lance-catalog branch 2 times, most recently from ce3b540 to f760bd0 Compare July 30, 2026 12:25

zhangstar333 changed the title ~~Branch 4.1 test~~ [feature](lance) support lance catalog in doris Jul 30, 2026

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from 4dd1bee to eb379a8 Compare July 31, 2026 07:17

morningman mentioned this pull request Jul 31, 2026

[Tracking] Support Lance Catalog #66322

Closed

22 tasks

Gabriel39 reviewed Jul 31, 2026

View reviewed changes

github-actions Bot previously requested changes Jul 31, 2026

View reviewed changes

Gabriel39 mentioned this pull request Jul 31, 2026

[Tracking] Lance feature matrix for Doris 4.2 #66340

Open

68 of 87 tasks

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from eb379a8 to 56d7904 Compare August 3, 2026 15:08

Gabriel39 added the dev/5.0.x label Aug 4, 2026

zhangstar333 force-pushed the branch-4.1-lance-catalog branch from c9a4827 to 70516cf Compare August 4, 2026 11:32

zhangstar333 added 6 commits August 5, 2026 08:55

thrift

b43b4f3

be code about lance reader

d724df4

be ut test

47cf8a9

docker lance

38ab475

update format v2

8a919a3

update

19da132

Jay-ju mentioned this pull request Aug 23, 2026

[feature](lance) push COUNT(*) down to Lance dataset metadata #66999

Merged

Gabriel39 mentioned this pull request Sep 4, 2026

docs: add Apache Doris integration lance-format/lance#8983

Merged

zy-kkk mentioned this pull request Sep 23, 2026

[feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453

Merged

This was referenced Oct 24, 2026

[fix](fe) Load FE images that carry Lance catalogs written by 4.1.4+ #69575

Merged

[feature](be) Read Lance datasets with FileScannerV2 #69581

Merged

morningman mentioned this pull request Nov 6, 2026

[fix](catalog) Mask connector catalog secrets and the ALTER CATALOG audit text #70338

Open

Conversation

zhangstar333 commented Jul 16, 2026 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 16, 2026

Uh oh!

zhangstar333 commented Jul 31, 2026

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 31, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gabriel39 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Cloud UT Coverage Report

Uh oh!

zhangstar333 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Aug 4, 2026

FE Regression Coverage Report

Uh oh!

zhangstar333 commented Aug 5, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhangstar333 commented Jul 16, 2026 •

edited by morningman

Loading

Gabriel39 commented Jul 31, 2026 •

edited

Loading