Skip to content

[Feature] Introduce a common Table hierarchy for managed, format and system tables #526

Open

Description

@lucasfang

Search before asking

I searched existing issues. #459 tracks the broader, multi-step table refactoring; this issue covers the common table hierarchy step following the merged FileStore factory extraction in #516.

Motivation

Table-kind dispatch and operator assembly are concentrated in table.cpp, while format tables have a separate handle and system tables are not represented consistently through Catalog::GetTable(). The existing Table::NewScan/NewRead/NewWrite/NewCommit methods delegate to context-based factories, so a loaded table's schema and storage identity do not determine the resulting operator.

Introduce a common table hierarchy, following Java Paimon's separation between managed, format and system tables, so catalogs return the appropriate table implementation and loaded tables create operators from their bound state.

Solution

  • Make Table an abstract interface and implement it with FileStoreTable, FormatTable and SystemTable.
  • Split managed-table behavior into append-only and primary-key implementations, reusing the FileStore factories already merged into main.
  • Have filesystem and REST catalogs dispatch to the correct table type. System tables expose their own schemas and reject writes and commits.
  • Make static operator factories load a table and delegate to its virtual operator methods. Loaded-table operations retain their schema, path and filesystem, apply operation options, and validate conflicting managed-table branches and types.
  • Rename LatestSchema() to GetSchema() to describe the already-loaded schema without implying metadata refresh, and add regression coverage for the two operator entrypoints.

Anything else?

Part of #459 and a follow-up to #516. This issue is limited to the hierarchy step; execution-context separation and general table-copy/time-travel APIs remain separate follow-ups.

Public API changes are expected. Storage formats and protocols remain unchanged.

Are you willing to submit a PR?

  • I'm willing to submit a PR!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions