Skip to content

datalake_agent: Hadoop (filesystem) catalog #2017

Open

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B7.

Scope

  • HadoopCatalog over the volume's warehouse path (version-hint.text per table), FileIO from the volume; namespaces are directories. The hadoop server type already in the whitelist routes here.
  • Documented limit: commits rely on an atomic rename, which object stores lack, so concurrent writers to one table are unsafe on S3. Documentation only, no warning at CREATE SERVER.
  • End-to-end on local FS and on MinIO, including a test that shows what two concurrent commits do on each.

Out of scope

HDFS as storage (follow-up of A).

Depends on

B3.

Acceptance

  • Conformance suite green on local FS and MinIO; Spark opens the same warehouse path.
  • The S3 concurrent-commit behaviour is documented from the test's outcome.

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions