Skip to content

datalake_agent: service framework (Java) #2011

Open

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is B1.

Scope

contrib/datalake_agent: JDK 17, Maven, Guice, Netty gRPC. No Iceberg in it yet.

  • Two-phase startup: bind and serve health first, build the injector and flip to SERVING second.
  • Transport: Unix socket by default with configured permissions; TCP only with TLS and a 0600 bearer-token file (current plus rotation-grace token).
  • Interceptors: request context, exception → status with BusinessErrorCode (no stack trace leaves the process), version handshake refusing an old client and naming both versions.
  • HOCON config for process concerns only; everything about a table travels in the request. One request mapper turns the fdw's options into Iceberg property maps.
  • Credential scrubbing in logs; opt-in metrics.
  • Every business RPC answers UNIMPLEMENTED naming the RPC. Catalog management present, off by default.
  • In-process gRPC tests, a Unix-socket test, negative TLS/token tests; CI runs mvn -B verify; RAT headers. The PostgreSQL build does not need Java.

Out of scope

Iceberg operations and catalogs (B3+), the proxy bgworker, installing the jar through make install.

Depends on

B0.

Acceptance

  • Cold start to SERVING under 1.5 s in CI; health answers on the socket.
  • Old client refused naming both versions; TCP without TLS refuses to start; bad token is UNAUTHENTICATED; business RPCs are UNIMPLEMENTED naming the RPC.
  • A fake key in config and request never appears in DEBUG logs (asserted by test).
  • mvn -B verify and RAT green in CI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions