Repository navigation
datalake_agent: the gRPC contract between the extension and the agent - #2071
MisterRaindrop wants to merge 1 commit into
Conversation
The proto files datalake_fdw and datalake_agent agree on, before either side has code that uses them. IcebergCatalogService has one RPC per IcebergMetaEngine entry the agent will carry; CatalogManagementService creates and lists catalogs and namespaces; fragment.proto holds the messages the C side reads and writes directly -- fragments, written-file reports, pushed-down predicates -- and so lives with it, in the same package. alter_table and truncate_table have no RPC yet: an engine without them leaves the capability unset and the dispatch refuses the call. Each gets its RPC in the change that implements it. What a statement acts on is fixed in the request rather than resolved when the request arrives. LoadTable and CreateTable return the exact metadata location and the table's UUID; GetFragment plans against that location; every write and DropTable must name the UUID, so a table dropped and recreated under the same name is never the one written to or purged; UPDATE, DELETE and rewrite commits name the snapshot they were planned against. Commit state has one authority, the three-way outcome, since a commit whose answer was lost is neither applied nor not. CommitFileGroups streams both ways, one group per message, so a rewrite of any size fits gRPC's message limit. Errors carry an ErrorDetail whose business_code a client branches on. The client sends its version in x-cloudberry-client-version; every response, streamed batches included, carries the server's version and the oldest client it accepts, and each side refuses the other when it is too old. Temporal literals count from the Unix epoch, as Iceberg's do, not PostgreSQL's. Nothing generated is committed. A workflow runs the C++ and the Java gRPC generators on every change to these files, with warnings fatal, and fails if any Java class lands outside the agent's package; the Java generators are pinned and their digests checked.
| sudo apt-get install -y -q --no-install-recommends \ | ||
| protobuf-compiler protobuf-compiler-grpc libprotobuf-dev |
There was a problem hiding this comment.
I believe our Rocky Linux and Ubuntu development Docker image already includes protobuf*, so there is no need to install them again.
There was a problem hiding this comment.
This job doesn't run in our development images: it runs directly on the GitHub-hosted ubuntu-24.04 runner, which has no protoc. The images wouldn't cover it either. Their Dockerfiles on main install protobuf (protobuf-compiler/libprotobuf-dev, protobuf-devel) but no gRPC, and this step needs grpc_cpp_plugin. I also checked the Rocky 9 and Ubuntu 24.04 images: both have protoc, neither has grpc_cpp_plugin. So I'd keep the install for now. Once the datalake build actually needs gRPC, I can add it to the images in a devops PR, and then this job can run in an image instead of installing anything.
What does this PR do?
The contract between
datalake_fdwanddatalake_agent: the proto filesboth sides will build from, before either has code that uses them. Nothing
here is compiled into the extension yet, and the PostgreSQL build is
unchanged.
Closes #2010. Part of #2008 (B0).
contrib/datalake_agent/proto/:common.proto,iceberg_catalog.proto(
IcebergCatalogService),catalog_mgmt.proto(
CatalogManagementService), and a README.contrib/datalake_fdw/src/meta/fragment.proto: fragments, written-filereports and pushed-down predicates -- what the C side reads and writes
directly, so it lives with the C side, in the same package.
.github/workflows/datalake-proto.yml: runs the C++ and the Java gRPCgenerators on every change to these files.
Decisions worth a look:
IcebergMetaEngineentry, and the README maps them in bothdirections.
alter_tableandtruncate_tablehave none yet: an enginewithout them leaves the capability bits unset and the central dispatch
answers
DL_ERR_NOT_SUPPORTEDwithout calling it. Each gets its RPC in thechange that implements it.
arrival.
LoadTableandCreateTablereturn the exact metadata locationand the table UUID;
GetFragmentplans against that location; every writeand
DropTablemust name the UUID, so a table dropped and recreated underthe same name is never the one written to or purged; UPDATE, DELETE and
rewrite commits name the snapshot they were planned against, so conflict
detection starts there and not at whatever is current at commit time.
CommitOutcome. A commitwhose answer was lost is neither applied nor not; a boolean would have to
lie one way or the other.
CommitFileGroupsstreams both ways, one group per message, so arewrite of any size stays under gRPC's message limit.
x-cloudberry-client-version; every response -- streamed batches included-- carries the server's version and the oldest client it accepts. The
server refuses a client that is too old, and the client refuses a server
that is too old for it: a field an older server does not know arrives as
zero, and for several fields here zero means "none".
google.rpc.Statuswith anErrorDetailwhosebusiness_codea client branches on; no stack trace leaves the server.comments give the offset from PostgreSQL's 2000-01-01, because a predicate
off by 10957 days prunes the wrong files and no recheck can bring them back.
protocat build timewith both directories as include roots.
Type of Change
Test Plan
ubuntu:24.04: C++ with the distribution'sprotoc3.21.12 andgrpc_cpp_plugin, Java withprotoc3.25.5 andprotoc-gen-grpc-java1.81.0 -- the versions the agent's Maven build will pin -- all with
--fatal_warnings. Clean on both.--fatal_warnings, and a file withoutjava_packagefails thepackage assertion (its classes land in
cloudberry/datalake/v1/).make installcheck(no C change)Impact
Dependencies: none for the PostgreSQL build. The workflow downloads two
generators from Maven Central, pinned by version and checked against SHA-256
digests that match Maven Central's published SHA-1.
User-facing changes: none.
Checklist
Additional Context
Known limit, in the README:
AppendResponseandUpdateResponsecarryoutcomeandcommitted_metadata_location, which only a commit cantruthfully report.
AppendandUpdatestage; the table changes atCommitAppendandCommitUpdate, and commit state is read from those.B1 (#2011, the Java service framework) builds on this and follows as its own PR.