[Feature] Support query auth for REST catalog #520

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Java Paimon lets a REST catalog enforce row-level filters and column masks on reads through the table option query-auth.enabled (CoreOptions.QUERY_AUTH_ENABLED, default false):

• When a table with query-auth.enabled=true plans a scan, AbstractDataTableScan.authQuery() asks the catalog for the rules through RESTCatalog.authTableQuery(), which sends POST /v1/{prefix}/databases/{database}/tables/{table}/auth with {"select": [...]}. The selected columns also include the columns referenced by the user filter and the partition filter, so their values are authorized before they are used for pruning.
• The response is {"filter": [...], "columnMasking": {...}}. filter is a list of JSON-serialized Predicates, combined with AND. columnMasking maps a column name to a JSON-serialized Transform (TableQueryAuthResult).
• The splits are wrapped in QueryAuthSplit with the auth result, and AbstractDataTableRead.createReader() applies the row filter and the column masks when it reads them. With query auth enabled, withFilter() does not push the user predicate into the reader (it returns before innerWithFilter()).
• The rules are re-validated against the latest schema on every plan, so a schema change under a live scan fails closed (validateAgainstSchema, validateReadableWithoutRename).
• The option cannot be disabled through dynamic options once the table enables it (AbstractFileStoreTable.checkImmutability). Search that a column mask would invalidate is rejected (TableQueryAuthResult.rejectSearchUnderQueryAuth).
• CatalogEnvironment.tableQueryAuth() skips the call when the table has no catalog loader, for example an external table of a REST catalog.

paimon-cpp implements none of this:

• query-auth.enabled is not referenced anywhere under include/ or src/;
• RestApi and ResourcePaths have no auth endpoint, and Catalog has no authTableQuery();
• there is no deserialization of the Java Predicate or Transform JSON.

As a result, a paimon-cpp reader that opens a query-auth.enabled=true table through RestCatalog reads every row and every column unmasked. The row filters and column masks the catalog defines for the table are not applied, while a Java reader of the same table gets the filtered, masked result.

Solution

Fail closed first, then add the feature.

1. Fail closed. Until the rules can be applied, reject scans and reads of a table with query-auth.enabled=true with a clear NotImplemented status, so the table can never be read without its rules. Also reject disabling the option through dynamic options, as Java does.

2. REST protocol:

   • add ResourcePaths::AuthTable(database, table) -> /v1/{prefix}/databases/{database}/tables/{table}/auth;
   • add AuthTableQueryRequest (select, optional) and AuthTableQueryResponse (filter: list of strings, columnMasking: string map, both optional) to rest_messages.h/.cpp;
   • add RestApi::AuthTableQuery(identifier, select). It is a read and is safe to retry;
   • map the errors the same way Java does: 404 is table not found, 403 is no permission, 501 is not supported.

3. Catalog: expose the call to the scan, for example as an internal interface next to VersionManagedCatalog that RestCatalog implements. Reject system tables, and skip it for tables without catalog-managed versions (external tables), following CatalogEnvironment.tableQueryAuth().

4. Rule parsing: deserialize the Java Predicate JSON (leaf and compound predicates, literals of every supported type) and the Transform JSON used by column masks. Reject an empty column name, an empty transform, and JSON null, as TableQueryAuthResult does. An unknown predicate or transform must fail the query instead of being ignored.

5. Scan and read:

   • at plan time, request the rules with the projected columns plus the columns referenced by the user filter and the partition filter;
   • validate the rules against the latest schema;
   • carry the result with the splits to the reader;
   • in the reader, apply the row filter and the masks, and do not push the user predicate or column statistics of masked columns down;
   • reject search that a mask would invalidate.

Tests:

• rest_messages_test.cpp: request/response JSON round trips with absent fields;
• rest_catalog_test.cpp: the endpoint and error mapping, using MockRestServer;
• a read test: rows filtered, columns masked, the user filter not pushed down, fail closed when the rules cannot be parsed or reference unknown columns, and the option not being disabled through dynamic options.

Steps 1 and 2 can land on their own before the rest.

Anything else?

Compared against Java Paimon at apache/paimon@c2e001b7f2 and paimon-cpp at 41ed4ea.

Java references:

• paimon-api/.../rest/RESTApi.java (authTableQuery);
• paimon-api/.../rest/ResourcePaths.java (authTable);
• paimon-api/.../rest/requests/AuthTableQueryRequest.java;
• paimon-api/.../rest/responses/AuthTableQueryResponse.java;
• paimon-core/.../rest/RESTCatalog.java (authTableQuery);
• paimon-core/.../catalog/TableQueryAuthResult.java;
• paimon-core/.../table/CatalogEnvironment.java (tableQueryAuth);
• paimon-core/.../table/source/AbstractDataTableScan.java (authQuery);
• paimon-core/.../table/source/AbstractDataTableRead.java;
• paimon-core/.../table/source/QueryAuthSplit.java.

Are you willing to submit a PR?

• I'm willing to submit a PR!