[Feature] Support batch reads of primary-key tables with dynamic buckets #493

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Primary-key tables created by Paimon Java with bucket = -1 (HASH_DYNAMIC, or KEY_DYNAMIC
when the primary key does not contain all partition keys) are common in Flink and Spark
pipelines. Paimon C++ currently rejects every scan of such a table:

do not support pk table bucket=-1 in scan process

Batch planning does not need the dynamic bucket assigner. It only uses the buckets recorded in
the manifest entries, so a batch read can be supported without implementing dynamic bucket
writes.

One prerequisite is the deletion vector index. IndexManifestFile and IndexManifestFileHandler
take the configured bucket option (int32_t) and treat -1 as BUCKET_UNAWARE, which then
combines deletion vectors by data file name. A dynamic bucket primary-key table also has
bucket = -1, but keeps one deletion vector index per (partition, bucket), like Java's
BucketMode.HASH_DYNAMIC. The existing comment in GetIndexManifestFileCombine notes this is
only safe while dynamic bucket primary-key tables cannot be written, and that lifting the
restriction requires passing the resolved BucketMode.

Solution

1. Pass the resolved BucketMode (ResolveBucketMode(options.GetBucket(), table_schema)) to
   IndexManifestFile / IndexManifestFileHandler instead of the raw bucket option, and only
   use the global deletion vector combiner for BUCKET_UNAWARE. No behavior change for tables
   that can be written today.
2. In NewDataTableScan, keep rejecting dynamic bucket primary-key tables only in streaming
   mode, and allow batch scans.
3. Add an integration test reading a Java-written dynamic bucket primary-key table with
   deletion vectors spread over several buckets (insert, upsert, delete snapshots).

Writing dynamic bucket primary-key tables and streaming reads stay out of scope.

Anything else?

I will split this into two PRs: the BucketMode change first, then the batch read with the
test fixture.

Are you willing to submit a PR?

• I'm willing to submit a PR!