Skip to content

[core] Add Multivalue global index primitives - #9288

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/multivalue-core-index
Aug 18, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/multivalue-core-index

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

This PR extracts the common Multivalue index foundation from #9282 into a smaller, independently mergeable change.

What changed

  • Add ARRAY_CONTAINS to the predicate API, JSON serde, evaluator dispatch, and format visitor fallbacks.
  • Add SortedGlobalIndexer and GlobalIndexKeyExtractor so sorted indexes can expose normalized keys instead of requiring builders to sort raw values.
  • Add a bitmap-backed Multivalue reader, streaming writer, indexer, options, factory, and manifest metadata.
  • Preserve membership-only semantics: unsupported array predicates conservatively fall back to normal scanning.
  • Keep the shared bitmap v1 data-file format unchanged.

Split boundary

This foundation PR intentionally does not register the Multivalue SPI or expose table configuration yet. Primary-key, Data Evolution, Flink, and Spark integration remain in #9282 and will register the factory only after their sorted build paths consume normalized element keys.

Validation

  • 152 focused tests passed across paimon-common and paimon-format.
  • MultiValueBitmapIndexReaderTest passed after removing premature SPI registration.
  • Non-fast Maven validate passed for paimon-common and paimon-format, including Checkstyle, Spotless, and Enforcer.
  • Independent dependency and compatibility review passed with no remaining P0-P2 findings.

@JingsongLi
JingsongLi marked this pull request as ready for review August 18, 2026 08:30

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed predicate semantics, JSON/Java serialization compatibility, conservative visitor fallbacks, sorted-key extraction, bitmap metadata and format compatibility, and the focused test suite (152 tests passed). Looks good to me.

@JingsongLi
JingsongLi merged commit fcae207 into apache:master Aug 18, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants