[python] Add array predicates to PredicateBuilder

[jackylee-ch]

Purpose

Java's PredicateBuilder has arrayContains / arraysOverlap /
arrayContainsAll (Spark pushes them down), but PyPaimon's
PredicateBuilder had no array predicates. A filter on an
ARRAY<STRING> label column ("samples tagged cat") could not be
expressed against a Paimon table and had to be written by the caller
against the materialized result. Array element bounds aren't tracked in
stats, so these do not prune files or reduce table I/O; they make the
filter part of the read plan and return exactly the matching rows.

Change

• Add array_contains / arrays_overlap / array_contains_all to
  PredicateBuilder and the matching Predicate testers, mirroring Java
  ArrayContains / ArraysOverlap / ArrayContainsAll row semantics
  (null array or null element → no match; arrays_overlap needs one
  shared non-null element; array_contains_all is vacuously true for an
  empty literal list on a non-null array).
• Validate the field is ARRAY (mirroring Java
  ArrayContains.elementType) so an array predicate on a STRING/INT
  column raises instead of silently running literal in value and
  selecting wrong rows.
• Arrays have no dataset-expression form, so the three methods are marked
  arrow-unsafe and run on Paimon's exact row-level filter path (like
  startsWith/contains/like) instead of a no-op truthy Arrow
  expression. Stats testers return True (no file is pruned).

Tests

• Row-level tester parity with Java (match/no-match, null array, null
  element, empty literals) and that the three methods are not
  arrow-pushable.
• End-to-end: filter an ARRAY<STRING> column and assert the exact ids
  returned; a negative case asserts a non-ARRAY field is rejected.

Written with Claude Code; verification is mine.

[JingsongLi]

Reviewed 1e800e9. The ARRAY predicates have real end-to-end functional value: the new read-path tests return the exact matching row IDs, and the row-level implementations agree with Java for null and empty-literal cases. One correctness gap needs attention:

[P2] Reject non-array fields instead of silently applying Python containment. The new builder methods only check the field name, and ArrayContains.test_by_value / ArraysOverlap.test_by_value then use literal in val. For a STRING column s = "cat", array_contains("s", "a") and arrays_overlap("s", ["a"]) both return true (I reproduced this locally), whereas Java ArrayContains.elementType rejects a non-ARRAY field. A wrong field name or schema change can therefore silently select incorrect rows. Retain field types in PredicateBuilder and validate these three methods against ArrayType, with a negative test.

Validation: array_predicate_test 7 passed; adjacent projection/predicate tests 15 passed; git diff --check passed; CI run 36100713243 is green. Because the new stats testers always return true and these methods are Arrow-unsafe, this is exact row-level filtering but does not prune files; the PR description should not imply reduced table I/O. Please remove the # placeholder-e2e comment too.

[jackylee-ch]

Thanks for the review. Addressed both points:

• PredicateBuilder now retains the declared field types; array_contains / arrays_overlap / array_contains_all validate the field is ARRAY (mirroring Java ArrayContains.elementType) and raise on a STRING/INT column instead of running literal in val. Added a negative test for the three methods, and removed the # placeholder-e2e comment.
• Reworded Purpose so it no longer implies reduced table I/O: these run on the row-level filter path and don't prune files.

Force-pushed.

[JingsongLi]

Reviewed head 6f4b683234542e086714d2e3c15218f356334d7f. The read-plan array filtering use case has end-to-end value, and the earlier non-ARRAY-field validation finding is fixed. I found one additional numeric-array correctness issue.

[P2] Compare array elements using the declared element type (predicate.py:516,535,553). The new testers all use Python list membership, but the builder accepts every ArrayType and discards the element type when constructing the predicate. For a persisted ARRAY<FLOAT> written from [0.1], array_contains("floats", 0.1) returns no rows: the stored float32 becomes Python 0.10000000149011612, while the literal remains a double-precision Python float. arrays_overlap and array_contains_all have the same false negative. There are also direct ARRAY<DOUBLE> differences from Java's element comparator: a NaN literal misses a stored NaN, and either zero literal matches both -0.0 and +0.0, whereas Java Float/Double.compareTo matches NaN and distinguishes signed zero.

I reproduced these through actual table writes, commits, reloads and reads in Parquet, Avro, ORC and row formats, projecting only the ID column. Please retain the element type, normalize literals accordingly, and use the same typed comparison contract as Java; alternatively reject unsupported element types explicitly. Add numeric-array tests for all three methods. This finding is scoped to the newly accepted array-predicate paths, not the existing scalar Python predicates.

Validation: 86 Python array/predicate/read/projection tests passed; 22 Java PredicateBuilder tests passed on JDK 8 with normal Maven checks. Across all four formats, 24 additional string-array controls passed for exact row IDs, projection, compound OR, empty literal lists and nulls; the numeric mismatches above reproduced in every format. Configured flake8, Python 3.6 grammar and diff checks passed; current-head CI is green. The no-file-pruning limitation is now accurately described in the PR.

[jackylee-ch]

Fixed in de846e4. The array predicates now compare elements with Java's element contract instead of plain Python membership:

• the builder normalizes a FLOAT array's literals to float32, so array_contains('floats', 0.1) matches a stored 0.1 (float32 0.10000000149…) rather than missing it;
• element equality treats NaN as equal to a stored NaN and distinguishes signed zero (+0.0 ≠ -0.0), mirroring Double.compare/Float.compare.

Non-float element types keep exact == semantics. Added ArrayPredicateNumericTest covering float32 precision, NaN, and signed zero across array_contains; neutralizing either half (literal normalization or the typed comparison) fails them. flake8 and git diff --check clean.

[JingsongLi]

Reviewed current head de846e431f2db05bfffe9d8b2cefda951ad3d8e9. The read-plan array filtering use case remains valuable, and the FLOAT 0.1, NaN and floating-literal signed-zero cases from the previous review are fixed. One part of that numeric literal/type finding remains.

[P2] Normalize DOUBLE numeric literals before the signed-zero comparison (paimon-python/pypaimon/common/predicate_builder.py:63-66, predicate.py:518-524). Only FLOAT literals are normalized. For an ARRAY<DOUBLE> column, the accepted Python integer literal 0 remains an int, so _elements_equal(-0.0, 0) falls back to Python == instead of the signed-zero comparator. The same public query therefore returns different rows depending on whether its numeric literal is spelled 0 or 0.0.

On actual written, committed and reopened tables with negative-zero-only rows 3 and 10, all three methods (array_contains("doubles", 0), arrays_overlap("doubles", [0]), array_contains_all("doubles", [0])) return [3, 4, 10, 11, 12], while their 0.0 versions return the correct positive-zero rows [4, 11, 12]. Projecting only ID and combining the filter with OR preserve the incorrect extra rows. This is a remaining issue in the new array APIs, rather than a newly introduced regression in this revision.

For the Java reference, I explicitly normalized Integer.valueOf(0) using PredicateBuilder.convertJavaObject(DataTypes.DOUBLE(), ...) and passed the resulting typed Double to the actual array predicates; those return only [4, 11, 12]. Java's direct builder requires typed literals and does not itself perform that conversion. Please normalize accepted DOUBLE numeric literals too, or explicitly reject unsupported input types, and cover all three public methods with persisted-table integer-zero cases.

Validation: 68 Python array/predicate/read/projection tests and 22 Java PredicateBuilder tests pass; Java tests ran on JDK 8 with normal Maven checks. Configured flake8 and Python 3.6 grammar checks pass. The additional actual-file probes reproduce this mismatch in Parquet, Avro, ORC and row format; current-head CI is green.

[jackylee-ch]

Thanks — fixed in 02d5140. _normalize_array_literals now widens an accepted numeric literal to float for a DOUBLE array too (not just float32 for FLOAT), so an integer 0 reaches the signed-zero-aware comparator instead of falling back to Python ==. array_contains('doubles', 0) / arrays_overlap / array_contains_all now match only the +0.0 rows, same as the 0.0 spelling.

Added test_integer_zero_literal_matches_double_zero_spelling covering all three methods on persisted negative/positive-zero rows; dropping the DOUBLE normalization makes it return the extra -0.0 row. flake8 and git diff --check clean.

[JingsongLi]

Requirement fit: SUPPORTED. The ARRAY read-plan filters have end-to-end value; non-ARRAY validation, FLOAT 0.1 precision and NaN handling are fixed. Implementation: FINDINGS: one accepted-float normalization boundary remains. Signed-zero equivalence is excluded from this review as requested. Validation: 106 Python tests pass in the pinned-compatible runtime, 22 Java PredicateBuilder tests pass with normal Maven checks, and configured flake8 passes. Actual reopened Parquet/Avro/ORC/row tables matched the Java oracle for 444 additional queries; each format still has three query-construction failures described below. A conversion-only control passes all 480 queries. CI is green.

[JingsongLi]

[P2] Handle finite FLOAT literal overflow during normalization

This new conversion accepts Python int/float literals but struct.pack("<f", ...) raises OverflowError for a finite double such as 1e40. On actual committed and reopened ARRAY tables, array_contains("floats", 1e40), arrays_overlap("floats", [1e40]) and array_contains_all("floats", [1e40]) all fail before reading, in Parquet, Avro, ORC and row formats. The same APIs with float("inf") correctly match the stored positive-infinity row. This is inconsistent with the typed normalization used for 0.1: Java PredicateBuilder.convertJavaObject(FLOAT, Double.valueOf(1e40)) uses Number.floatValue() and yields positive infinity, as does PyArrow float32 conversion. A source-free control changing only overflow conversion to the corresponding signed infinity makes the three queries return the correct row IDs, including ID-only projection and compound OR. Please preserve that float32 conversion behavior for finite overflow (both signs) and add a regression test for the three public methods. This concerns finite float overflow, independent of the excluded signed-zero comparison issue.

[jackylee-ch]

Good catch, fixed in 581c4b1. _to_float32 now catches the OverflowError from struct.pack('<f', ...) on a finite double beyond the float32 range and returns the signed infinity, matching Java Number.floatValue() and PyArrow float32 narrowing — instead of aborting the predicate build. Added a regression test over a FLOAT array holding +inf/-inf asserting all three methods (array_contains / arrays_overlap / array_contains_all) match the stored infinity for a 1e40 / -1e40 literal. Mutant-checked: reverting the catch reproduces the OverflowError.