[Bug] Partial-update staged merging can lose delete boundaries and field-version information #524

Summary

Partial-update merging can produce incorrect results when intermediate merged records are merged again: a delete boundary can be lost, or a field value can lose its association with its originating sequence number.

Both cases below were reproduced with the ordinary Writer on baseline becc48a51c8a740bf36f201df5fb23fbb2d2b0f1, without production-code changes. Only diagnostic tests were added. These existing defects are tracked independently from the realtime partial-update feature.

Reproduction setup

The examples below are pseudocode, not runnable Paimon APIs. Each write_batch represents one input batch; record order, batch boundaries, and commit order are intentional. commit(writer, n) represents ordinary Write → PrepareCommit(true, n) → Commit. read_all() performs a fresh scan/plan/read of committed data; reopen_and_read_all() destroys and recreates the writer/commit context before reading again. No realtime context, transport offsets, or progress APIs are involved.

Common options for both cases:

file.format=parquet
file-system=local
bucket=1
bucket-key=id
target-file-size=1048576
merge-engine=partial-update
read.batch-size=1
num-sorted-run.compaction-trigger=100
num-sorted-run.stop-trigger=200

The ordinary Writer retained its normal MergeTreeCompactManager; compaction was not triggered. This was not a Noop-compaction control, and write-only was not explicitly set.

Case 1: delete followed by reinsertion in the same batch retains an old field

Schema: id BIGINT, a BIGINT, b BIGINT, primary key id.

Additional option:

partial-update.remove-record-on-delete=true

writer.write_batch([INSERT(id=1, a=10, b=20)])
commit(writer, 0)

writer.write_batch([
    DELETE(id=1, a=null, b=null),
    UPDATE_AFTER(id=1, a=30, b=null)
])
commit(writer, 1)

read_all()
# expected: [(id=1, a=30, b=null)]
# actual:   [(id=1, a=30, b=20)]

reopen_and_read_all()
# actual:   [(id=1, a=30, b=20)]

The delete should reset the record. The subsequent partial update must not resurrect b=20 from before the delete.

Passing control: with the same schema, options, seed, and operation order, committing the DELETE separately as commit 1 and then the reinsertion separately as commit 2 passes. Commit 1 and its reopened read are empty; commit 2 and its reopened read return [(id=1, a=30, b=null)].

Source analysis suggests that an intermediate summary loses the preceding delete boundary, allowing a later null field to retain an older value. This is a hypothesis from source analysis, not an instrumented observation of intermediate records.

Case 2: sum is correct but last_non_null_value comes from the wrong sequence

Schema: id BIGINT, seq BIGINT, total BIGINT, last STRING, primary key id.

Additional options:

fields.seq.sequence-group=total,last
fields.total.aggregate-function=sum
fields.last.aggregate-function=last_non_null_value

writer.write_batch([
    INSERT(id=1, seq=9, total=6, last="nine"),
    INSERT(id=1, seq=10, total=4, last="ten")
])
commit(writer, 0)
# committed seed: [(id=1, seq=10, total=10, last="ten")]

writer.write_batch([
    INSERT(id=1, seq=30, total=30, last="thirty"),
    INSERT(id=1, seq=20, total=20, last="twenty")
])
writer.write_batch([
    UPDATE_AFTER(id=1, seq=25, total=5, last="middle"),
    UPDATE_BEFORE(id=1, seq=20, total=20, last=null),
    UPDATE_AFTER(id=1, seq=30, total=2, last=null)
])
commit(writer, 1)

read_all()
# expected: [(id=1, seq=30, total=47, last="thirty")]
# actual:   [(id=1, seq=30, total=47, last="middle")]

reopen_and_read_all()
# actual:   [(id=1, seq=30, total=47, last="middle")]

There is no commit or explicit ordinary Seal/spill between the two batches preceding commit 1; the ordinary reproduction uses one flush per commit. The sum remains correct at 47, but last should retain "thirty" from sequence 30 rather than "middle" from sequence 25.

Source analysis suggests that staged merging loses the association between a field value and its originating sequence: a summary may associate "middle" with sequence 30 despite its origin at sequence 25. The original run did not instrument intermediate stages, so the precise mechanism still needs confirmation.

Evidence and scope

The original test patches, exact inputs, logs, JUnit results, and passing separate-commit control are preserved locally. They have not been uploaded as public attachments; no external evidence links are claimed here. Expected results were preserved without weakening assertions or replacing them with the observed incorrect output.

The realtime feature reuses the same merge implementation and may inherit these limitations. These examples were not independently reproduced on the current realtime feature commit 2e2f194ccdab76831b2988a9b749f5d8c299d8f7; this issue does not claim that realtime introduced these defects or that the feature fixes them. Separate hidden sequence-group projection/deletion dependency problems are outside the two reproductions documented here.

For a fixed operation order and configuration, expected logical results must remain invariant under changes to batch or intermediate-merge boundaries. An independent fix should preserve the expectations above, retain the passing separate-commit control, and verify ordinary committed and reopened reads. Realtime coverage should be established separately against an explicitly identified source revision and its observable stages.