[liang8283]

Summary

Adds an opt-in --heap-file-hash flag to gpbackup so that --incremental can skip heap tables whose data has not changed. Today every heap table is re-copied by every incremental backup; with this flag, a heap table is skipped when its on-disk file state, its last DDL timestamp and the definitions of the enum and composite types its columns use all match the previous backup.

Without the flag, behavior is unchanged: heap tables are always included, and AO tables keep using modcount + DDL timestamp.

--ao-file-hash from the first revision has been dropped: the aoseg content hash does not change on DELETE, so it is not a safe replacement for modcount (see the discussion below).

How it works

• What is hashed. A plpgsql helper, gp_toolkit.gpbackup_file_info(oid), runs on every segment via gp_dist_random. It resolves the relation's path with pg_relation_filepath() and stats every segment file of the main fork (path, path.1, path.2, ... until one is missing, the same way md.c walks them), so tables larger than RELSEG_SIZE are fully covered. The path itself is part of the hash, so a new relfilenode (TRUNCATE, VACUUM FULL, CLUSTER) is detected as a change.
• When it is hashed. Hashes are collected before the backup snapshot is taken, on a separate connection, right after a CHECKPOINT so that pg_stat_file sees flushed data. Any change committed after the snapshot moves the data files on by the next backup's CHECKPOINT, so the next incremental sees a different hash and includes the table. (Hashing after the snapshot would let changes the backup cannot see enter the hash.)
• In-progress writers. The data files also hold rows of transactions that are still open, and their COMMIT only updates the commit log. After reading the hashes and before the snapshot, gpbackup checks pg_locks and drops the hash of every table on which another transaction holds a write lock (RowExclusiveLock or stronger).
• Verification after the snapshot. A change committed after the first CHECKPOINT but before the snapshot may still sit in shared buffers, invisible to the hash but visible to the backup. Once the snapshot is taken, gpbackup issues a second CHECKPOINT, re-reads the file info of every hashed table, and drops the hash of any table whose files changed.
• DDL. Each heap entry also records the table's last DDL timestamp from pg_stat_last_operation, read inside the backup snapshot. A DDL that does not rewrite the table, such as DROP COLUMN, therefore still forces the table into the incremental.
• Columns. Restore loads a skipped table's old data file into the newest column list, and some changes alter that list or how stored values are dumped without touching the data files or the table's DDL timestamp: ALTER TYPE ... RENAME VALUE on an enum, ALTER TYPE ... ADD/DROP ATTRIBUTE on a composite type, adding a column to a table whose row type another table uses as a column type, or DROP DOMAIN ... CASCADE dropping a column. Each heap entry also records a hash of the table's columns (number, name, type, typmod, dropped) and of the definitions of the enum and composite types they use, directly or through domains, arrays, composite, range and multirange types, read inside the backup snapshot under a savepoint. If it cannot be read, no heap table is skipped.
• Hashes are keyed by oid until they are matched with the table's name inside the backup snapshot, so a rename between hashing and the snapshot cannot attach a hash to the wrong table.
• Unlogged tables are never hashed: a regular CHECKPOINT does not write their buffers, so their files may not move when they change.
• Partitions. Leaf partitions of partitioned heap tables are hashed like any other heap table.
• Failing safe. Whenever a hash cannot be trusted, no hash is recorded and the table is included in the next incremental:
  • either CHECKPOINT fails, or the lock check fails;
  • another transaction holds a write lock on the table;
  • the table's files changed between the first read and the post-snapshot verification;
  • a segment file was modified in the current second (pg_stat_file reports mtime in whole seconds, so a later write in the same second would not move it);
  • any segment cannot describe the relation, or the query fails.
• Decision. FilterTablesForIncremental skips a non-AO table only when the previous restore plan has data for it (a table that belonged to an extension in the previous backup has none), both the previous and the current TOC carry a hash for it, and the hash, the DDL timestamp and the column hash all match.

Files changed

File	Change
toc/toc.go	Add HeapEntry{FileHashMD5, LastDDLTimestamp, ColumnsMD5} and IncrementalEntries.Heap (omitempty).
options/flag.go	Add --heap-file-hash.
backup/backup.go	Collect heap file hashes in DoSetup before the connection pool takes the snapshot, and verify them after it.
backup/queries_incremental.go	CollectHeapFileHashes, getTablesWithActiveWriters, VerifyHeapFileHashes, ensureFileStatFunction, getHeapTables, getFileHashesForTables (batched), getHeapIncrementalEntries, getHeapLastDDLTimestamps, getHeapColumnHashes.
backup/wrappers.go	backupIncrementalMetadata writes heap entries (hash, DDL timestamp, column hash) into the TOC.
backup/incremental.go	FilterTablesForIncremental skips heap tables via heapTableUnchanged, only if the previous restore plan has data for them.
backup/validate.go	--heap-file-hash requires --leaf-partition-data.
backup/incremental_test.go	Unit tests for the heap branches: unchanged, hash changed, DDL changed, column types changed, hash missing in either backup.

Backward compatibility

• Without the flag, the TOC and the incremental decision are identical to before.
• TOC additions are omitempty; older TOCs load fine, and older gprestore ignores the new fields.
• An incremental based on a backup taken without the flag includes every heap table (no previous hash).
• gp_toolkit.gpbackup_file_info is created only when the flag is used, for the duration of each hashing step, and dropped afterwards, so a stale body is never reused and nothing is left in gp_toolkit. The (text, text) signature from the first revision of this PR is dropped.

Operational notes

• The flag issues a cluster-wide CHECKPOINT at the start of each backup, plus a one-second wait so files flushed by it are not treated as "modified in the current second", and a second CHECKPOINT right after the snapshot is taken (usually cheap, since little is dirty by then).
• Hashes are collected for all heap tables in the schemas being backed up; --include-table / --exclude-table are applied afterwards.

How tested

• make unit: all suites pass, including the new FilterTablesForIncremental heap tests.
• End to end on a Cloudberry cluster: full backup with --heap-file-hash, then mutations, then an incremental, then a restore of the incremental. Every table's row count matches the source, and:

Scenario	Result
Append to a heap table >1GB (first segment file unchanged, only .1 grows)	Included
Insert committed while the full backup waits for its table lock (after the snapshot)	Included; restore returns both rows
Uncommitted INSERT / UPDATE / DELETE during the full backup, committed after it	Included; restore matches the source
Heap + AO insert committed in one transaction during the wait after the CHECKPOINT	Included; restore returns the new row in both tables
ALTER TABLE ... DROP COLUMN	Included; restore succeeds with the new column list
ALTER TYPE ... RENAME VALUE on an enum used directly, via a domain array, or via a composite type	Included; restore returns the new label
ALTER TYPE ... ADD ATTRIBUTE / DROP ATTRIBUTE ... CASCADE on a composite type used directly, nested in another composite, or via a domain array	Included; restore matches the source
ALTER TABLE ... ADD COLUMN on a table whose row type another heap table uses as a column type	Both tables included; restore matches the source
Partitioned heap table, one leaf modified	Modified leaf included, sibling leaves skipped
DROP DOMAIN ... CASCADE dropping a column of a heap table	Included; restore succeeds
Heap table removed from an extension after the full backup (ALTER EXTENSION ... DROP TABLE)	Included; restore returns its rows
Insert into an existing page of an UNLOGGED heap table	Included (unlogged tables are never skipped)
Chain full → incremental → incremental	Unchanged table skipped twice and restored from the full backup
Unmodified heap table	Skipped

[talmacschen-arch]

Thanks for this — thoughtful PR and the overall design is sound: with both flags off the path is logically identical to the previous FilterTablesForIncremental, the TOC additions are all omitempty so backward compatibility holds, using a dedicated dbconn to avoid transaction-abort propagation is the right call, and excluding modcount from the AO content hash to dodge GP5's cross-partition modcount propagation addresses a real pain point.

I checked the two issues below against the Cloudberry kernel source and reproduced both on a live cluster, so they are not speculation — details are in the inline comments:

• [correctness] --heap-file-hash can silently miss changes on heap tables larger than 1GB — inline on gpbackup_file_info.
• [gap] --heap-file-hash does not cover leaf partitions of partitioned heap tables — inline on getHeapTables.

Two more points:

Test coverage: the new decision logic in FilterTablesForIncremental (AO-hash / heap-hash branches: prev vs current equal → skip, differ → include, either empty → include, Heap == nil → include) has no unit test — the test change only moves the existing flags-off call into JustBeforeEach. These branches are pure map logic over constructed TOCs and can be unit-tested without a live DB; worth adding since this is the heart of the feature.

nits:

• ensureFileStatFunction only CREATE OR REPLACEs when the function is absent; an older same-signature (oid) body would not be refreshed.
• The plpgsql EXCEPTION WHEN OTHERS THEN RETURN '' swallows every error (including permission failures) — safe-by-degradation, but hides real errors when debugging.
• CHECKPOINT is a cluster-wide heavy operation; frequent incrementals add I/O — worth a note in the docs.

Nice work overall — physical-state hashing for change detection is the right direction; both issues are about which physical state is being hashed.

[talmacschen-arch]

pg_relation_filepath(oid) returns only the first 1GB segment, so this hash can silently miss changes on heap tables larger than 1GB.

Checked against the kernel source and reproduced on a live cluster.

Source: a heap fork is split into RELSEG_SIZE-block segments (storage/smgr/md.c: sprintf(segpath, "%s.%u", path, segno), with the comment "Zero or more full segments of exactly RELSEG_SIZE blocks each"), RELSEG_SIZE defaulting to 1GB. But GetRelationPath (common/relpath.c) never appends a .N suffix — the segment suffix is a storage-layer detail in md.c only. So pg_relation_filepath always yields the base segment, and pg_stat_file only stats that one file.

Live reproduction (single-segment heap table, all rows hashed to one segment):

• After loading 1.5GB: main segment = 1073741824 | 23:51:12 (exactly 1GB, full); .1 segment = 511836160 | 23:51:13.
• After appending 0.5GB more (lands in the tail segment, since the main one is already full) + CHECKPOINT: main segment = 1073741824 | 23:51:12 — byte-for-byte identical mtime and size — while .1 grew to 1040351232 | 23:53:03.

Since gpbackup_file_info hashes only the main segment, its hash is unchanged across those two states, so a >1GB append-mostly table is judged "unchanged" and skipped from the incremental, losing the new rows. The size fallback does not help, because a full segment's size is pinned at exactly 1GB.

The window is narrow (table >1GB, changes land only in tail segments, main segment untouched by UPDATE/VACUUM/hint-bits), but that is exactly the append-only large-table pattern incremental backup most wants to optimize.

Suggestion: stat every segment of the relation (path, path.1, path.2, … until pg_stat_file returns ENOENT, the way md.c walks segments) and fold each segment's mtime+size into the hash — or document the limitation / skip --heap-file-hash for tables over 1GB.

[liang8283]

Thanks for tracking this down to md.c / relpath.c. Fixed in fbd2786: gpbackup_file_info now walks path, path.1, path.2, ... until pg_stat_file raises undefined_file, the same way md.c does, and folds each segment's mtime and size into the result. The relation path is part of the hash too, so a relfilenode change (TRUNCATE / VACUUM FULL / CLUSTER) is also detected.

I reproduced your scenario end to end: after a full backup, I appended to a >1GB table so that only .1 grew, while the first segment kept its size (exactly 1GB) and mtime. The table is now included in the incremental, and the restored row count matches the source.

[talmacschen-arch]

NOT IN (SELECT inhrelid FROM pg_inherits) excludes the leaf partitions of partitioned heap tables, so --heap-file-hash is effectively a no-op for them. (Same clause in the pre-GP7 branch above.)

Verified against source and on a live cluster.

Partition children — whether created via CREATE … PARTITION OF or ATTACH PARTITION — are registered in pg_inherits as inhrelid by design (commands/tablecmds.c: StoreCatalogInheritance(…, stmt->partbound != NULL), and StoreCatalogInheritance1 on the attach path). So every heap leaf partition is an inhrelid and is filtered out here.

Live reproduction: a range-partitioned heap table with 3 leaves — the leaves are relkind='r', amname='heap', and all present in pg_inherits.inhrelid. Running this exact getHeapTables query returns only a non-partitioned control table; all 3 leaves are excluded (the root is already excluded by relkind='r').

With leaves absent from IncrementalMetadata.Heap, FilterTablesForIncremental sees isHeap=false and falls back to "always include", so leaf partitions are always re-backed up. This is safe (over-backup, not data loss), but it means the feature does not apply to partitioned heap tables. Worth calling out — note the root partition was already excluded by relkind='r', so this clause's net effect is to drop the leaves, which are the tables that actually carry data under --leaf-partition-data.

[liang8283]

Agreed. The NOT IN (SELECT inhrelid FROM pg_inherits) clause is removed from both branches, so leaf partitions are hashed like any other heap table. Verified with a range-partitioned heap table: after modifying one leaf, only that leaf was included in the incremental, and its siblings were skipped.

[talmacschen-arch]

Requesting changes on the strength of the correctness issue in my detailed review above (F1).

On heap tables larger than 1GB, --heap-file-hash stats only the first 1GB segment (pg_relation_filepath never returns the .N tail segments — confirmed in md.c/relpath.c), so append-only changes that land in tail segments are silently missed and dropped from the incremental. I reproduced this on a live cluster: after appending 0.5GB to a >1GB table, the main segment's size|mtime was byte-for-byte unchanged while the data went to .1. That is a data-loss path for the feature.

Please either hash all segments of the relation (walk path, path.1, path.2, … like md.c does) or gate/document the flag for tables over 1GB before merge. F2 (leaf partitions excluded) and the missing unit tests are non-blocking. Happy to re-review once segment handling is addressed — the overall approach is good.

[talmacschen-arch]

@liang8283 heads-up — I've reviewed this PR and left a Request Changes.

The one blocker is a correctness issue (F1): on heap tables larger than 1GB, --heap-file-hash stats only the first 1GB segment, because pg_relation_filepath never returns the .N tail segments (confirmed in md.c / relpath.c). I reproduced it on a live cluster — after appending 0.5GB to a >1GB table, the main segment's size|mtime was byte-for-byte unchanged while the new data went to .1, so an append-mostly large table would be silently skipped from the incremental. A fix suggestion (walk all segments like md.c does) is in the inline comment.

F2 (leaf partitions of partitioned heap tables are excluded via NOT IN inhrelid) and the missing unit tests for the new hash branches are non-blocking. The overall approach is solid — happy to re-review once the segment handling is sorted out. Thanks!

[woblerr]

Preferably splitting --heap-file-hash and --ao-file-hash into two separate PRs. They are independent features, so each can be reviewed and merged on its own.

The --heap-file-hash functionality looks like the most useful change.

[woblerr]

The heap filter only compares file hashes, so it misses schema changes that do not rewrite the table. After a full backup with --heap-file-hash, dropping a non-key column left the heap hash unchanged in my test.
The incremental skipped the table and reused the previous data backup, whose COPY column list still included the dropped column.
Restore failed with column "payload" of relation "t" does not exist.

[liang8283]

Thanks for the repro. HeapEntry now records the table's last DDL timestamp from pg_stat_last_operation (read inside the backup snapshot), and a heap table is skipped only if both the file hash and the DDL timestamp match the previous backup, the same rule AO tables already follow. With your DROP COLUMN scenario, the table is now included in the incremental and the restore succeeds with the new column list.

[woblerr]

The heap hash can include writes that are not visible to the backup snapshot.

Holding a table lock, starting a full backup, and committing an insert after the backup established its snapshot but before its checkpoint and hash collection.
The full backup omitted the inserted row, while the next incremental saw the same hash and skipped the table.
Restore completed successfully but returned only id = 1, although the source contained both id = 1 and id = 2.

• Setup in a fresh test database

CREATE TABLE public.t(id int)
USING heap WITH (autovacuum_enabled=false)
DISTRIBUTED BY (id);

INSERT INTO public.t VALUES (1);
SELECT * FROM public.t;
CHECKPOINT;

• Session A — hold the table lock

BEGIN;
LOCK TABLE public.t IN ACCESS EXCLUSIVE MODE;

• Another terminal — start the full backup

gpbackup --dbname test \
  --leaf-partition-data --heap-file-hash 
  ...

• Session A — after backup is waiting at Acquiring ACCESS SHARE locks on tables

INSERT INTO public.t VALUES (2);
COMMIT;

• After the full backup finishes — incremental using its printed timestamp

gpbackup 
  --leaf-partition-data --heap-file-hash \
  --incremental --from-timestamp '<FULL_TIMESTAMP>' \
...

• Restore using the incremental timestamp

gprestore --backup-dir /tmp/review98-backups \
  --timestamp '<INCREMENTAL_TIMESTAMP>' \
...

[liang8283]

Good catch. This was a real ordering bug: the hash was read after the backup snapshot was established. Heap file hashes are now collected in DoSetup on a separate connection (CHECKPOINT, then hash) before initializeConnectionPool begins the transactions and takes the snapshot. Any change committed after the hash is read moves the data files on, so the next incremental sees a different hash. At worst that is an extra copy, never a skipped change.

I ran your exact sequence (lock held by session A, full backup waiting on ACCESS SHARE, insert committed in the meantime): the incremental now includes the table, and the restore returns both id = 1 and id = 2.

While looking at this I also tightened two related cases. If CHECKPOINT fails, no hashes are recorded. And because pg_stat_file reports mtime in whole seconds, a table whose file was modified in the same second as the hash read gets no hash. In both cases the table is included in the next incremental.

[liang8283]

Thanks both for the careful reviews and the reproductions. I pushed fbd2786, which addresses all of them; details are in the inline replies, and the PR description is updated.

• @talmacschen-arch: F1 (tail segments) and F2 (leaf partitions) are fixed. On the other points:
  • Unit tests now cover the heap branches of FilterTablesForIncremental: unchanged, hash changed, DDL changed, and hash missing in either backup.
  • gpbackup_file_info is recreated on every run, so a stale body is never reused.
  • The helper now only treats a missing segment file as "end of relation"; any other error surfaces as a warning, and the table is included rather than silently hashed.
  • The CHECKPOINT cost is now stated in the flag help and the PR description.
• @woblerr: both issues (DDL without a rewrite, and changes committed between snapshot and hash) are fixed, and your reproductions pass. As you suggested, I split the feature: this PR now contains only --heap-file-hash, and --ao-file-hash will come as a separate PR.

Could you take another look when you have a moment?

[liang8283]

A follow-up on --ao-file-hash: I said earlier that it would come as a separate PR, but I'm dropping it instead. Looking at it again before the split, I found it is not safe, and on Cloudberry it brings no benefit over the existing modcount check:

• DELETE is invisible to the aoseg hash. Deleting rows from an AO/AOCS table only updates the visimap; segno, eof, tupcount and vpinfo in pg_aoseg / pg_aocsseg are unchanged, while modcount does increase. The hash replaced the modcount + DDL-timestamp comparison, so an AO table that only saw DELETEs would be skipped, and a restore would bring the deleted rows back. Dropping the DDL timestamp would also reintroduce the DROP COLUMN issue raised above, this time for AO tables.
• No partition-granularity gain on Cloudberry. The motivation was GP5, where a parent's modcount propagates to sibling partitions. On Cloudberry each leaf partition's modcount is already independent: inserting into one leaf leaves its siblings' modcount unchanged, so the current check already works per leaf.
• The GP6 AOCS branch read columns that pg_aocsseg does not have (column_num, physical_segno, eof_uncompressed), so that query would always fail on GP6.

Making it correct would mean adding modcount (or the visimap) back into the comparison, which leaves nothing to gain over today's behavior. So this PR stays --heap-file-hash only, and no AO follow-up is planned.

[woblerr]

@liang8283
There is another issue. Please, check it out.

The full backup hashes the heap file after CHECKPOINT flushes an uncommitted insert, but COPY cannot see that row. Committing the transaction later leaves the hash unchanged, so the next incremental skips the table.

In psql, create a heap table with one committed row, then leave the second insert uncommitted.

CREATE TABLE t(id int) USING heap DISTRIBUTED BY (id);
INSERT INTO t VALUES (1);
BEGIN;
INSERT INTO t VALUES (2);

In another terminal, run a full backup with --leaf-partition-data --heap-file-hash. Once it finishes, run COMMIT; in the original psql session.

Run an incremental from that full backup with --heap-file-hash and restore it. The restore succeeds, but SELECT id FROM t ORDER BY id returns only 1.

Repeating the incremental from the same full backup with --heap-file-hash disabled restores both 1 and 2.

[liang8283]

@woblerr Thanks, confirmed and fixed in b60ef8f. The heap files also contain rows of transactions that are still in progress, and their COMMIT only updates the commit log, so the hash never changed after the commit.

The fix: after reading the file hashes, and before the backup snapshot is taken, gpbackup now checks pg_locks and drops the hash of every table on which another transaction holds a lock taken by data-modifying commands (RowExclusiveLock and stronger). That lock is held until the writer's transaction ends, so the ordering covers both cases:

• the writer commits before the snapshot: its rows are visible to the backup;
• the writer is still open: it still holds the lock at the check, so the table gets no hash and the next incremental includes it.

If the lock check itself fails, no heap hashes are recorded at all.

With your exact steps (uncommitted insert during the full backup, committed after it), the incremental now includes t and the restore returns both 1 and 2. I also ran the same sequence with an uncommitted UPDATE and an uncommitted DELETE (which take ExclusiveLock with the global deadlock detector off): before this fix both were skipped by the incremental and restored the pre-commit values; now the table is included and the restore matches the source. The earlier scenarios (tail segments, snapshot race, DROP COLUMN, leaf partitions) still behave as before.

[woblerr]

@woblerr Thanks, confirmed and fixed in b60ef8f. The heap files also contain rows of transactions that are still in progress, and their COMMIT only updates the commit log, so the hash never changed after the commit.

The fix: after reading the file hashes, and before the backup snapshot is taken, gpbackup now checks pg_locks and drops the hash of every table on which another transaction holds a lock taken by data-modifying commands (RowExclusiveLock and stronger). That lock is held until the writer's transaction ends, so the ordering covers both cases:

* the writer commits before the snapshot: its rows are visible to the backup;

* the writer is still open: it still holds the lock at the check, so the table gets no hash and the next incremental includes it.

If the lock check itself fails, no heap hashes are recorded at all.

With your exact steps (uncommitted insert during the full backup, committed after it), the incremental now includes t and the restore returns both 1 and 2. I also ran the same sequence with an uncommitted UPDATE and an uncommitted DELETE (which take ExclusiveLock with the global deadlock detector off): before this fix both were skipped by the incremental and restored the pre-commit values; now the table is included and the restore matches the source. The earlier scenarios (tail segments, snapshot race, DROP COLUMN, leaf partitions) still behave as before.

Thanks, the original case is fixed.
During the one-second wait after the incremental’s CHECKPOINT, I inserted id = 2 into a heap table and an AO table in the same transaction and committed.
By the time gpbackup checked pg_locks, the locks were gone and the heap file hash still matched the full backup. Restore succeeded, but only the AO table contained id = 2.

Could you check it out, please.

[woblerr]

A full backup of a heap table records the enum value pending. Running ALTER TYPE state RENAME VALUE 'pending' TO 'queued' leaves both the heap file hash and the table’s DDL timestamp unchanged, so the incremental skips the table. During restore, the enum contains queued, but COPY reads pending from the full backup and fails with invalid input value for enum public.state: "pending". An incremental from the same full backup without --heap-file-hash restores the row as queued.

[liang8283]

Confirmed and fixed in 9ef673a: each heap entry now also records a hash of the labels of the enum types the table uses (directly or through domains, arrays, composite and range types), and the table is skipped only if that matches too. With your scenario the table is included and restored as queued. Details in #98 (comment).

[liang8283]

@woblerr Thanks, both confirmed and fixed in 9ef673a.

Commit during the wait after CHECKPOINT. A row written after the CHECKPOINT can stay in shared buffers, so the file hash read a moment later does not see it, and once the writer commits before the lock check there is no lock left to catch it either. The hash is now verified after the backup snapshot is taken: gpbackup issues a second CHECKPOINT, re-reads the file info of every hashed table, and drops the hash of any table whose files changed since the first read. Any change committed before the snapshot is flushed by that second CHECKPOINT, so it can no longer hide behind a matching hash. The pre-snapshot lock check stays, since it covers rows written before the first CHECKPOINT and committed after the snapshot.

With your sequence (heap and AO insert committed in one transaction during the one-second wait), the incremental now includes the heap table and the restore returns the new row in both tables. To reproduce it reliably the row has to land in an existing page; a new row that extends the file moves its mtime immediately and was already caught.

ALTER TYPE ... RENAME VALUE. Enum values are stored by oid, so renaming a label changes the dumped text without touching the data files or the table's DDL timestamp. Each heap entry now also records a hash of the labels of the enum types its columns use, directly or through domains, arrays, composite types and range types, read inside the backup snapshot. A heap table is skipped only if the file hash, the DDL timestamp and this enum-label hash all match. With your scenario the table is included and restored as queued; I also checked a column of a domain-over-enum array and a composite type with an enum attribute, and an unrelated heap table in the same database is still skipped.

The earlier scenarios (uncommitted writes, tail segments, the lock-wait race, DROP COLUMN, leaf partitions) still behave as before.

[woblerr]

@liang8283 Thanks for fixing the previous issues.
Changing the structure of a composite type still leaves the heap file hash, the table’s DDL timestamp and the enum-label hash unchanged, so the incremental skips the table.
Restore then loads records from the full backup into the updated type and fails with malformed record literal. An incremental from the same full backup without --heap-file-hash restores successfully.

Create a test database and table.

CREATE TYPE pair AS (a int, b int);
CREATE TABLE t(id int, v pair)
  USING heap WITH (autovacuum_enabled=false)
  DISTRIBUTED BY (id);

INSERT INTO t VALUES (1, ROW(10,20)::pair);
SELECT * FROM t;

Take a full backup and note its printed timestamp.

Add an attribute to the composite type.

ALTER TYPE pair ADD ATTRIBUTE c int CASCADE;
SELECT * FROM t;
-- (10,20,)

Take an incremental using the full backup’s timestamp.
Restore using the incremental’s timestamp:

ERROR: malformed record literal: "(10,20)"

[liang8283]

@woblerr Thanks, confirmed. Fixed in e90efe1. The enum fix only covered one case of a wider problem: restore loads a skipped table's old data file into the newest column list, and some changes alter that list or how stored values are dumped without touching the data files or the table's DDL timestamp. Besides ALTER TYPE ... ADD/DROP ATTRIBUTE, the same happens when a column is added to a table whose row type another table uses as a column type, and when DROP DOMAIN ... CASCADE drops a column.

The enum-label hash is replaced by a hash of the table's own columns (number, name, type, typmod, dropped) and of the definitions of every enum and composite type they use, directly or through domains, arrays, nested composite, range and multirange types, read inside the backup snapshot. The table is skipped only if this hash matches too. The query runs under a savepoint; if it fails, no heap table is skipped.

While going over the whole feature again I also fixed a few related cases:

• A table is now skipped only if the previous restore plan has data for it. A table that belonged to an extension in the previous backup (whose data is not backed up) and was then removed from it with ALTER EXTENSION ... DROP TABLE was skipped and restored empty.
• Unlogged tables are never hashed: a regular CHECKPOINT does not write their buffers, so their files may not move when they change.
• Hashes are keyed by oid until they are matched with the table's name inside the snapshot.
• gp_toolkit.gpbackup_file_info is dropped after each hashing step instead of being left in gp_toolkit.

With your steps the incremental now includes t, and the restore returns (10,20,). I also checked DROP ATTRIBUTE ... CASCADE, a composite nested in another composite, an array of a domain over the composite, the table row-type case, DROP DOMAIN ... CASCADE, the extension case, an unlogged table, RENAME COLUMN on a partitioned and on an inheritance parent, a typed table, and a full → incremental → incremental chain: each affected table is included and restored to match the source, and unrelated heap tables are still skipped. The earlier scenarios still behave as before.

[woblerr]

There is one more issue with regclass columns. Could you take a look?

With a heap table storing 'public.target'::regclass, taking a full backup and then running ALTER TABLE public.target RENAME TO renamed leaves the referencing table’s file hash, DDL timestamp and column hash unchanged, so the incremental skips it.
Restore then fails with relation "public.target" does not exist when loading the old reference from the full backup. An incremental from the same full backup without --heap-file-hash restores the reference successfully.

And branch conflicts need to be resolved.