Skip to content

docs(readmes): move READMEs to the new readmes dir - #2

Merged
tuhaihe merged 1 commit into
mainfrom
readmes
Jun 19, 2023
Merged

tuhaihe merged 1 commit into
mainfrom
readmes

Conversation

@tuhaihe

@tuhaihe tuhaihe commented Jun 9, 2023 •

Copy link
Copy Markdown
Member

README fiels changed:

Here are some reminders before you submit the pull request

  • Add tests for the change
  • Document changes
  • Communicate in the mailing list if needed
  • Pass make installcheck
  • Review a PR in return to support the community

@tuhaihe
tuhaihe requested a review from my-ship-it June 9, 2023 02:55
@tuhaihe
tuhaihe merged commit 66124c3 into main Jun 19, 2023
@tuhaihe
tuhaihe deleted the readmes branch June 19, 2023 02:08
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Dec 20, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so apache#2 is a reasonable defense,
and apache#3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Dec 22, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so apache#2 is a reasonable defense,
and apache#3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Dec 24, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so apache#2 is a reasonable defense,
and apache#3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
my-ship-it pushed a commit that referenced this pull request Dec 24, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so #2 is a reasonable defense,
and #3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Dec 24, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so apache#2 is a reasonable defense,
and apache#3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Dec 24, 2024
We've heard a couple of reports of people having trouble with
multi-gigabyte-sized query-texts files.  It occurred to me that on
32-bit platforms, there could be an issue with integer overflow
of calculations associated with the total query text size.
Address that with several changes:

1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX.
The hashtable code will bound it to that anyway unless "long"
is 64 bits.  We still need overflow guards on its use, but
this helps.

2. Add a check to prevent extending the query-texts file to
more than MaxAllocHugeSize.  If it got that big, qtext_load_file
would certainly fail, so there's not much point in allowing it.
Without this, we'd need to consider whether extent, query_offset,
and related variables shouldn't be off_t not size_t.

3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit
arithmetic on all platforms.  It appears possible that under duress
those multiplications could overflow 32 bits, yielding a false
conclusion that we need to garbage-collect the texts file, which
could lead to repeatedly garbage-collecting after every hash table
insertion.

Per report from Bruno da Silva.  I'm not convinced that these
issues fully explain his problem; there may be some other bug that's
contributing to the query-texts file becoming so large in the first
place.  But it did get that big, so apache#2 is a reasonable defense,
and apache#3 could explain the reported performance difficulties.

(See also commit 8bbe4cb, which addressed some related bugs.
The second Discussion: link is the thread that led up to that.)

This issue is old, and is primarily a problem for old platforms,
so back-patch.

Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com
Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Jan 2, 2025
We used to rename index-backed constraints in the EXCHANGE PARTITION
command in 6X. Now we don't. We've decided to keep that behavior
in 7X after looking into the opposing arguments:

Argument apache#1. It might cause problem during upgrade.
- Firstly, we won't be using legacy syntax in the dump scripts so we
just need to worry about the existing tables produced by EXCHANGE
PARTITION. I.e. whether or not they can be restored correctly.
- For upgrading from 6X->7X, since those tables already have matched
constraint and index names with the table names, we should be OK.
- For upgrading 7X->onward, since we implement EXCHANGE PARTIITON
simply as a combination of upstream-syntax commands (see
AtExecGPExchangePartition()), pg_upgrade should be able to handle
them. We've verified that manually and the automated test should
cover that too. In fact, this gives another point that we shouldn't
do our own hacky things as part of EXCHANGE PARTITION which might
confuse pg_upgrade.

Argument apache#2. It might surprise the users and their existing workloads.
- The indexed constraint names are all implicitly generated and
shouldn't be directly used in most cases.
- They are not the only thing that will appear changed. E.g. the
normal indexes (e.g. B-tree ones) will be too. So the decision to
change one should be made with changing all of them.

More details see: https://docs.google.com/document/d/1enJdKYxkk5WFRV1WoqIgLgRxCGxOqI2nglJVE_Wglec
yjhjstz pushed a commit that referenced this pull request Jan 2, 2025
We used to rename index-backed constraints in the EXCHANGE PARTITION
command in 6X. Now we don't. We've decided to keep that behavior
in 7X after looking into the opposing arguments:

Argument #1. It might cause problem during upgrade.
- Firstly, we won't be using legacy syntax in the dump scripts so we
just need to worry about the existing tables produced by EXCHANGE
PARTITION. I.e. whether or not they can be restored correctly.
- For upgrading from 6X->7X, since those tables already have matched
constraint and index names with the table names, we should be OK.
- For upgrading 7X->onward, since we implement EXCHANGE PARTIITON
simply as a combination of upstream-syntax commands (see
AtExecGPExchangePartition()), pg_upgrade should be able to handle
them. We've verified that manually and the automated test should
cover that too. In fact, this gives another point that we shouldn't
do our own hacky things as part of EXCHANGE PARTITION which might
confuse pg_upgrade.

Argument #2. It might surprise the users and their existing workloads.
- The indexed constraint names are all implicitly generated and
shouldn't be directly used in most cases.
- They are not the only thing that will appear changed. E.g. the
normal indexes (e.g. B-tree ones) will be too. So the decision to
change one should be made with changing all of them.

More details see: https://docs.google.com/document/d/1enJdKYxkk5WFRV1WoqIgLgRxCGxOqI2nglJVE_Wglec
avamingli pushed a commit that referenced this pull request Jan 23, 2025
## Problem
An error occurs in python lib when a plpython function is executed.
After our analysis, in the user's cluster, a plpython UDF 
was running with the unstable network, and got a timeout error:
`failed to acquire resources on one or more segments`.
Then a plpython UDF was run in the same session, and the UDF
failed with GC error.

Here is the core dump:
```
2023-11-24 10:15:18.945507 CST,,,p2705198,th2081832064,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
    #0 0x7f7c68b6d55b in frame_dealloc /home/cc/repo/cpython/Objects/frameobject.c:509:5
    #1 0x7f7c68b5109d in gen_send_ex /home/cc/repo/cpython/Objects/genobject.c:108:9
    #2 0x7f7c68af9ddd in PyIter_Next /home/cc/repo/cpython/Objects/abstract.c:3118:14
    #3 0x7f7c78caa5c0 in PLy_exec_function /home/cc/repo/gpdb6/src/pl/plpython/plpy_exec.c:134:11
    #4 0x7f7c78cb5ffb in plpython_call_handler /home/cc/repo/gpdb6/src/pl/plpython/plpy_main.c:387:13
    #5 0x562f5e008bb5 in ExecMakeTableFunctionResult /home/cc/repo/gpdb6/src/backend/executor/execQual.c:2395:13
    #6 0x562f5e0dddec in FunctionNext_guts /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:142:5
    #7 0x562f5e0da094 in FunctionNext /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:350:11
    #8 0x562f5e03d4b0 in ExecScanFetch /home/cc/repo/gpdb6/src/backend/executor/execScan.c:84:9
    #9 0x562f5e03cd8f in ExecScan /home/cc/repo/gpdb6/src/backend/executor/execScan.c:154:10
    #10 0x562f5e0da072 in ExecFunctionScan /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:380:9
    #11 0x562f5e001a1c in ExecProcNode /home/cc/repo/gpdb6/src/backend/executor/execProcnode.c:1071:13
    #12 0x562f5dfe6377 in ExecutePlan /home/cc/repo/gpdb6/src/backend/executor/execMain.c:3202:10
    #13 0x562f5dfe5bf4 in standard_ExecutorRun /home/cc/repo/gpdb6/src/backend/executor/execMain.c:1171:5
    #14 0x562f5dfe4877 in ExecutorRun /home/cc/repo/gpdb6/src/backend/executor/execMain.c:992:4
    #15 0x562f5e857e69 in PortalRunSelect /home/cc/repo/gpdb6/src/backend/tcop/pquery.c:1164:4
    #16 0x562f5e856d3f in PortalRun /home/cc/repo/gpdb6/src/backend/tcop/pquery.c:1005:18
    #17 0x562f5e84607a in exec_simple_query /home/cc/repo/gpdb6/src/backend/tcop/postgres.c:1848:10
```

## Reproduce
We can use a simple procedure to reproduce the above problem:
- set timeout GUC: `gpconfig -c gp_segment_connect_timeout -v 5` and `gpstop -ari`
- prepare function:
```
CREATE EXTENSION plpythonu;
CREATE OR REPLACE FUNCTION test_func() RETURNS SETOF int AS
$$
plpy.execute("select pg_backend_pid()")

for i in range(0, 5):
    yield (i)

$$ LANGUAGE plpythonu;
```
- exit from the current psql session.
- stop the postmaster of segment: `gdb -p "the pid of segment postmaster"`
- enter a psql session.
- call `SELECT test_func();` and get error
```
gpadmin=# select test_func();
ERROR:  function "test_func" error fetching next item from iterator (plpy_elog.c:121)
DETAIL:  Exception: failed to acquire resources on one or more segments
CONTEXT:  Traceback (most recent call last):
PL/Python function "test_func"
```
- quit gdb and make postmaster runnable.
- call  `SELECT test_func();` again and get panic
```
gpadmin=# SELECT test_func();
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!> 
```

## Analysis
- There is an SPI call in test_func(): `plpy.execute()`. 
- Then coordinator will start a subtransaction by PLy_spi_subtransaction_begin();
- Meanwhile, if the segment cannot receive the instruction from the coordinator,
  the subtransaction beginning procedure return fails.
- BUT! The Python processor does not know whether an error happened and
  does not clean its environment.
- Then the next plpython UDF in the same session will fail due to the wrong
  Python environment.

## Solution
- Use try-catch to catch the exception caused by PLy_spi_subtransaction_begin()
- set the python error indicator by PLy_spi_exception_set()


Co-authored-by: Chen Mulong <chenmulong@gmail.com>
reshke added a commit that referenced this pull request Sep 19, 2025
leborchuk added a commit that referenced this pull request Sep 11, 2026
Upstream PostgreSQL commit "Remove configure probe for sockaddr_in6 and
require AF_INET6." (bcc8b14) deleted the HAVE_IPV6 probe and stripped the
#ifdef HAVE_IPV6 guards from its own code, since AF_INET6 is now always
available. Cloudberry-specific code still gated IPv6 handling behind
#ifdef HAVE_IPV6, so after the PG16 merge those blocks became dead code:
HAVE_IPV6 is never defined on non-Windows builds.

The practical effect: on an IPv6-only cluster, getDnsCachedAddress() never
populates its cache entry (the IPv6 branch was compiled out), then returns
e->hostinfo with e == NULL -- a bogus non-NULL pointer (offsetof key[]) --
which the caller passes to pstrdup(), crashing in strlen(). This shows up
as a coordinator/FtsProbe SIGSEGV:

  #0 __strlen_evex
  #1 MemoryContextStrdup
  #2 getCdbComponentInfo
  #3 cdbcomponent_getCdbComponents
  #4 FtsProbeMain

Remove the leftover #ifdef HAVE_IPV6 guards so the IPv6 paths compile
unconditionally, matching what upstream did to its own files. Also guard
the cache return against a NULL entry so an unresolvable segment logs a
clean "cannot resolve network address" error instead of segfaulting.

Files: cdbutil.c (both getDnsCachedAddress copies), auth.c, and the
interconnect listener setup (ic_common.c, ic_tcp.c, ic_udpifc.c).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
tuhaihe pushed a commit that referenced this pull request Oct 28, 2026
UBSan reported an unaligned uint16 load while parsing tuple chunks.
Copy the length into an aligned local with memcpy before adding the
header size, preserving the existing native-endian interpretation.

Sanitizer report (UBSan, instrumented build, distributed query with
Motion traffic over UDP):
  ic_udpifc.c:4572:39: runtime error: load of misaligned address
  0x52500016fa59 for type 'uint16', which requires 2 byte alignment
  #0 RecvTupleChunkUDPIFC ic_udpifc.c:4572
  #1 receiveChunksUDPIFCLoop ic_udpifc.c:4462
  #2 receiveChunksUDPIFC ic_udpifc.c:4406
  #3 RecvTupleChunkFromAnyUDPIFC ic_udpifc.c:4739
  #4 processIncomingChunks cdb/cdbmotion.c:653
  #5 RecvTupleFrom cdbmotion.c:607
  #6 execMotionUnsortedReceiver executor/nodeMotion.c:356
Reached whenever bytesProcessed walks the receive buffer past an
even offset while splitting a message into tuple chunks. Benign on
x86_64, but undefined behavior on strictly aligned targets.
reshke added a commit that referenced this pull request Oct 28, 2026
UBSan reported an unaligned uint64_t load in GpMurmurHash64A.
Read each word with memcpy so input need not have uint64_t alignment,
while preserving the little-endian hash computation.

Sanitizer report (UBSan, instrumented build, ANALYZE on a table with
short text values):
  gp_hyperloglog.c:897:11: runtime error: load of misaligned address
  0x529000093515 for type 'uint64_t', which requires 8 byte alignment
  #0 GpMurmurHash64A gp_hyperloglog.c:897
  #1 gp_hll_add_element gp_hyperloglog.c:471
  #2 gp_hyperloglog_add_item gp_hyperloglog.c:788
  #3 compute_scalar_stats commands/analyze.c:3968
  #4 do_analyze_rel commands/analyze.c:971
  #5 analyze_rel_internal commands/analyze.c:429
Triggered by ordinary ndistinct estimation during ANALYZE: short varlena
values hashed via VARDATA_ANY have a payload offset that is not a
multiple of 8. Benign on x86_64, but undefined behavior on strictly
aligned targets.
yjhjstz pushed a commit to yjhjstz/cloudberry that referenced this pull request Oct 28, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864)

Sampling a CO table walks every projected column to the target row using
one shared logical counter, which assumes each column has a datum for
every row.  Missing-mode ADD COLUMN breaks that: the new column's file
starts after its lastrownum, so the walk overshoots and runs off the end
of the segfile.  Take the row number the anchor column produced and fetch
the other columns at that same physical row instead of recomputing a
target per column.

This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md
flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO
table whose added column is *partially* present crashes the segment.

  CREATE TABLE t(a int, b int) USING ao_column;   -- 200000 rows
  ALTER TABLE t ADD COLUMN c int DEFAULT 7;
  INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i;
  ANALYZE t;                                      -- QE dies

The core shows exactly the coordinate mismatch, at the pg_unreachable()
in aocs_gettuple():

  apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339
  apache#3 aocs_get_target_tuple
  apache#4 aoco_acquire_sample_rows
  (gdb) p startrow                                        -> 69893
  (gdb) p phyrow                                          -> 69944
  (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum     -> 66701
  (gdb) p attnum_to_rownum[2 * 128 + segno]                -> 66700

startrow/targrow are logical positions driven by the anchor, phyrow is a
physical row number, and column c's file only begins at row 66701 -- so
walking c by the logical counter looks for a row 66700 rows past its end.

The case upstream's own test covers -- a column added and never written
to afterwards -- does not crash, because then every row is missing and
the check added by 3986f92616 returns the missing value before any walk
happens.  It takes a *partially* present column to reach the walk.

Ported to the PG16 kernel:
- aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that
  difference predates this commit); only upstream's rowstoprocess
  initialisation is taken, which the new branch needs.
- the two restructured call sites in aocs_gettuple() take upstream's
  shape, with cbdb's "goto out" restored -- it has to update the scan's
  tracking variables before returning.
- pgstat_count_buffer_read_ao() was dropped from all three places the
  merge reintroduced it; it belongs to 67da2c369b, which cbdb does not
  carry.

The test additions needed one cbdb-specific correction:
gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 --
cbdb returns an extra stadistinct_array column -- so every
"as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the
file gains it.  Without that even a plain three-column ao_row table fails
with "Returned row contains 7 attributes, but query expects 6".  The
output template was rebuilt from an ao_row run and verified against both
the _row and _column variants.

Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus
ao_partial_scan.  ANALYZE on the reproducer above now reports
reltuples = 210000 exactly, and pg_stats for the text column added the
same way has n_distinct = 2, matching the data.
yjhjstz pushed a commit that referenced this pull request Oct 31, 2026
UBSan reported a misaligned member access when decoding the
distributed-transaction field of commit WAL records during
crash-recovery redo. The WAL record payload has no alignment
guarantee; the neighboring xl_origin branch already copies onto the
stack for exactly this reason. Do the same for xl_xact_distrib.

Sanitizer report (UBSan, instrumented build, crash recovery):
  xactdesc.c:155:34: runtime error: member access within misaligned
  address 0x7b39d104de44 for type 'struct xl_xact_distrib', which
  requires 8 byte alignment
  #0 ParseCommitRecord xactdesc.c:155
  #1 xact_redo xactdesc.c:...
  #2 ApplyWalRecord xlogrecovery.c:2023
  #3 PerformWalRecovery xlogrecovery.c:1845
Undefined behavior at recovery time; harmless on x86_64 but
incorrect on strictly aligned targets.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants