Repository navigation
Conversation
README fiels changed:
my-ship-it
approved these changes
Jun 16, 2023
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Dec 20, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so apache#2 is a reasonable defense, and apache#3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Dec 22, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so apache#2 is a reasonable defense, and apache#3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Dec 24, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so apache#2 is a reasonable defense, and apache#3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
my-ship-it
pushed a commit
that referenced
this pull request
Dec 24, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so #2 is a reasonable defense, and #3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Dec 24, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so apache#2 is a reasonable defense, and apache#3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Dec 24, 2024
We've heard a couple of reports of people having trouble with multi-gigabyte-sized query-texts files. It occurred to me that on 32-bit platforms, there could be an issue with integer overflow of calculations associated with the total query text size. Address that with several changes: 1. Limit pg_stat_statements.max to INT_MAX / 2 not INT_MAX. The hashtable code will bound it to that anyway unless "long" is 64 bits. We still need overflow guards on its use, but this helps. 2. Add a check to prevent extending the query-texts file to more than MaxAllocHugeSize. If it got that big, qtext_load_file would certainly fail, so there's not much point in allowing it. Without this, we'd need to consider whether extent, query_offset, and related variables shouldn't be off_t not size_t. 3. Adjust the comparisons in need_gc_qtexts() to be done in 64-bit arithmetic on all platforms. It appears possible that under duress those multiplications could overflow 32 bits, yielding a false conclusion that we need to garbage-collect the texts file, which could lead to repeatedly garbage-collecting after every hash table insertion. Per report from Bruno da Silva. I'm not convinced that these issues fully explain his problem; there may be some other bug that's contributing to the query-texts file becoming so large in the first place. But it did get that big, so apache#2 is a reasonable defense, and apache#3 could explain the reported performance difficulties. (See also commit 8bbe4cb, which addressed some related bugs. The second Discussion: link is the thread that led up to that.) This issue is old, and is primarily a problem for old platforms, so back-patch. Discussion: https://postgr.es/m/CAB+Nuk93fL1Q9eLOCotvLP07g7RAv4vbdrkm0cVQohDVMpAb9A@mail.gmail.com Discussion: https://postgr.es/m/5601D354.5000703@BlueTreble.com
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Jan 2, 2025
We used to rename index-backed constraints in the EXCHANGE PARTITION command in 6X. Now we don't. We've decided to keep that behavior in 7X after looking into the opposing arguments: Argument apache#1. It might cause problem during upgrade. - Firstly, we won't be using legacy syntax in the dump scripts so we just need to worry about the existing tables produced by EXCHANGE PARTITION. I.e. whether or not they can be restored correctly. - For upgrading from 6X->7X, since those tables already have matched constraint and index names with the table names, we should be OK. - For upgrading 7X->onward, since we implement EXCHANGE PARTIITON simply as a combination of upstream-syntax commands (see AtExecGPExchangePartition()), pg_upgrade should be able to handle them. We've verified that manually and the automated test should cover that too. In fact, this gives another point that we shouldn't do our own hacky things as part of EXCHANGE PARTITION which might confuse pg_upgrade. Argument apache#2. It might surprise the users and their existing workloads. - The indexed constraint names are all implicitly generated and shouldn't be directly used in most cases. - They are not the only thing that will appear changed. E.g. the normal indexes (e.g. B-tree ones) will be too. So the decision to change one should be made with changing all of them. More details see: https://docs.google.com/document/d/1enJdKYxkk5WFRV1WoqIgLgRxCGxOqI2nglJVE_Wglec
yjhjstz
pushed a commit
that referenced
this pull request
Jan 2, 2025
We used to rename index-backed constraints in the EXCHANGE PARTITION command in 6X. Now we don't. We've decided to keep that behavior in 7X after looking into the opposing arguments: Argument #1. It might cause problem during upgrade. - Firstly, we won't be using legacy syntax in the dump scripts so we just need to worry about the existing tables produced by EXCHANGE PARTITION. I.e. whether or not they can be restored correctly. - For upgrading from 6X->7X, since those tables already have matched constraint and index names with the table names, we should be OK. - For upgrading 7X->onward, since we implement EXCHANGE PARTIITON simply as a combination of upstream-syntax commands (see AtExecGPExchangePartition()), pg_upgrade should be able to handle them. We've verified that manually and the automated test should cover that too. In fact, this gives another point that we shouldn't do our own hacky things as part of EXCHANGE PARTITION which might confuse pg_upgrade. Argument #2. It might surprise the users and their existing workloads. - The indexed constraint names are all implicitly generated and shouldn't be directly used in most cases. - They are not the only thing that will appear changed. E.g. the normal indexes (e.g. B-tree ones) will be too. So the decision to change one should be made with changing all of them. More details see: https://docs.google.com/document/d/1enJdKYxkk5WFRV1WoqIgLgRxCGxOqI2nglJVE_Wglec
avamingli
pushed a commit
that referenced
this pull request
Jan 23, 2025
## Problem
An error occurs in python lib when a plpython function is executed.
After our analysis, in the user's cluster, a plpython UDF
was running with the unstable network, and got a timeout error:
`failed to acquire resources on one or more segments`.
Then a plpython UDF was run in the same session, and the UDF
failed with GC error.
Here is the core dump:
```
2023-11-24 10:15:18.945507 CST,,,p2705198,th2081832064,,,,0,,,seg-1,,,,,"LOG","00000","3rd party error log:
#0 0x7f7c68b6d55b in frame_dealloc /home/cc/repo/cpython/Objects/frameobject.c:509:5
#1 0x7f7c68b5109d in gen_send_ex /home/cc/repo/cpython/Objects/genobject.c:108:9
#2 0x7f7c68af9ddd in PyIter_Next /home/cc/repo/cpython/Objects/abstract.c:3118:14
#3 0x7f7c78caa5c0 in PLy_exec_function /home/cc/repo/gpdb6/src/pl/plpython/plpy_exec.c:134:11
#4 0x7f7c78cb5ffb in plpython_call_handler /home/cc/repo/gpdb6/src/pl/plpython/plpy_main.c:387:13
#5 0x562f5e008bb5 in ExecMakeTableFunctionResult /home/cc/repo/gpdb6/src/backend/executor/execQual.c:2395:13
#6 0x562f5e0dddec in FunctionNext_guts /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:142:5
#7 0x562f5e0da094 in FunctionNext /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:350:11
#8 0x562f5e03d4b0 in ExecScanFetch /home/cc/repo/gpdb6/src/backend/executor/execScan.c:84:9
#9 0x562f5e03cd8f in ExecScan /home/cc/repo/gpdb6/src/backend/executor/execScan.c:154:10
#10 0x562f5e0da072 in ExecFunctionScan /home/cc/repo/gpdb6/src/backend/executor/nodeFunctionscan.c:380:9
#11 0x562f5e001a1c in ExecProcNode /home/cc/repo/gpdb6/src/backend/executor/execProcnode.c:1071:13
#12 0x562f5dfe6377 in ExecutePlan /home/cc/repo/gpdb6/src/backend/executor/execMain.c:3202:10
#13 0x562f5dfe5bf4 in standard_ExecutorRun /home/cc/repo/gpdb6/src/backend/executor/execMain.c:1171:5
#14 0x562f5dfe4877 in ExecutorRun /home/cc/repo/gpdb6/src/backend/executor/execMain.c:992:4
#15 0x562f5e857e69 in PortalRunSelect /home/cc/repo/gpdb6/src/backend/tcop/pquery.c:1164:4
#16 0x562f5e856d3f in PortalRun /home/cc/repo/gpdb6/src/backend/tcop/pquery.c:1005:18
#17 0x562f5e84607a in exec_simple_query /home/cc/repo/gpdb6/src/backend/tcop/postgres.c:1848:10
```
## Reproduce
We can use a simple procedure to reproduce the above problem:
- set timeout GUC: `gpconfig -c gp_segment_connect_timeout -v 5` and `gpstop -ari`
- prepare function:
```
CREATE EXTENSION plpythonu;
CREATE OR REPLACE FUNCTION test_func() RETURNS SETOF int AS
$$
plpy.execute("select pg_backend_pid()")
for i in range(0, 5):
yield (i)
$$ LANGUAGE plpythonu;
```
- exit from the current psql session.
- stop the postmaster of segment: `gdb -p "the pid of segment postmaster"`
- enter a psql session.
- call `SELECT test_func();` and get error
```
gpadmin=# select test_func();
ERROR: function "test_func" error fetching next item from iterator (plpy_elog.c:121)
DETAIL: Exception: failed to acquire resources on one or more segments
CONTEXT: Traceback (most recent call last):
PL/Python function "test_func"
```
- quit gdb and make postmaster runnable.
- call `SELECT test_func();` again and get panic
```
gpadmin=# SELECT test_func();
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!>
```
## Analysis
- There is an SPI call in test_func(): `plpy.execute()`.
- Then coordinator will start a subtransaction by PLy_spi_subtransaction_begin();
- Meanwhile, if the segment cannot receive the instruction from the coordinator,
the subtransaction beginning procedure return fails.
- BUT! The Python processor does not know whether an error happened and
does not clean its environment.
- Then the next plpython UDF in the same session will fail due to the wrong
Python environment.
## Solution
- Use try-catch to catch the exception caused by PLy_spi_subtransaction_begin()
- set the python error indicator by PLy_spi_exception_set()
Co-authored-by: Chen Mulong <chenmulong@gmail.com>
12 tasks
1 of 13 tasks
5 of 13 tasks
leborchuk
added a commit
that referenced
this pull request
Sep 11, 2026
Upstream PostgreSQL commit "Remove configure probe for sockaddr_in6 and require AF_INET6." (bcc8b14) deleted the HAVE_IPV6 probe and stripped the #ifdef HAVE_IPV6 guards from its own code, since AF_INET6 is now always available. Cloudberry-specific code still gated IPv6 handling behind #ifdef HAVE_IPV6, so after the PG16 merge those blocks became dead code: HAVE_IPV6 is never defined on non-Windows builds. The practical effect: on an IPv6-only cluster, getDnsCachedAddress() never populates its cache entry (the IPv6 branch was compiled out), then returns e->hostinfo with e == NULL -- a bogus non-NULL pointer (offsetof key[]) -- which the caller passes to pstrdup(), crashing in strlen(). This shows up as a coordinator/FtsProbe SIGSEGV: #0 __strlen_evex #1 MemoryContextStrdup #2 getCdbComponentInfo #3 cdbcomponent_getCdbComponents #4 FtsProbeMain Remove the leftover #ifdef HAVE_IPV6 guards so the IPv6 paths compile unconditionally, matching what upstream did to its own files. Also guard the cache return against a NULL entry so an unresolvable segment logs a clean "cannot resolve network address" error instead of segfaulting. Files: cdbutil.c (both getDnsCachedAddress copies), auth.c, and the interconnect listener setup (ic_common.c, ic_tcp.c, ic_udpifc.c). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 of 2 tasks
This was referenced Oct 26, 2026
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 27, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
tuhaihe
pushed a commit
that referenced
this pull request
Oct 28, 2026
UBSan reported an unaligned uint16 load while parsing tuple chunks. Copy the length into an aligned local with memcpy before adding the header size, preserving the existing native-endian interpretation. Sanitizer report (UBSan, instrumented build, distributed query with Motion traffic over UDP): ic_udpifc.c:4572:39: runtime error: load of misaligned address 0x52500016fa59 for type 'uint16', which requires 2 byte alignment #0 RecvTupleChunkUDPIFC ic_udpifc.c:4572 #1 receiveChunksUDPIFCLoop ic_udpifc.c:4462 #2 receiveChunksUDPIFC ic_udpifc.c:4406 #3 RecvTupleChunkFromAnyUDPIFC ic_udpifc.c:4739 #4 processIncomingChunks cdb/cdbmotion.c:653 #5 RecvTupleFrom cdbmotion.c:607 #6 execMotionUnsortedReceiver executor/nodeMotion.c:356 Reached whenever bytesProcessed walks the receive buffer past an even offset while splitting a message into tuple chunks. Benign on x86_64, but undefined behavior on strictly aligned targets.
reshke
added a commit
that referenced
this pull request
Oct 28, 2026
UBSan reported an unaligned uint64_t load in GpMurmurHash64A. Read each word with memcpy so input need not have uint64_t alignment, while preserving the little-endian hash computation. Sanitizer report (UBSan, instrumented build, ANALYZE on a table with short text values): gp_hyperloglog.c:897:11: runtime error: load of misaligned address 0x529000093515 for type 'uint64_t', which requires 8 byte alignment #0 GpMurmurHash64A gp_hyperloglog.c:897 #1 gp_hll_add_element gp_hyperloglog.c:471 #2 gp_hyperloglog_add_item gp_hyperloglog.c:788 #3 compute_scalar_stats commands/analyze.c:3968 #4 do_analyze_rel commands/analyze.c:971 #5 analyze_rel_internal commands/analyze.c:429 Triggered by ordinary ndistinct estimation during ANALYZE: short varlena values hashed via VARDATA_ANY have a payload offset that is not a multiple of 8. Benign on x86_64, but undefined behavior on strictly aligned targets.
yjhjstz
pushed a commit
to yjhjstz/cloudberry
that referenced
this pull request
Oct 28, 2026
(cherry picked from commit c0730c2f0c58af4a7b12d626c21596e8b4f35864) Sampling a CO table walks every projected column to the target row using one shared logical counter, which assumes each column has a datum for every row. Missing-mode ADD COLUMN breaks that: the new column's file starts after its lastrownum, so the walk overshoots and runs off the end of the segfile. Take the row number the anchor column produced and fetch the other columns at that same physical row instead of recomputing a target per column. This is the hazard docs/gp7-avoid-table-rewrite-commit-correlation.md flagged as the easiest one to miss, and it is real: ANALYZE on an AOCO table whose added column is *partially* present crashes the segment. CREATE TABLE t(a int, b int) USING ao_column; -- 200000 rows ALTER TABLE t ADD COLUMN c int DEFAULT 7; INSERT INTO t SELECT i, i%100, i FROM generate_series(200001,210000) i; ANALYZE t; -- QE dies The core shows exactly the coordinate mismatch, at the pg_unreachable() in aocs_gettuple(): apache#2 aocs_gettuple (targrow=69896) at aocsam.c:1339 apache#3 aocs_get_target_tuple apache#4 aoco_acquire_sample_rows (gdb) p startrow -> 69893 (gdb) p phyrow -> 69944 (gdb) p scan->columnScanInfo.ds[2]->blockFirstRowNum -> 66701 (gdb) p attnum_to_rownum[2 * 128 + segno] -> 66700 startrow/targrow are logical positions driven by the anchor, phyrow is a physical row number, and column c's file only begins at row 66701 -- so walking c by the logical counter looks for a row 66700 rows past its end. The case upstream's own test covers -- a column added and never written to afterwards -- does not crash, because then every row is missing and the check added by 3986f92616 returns the missing value before any walk happens. It takes a *partially* present column to reach the walk. Ported to the PG16 kernel: - aocs_gettuple_column() keeps cbdb's AOTupleId-by-value local (that difference predates this commit); only upstream's rowstoprocess initialisation is taken, which the new branch needs. - the two restructured call sites in aocs_gettuple() take upstream's shape, with cbdb's "goto out" restored -- it has to update the scan's tracking variables before returning. - pgstat_count_buffer_read_ao() was dropped from all three places the merge reintroduced it; it belongs to 67da2c369b, which cbdb does not carry. The test additions needed one cbdb-specific correction: gp_acquire_sample_rows() has NUM_SAMPLE_FIXED_COLS = 4 here, not 3 -- cbdb returns an extra stadistinct_array column -- so every "as (totalrows, totaldeadrows, oversized_cols_length, ...)" list in the file gains it. Without that even a plain three-column ao_row table fails with "Returned row contains 7 attributes, but query expects 6". The output template was rebuilt from an ao_row run and verified against both the _row and _column variants. Verified: AO/AOCS regression 24/24, AO isolation2 136/136 plus ao_partial_scan. ANALYZE on the reproducer above now reports reltuples = 210000 exactly, and pg_stats for the text column added the same way has n_distinct = 2, matching the data.
yjhjstz
pushed a commit
that referenced
this pull request
Oct 31, 2026
UBSan reported a misaligned member access when decoding the distributed-transaction field of commit WAL records during crash-recovery redo. The WAL record payload has no alignment guarantee; the neighboring xl_origin branch already copies onto the stack for exactly this reason. Do the same for xl_xact_distrib. Sanitizer report (UBSan, instrumented build, crash recovery): xactdesc.c:155:34: runtime error: member access within misaligned address 0x7b39d104de44 for type 'struct xl_xact_distrib', which requires 8 byte alignment #0 ParseCommitRecord xactdesc.c:155 #1 xact_redo xactdesc.c:... #2 ApplyWalRecord xlogrecovery.c:2023 #3 PerformWalRecovery xlogrecovery.c:1845 Undefined behavior at recovery time; harmless on x86_64 but incorrect on strictly aligned targets.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
README fiels changed:
Here are some reminders before you submit the pull request
make installcheck