Skip to content

CI: run the parallel_customscan module test - #2248

Open
tuhaihe wants to merge 5 commits into
apache:mainfrom
tuhaihe:ci-parallel-customscan
Open

tuhaihe wants to merge 5 commits into
apache:mainfrom
tuhaihe:ci-parallel-customscan

Conversation

@tuhaihe

@tuhaihe tuhaihe commented Nov 5, 2026

Copy link
Copy Markdown
Member

Fixes #ISSUE_Number

What does this PR do?

Type of Change

  • Bug fix (non-breaking change)
  • New feature (non-breaking change)
  • Breaking change (fix or feature with breaking changes)
  • Documentation update

Breaking Changes

Test Plan

  • Unit tests added/updated
  • Integration tests added/updated
  • Passed make installcheck
  • Passed make -C src/test installcheck-cbdb-parallel

Impact

Performance:

User-facing changes:

Dependencies:

Checklist

Additional Context

CI Skip Instructions


The test steps join make_configs with spaces and then split the result
on spaces.  An entry that contains a space, such as "dir:all
installcheck", becomes two configurations, and the second one has no
directory.

Join the entries with newlines and read one entry per line instead.
All current entries give the same configurations as before.  apache#2237
has the same change.

Assisted-by: Claude Code
CI does not run the test in src/test/modules/parallel_customscan, which
apache#1855 added to check that the segments can run a parallel-aware custom
scan.  So nobody sees when the dispatch of custom scans breaks.

Add an ic-parallel-customscan test job that runs it on Rocky 8, 9 and
10.  The build does not install the module, so the job runs
"install installcheck".

Assisted-by: Claude Code
gen_node_support.pl also writes the binary out and read functions that
the dispatcher uses to send a plan to the segments.  For the methods of
a CustomScan, the binary out function writes the text format:

  appendStringInfoString(str, " :methods ");
  outToken(str, node->methods->CustomName);

But the binary read function reads a length and then the bytes.  So a
QE takes " :me" as the length of the name, and each query that sends a
custom scan to the segments fails:

  ERROR:  invalid memory alloc request size 1701657121

Before the REL_16_9 merge, outfast.c wrote the name with
WRITE_STRING_FIELD.  Write it with WRITE_STRING_VAR again.

Assisted-by: Claude Code
The library registers its custom scan methods when it is loaded.  The
coordinator loads it for CREATE EXTENSION.  The QEs that run the scan
do not, and neither do the QEs that CBDB parallel starts as parallel
workers.  PostgreSQL gives its parallel workers the libraries of the
leader, but the dispatcher does not do that for the QEs.  So a QE
cannot read the plan:

  ERROR:  ExtensibleNodeMethods "ParallelCustomScan" was not registered

Set session_preload_libraries for the test database and connect again,
so that each new QE loads the library.  Reset it at the end.

Assisted-by: Claude Code
In the EXPLAIN ANALYZE of the parallel hash join, a parallel worker can
find no outer tuples left to probe.  Then the outer custom scan of that
worker shows "(never executed)", and the test failed in 1 of 5 runs.
Replace it the same way as the actual row counts.

Assisted-by: Claude Code

@leborchuk leborchuk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, quite interesting, I bumped in a bug in gen_node_support.pl in Anser prototype. Spent some time to understand why it fails. And the issue could be found in tap tests, no need to dig it, just fix it out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants