Skip to content

Moved the pgsync-onehq project into the official pgsync repository and Fix O(N^2) reindex WAL catch-up with single-pass slot streaming - #3

Open
pabescx wants to merge 3 commits into
OneHQ:mainfrom
pabescx:main
Open

pabescx wants to merge 3 commits into
OneHQ:mainfrom
pabescx:main

Conversation

@pabescx

@pabescx pabescx commented Sep 24, 2026 •

Copy link
Copy Markdown

Description

[2026-10-27]

Fix O(N^2) reindex WAL catch-up with single-pass slot streaming

The reindex WAL catch-up paged through the logical slot with
PG_LOGICAL_SLOT_PEEK_CHANGES and a growing SQL OFFSET. The peek
function decodes the whole backlog (restart_lsn..upto_lsn) on every
call and LIMIT/OFFSET are applied only after the set-returning function
materialises its rows, so each page re-decoded the entire backlog:
O(N^2 / chunk_size) work plus repeated temp I/O. On large backlogs this
turned catch-up into hours and tripped the Kubernetes
activeDeadlineSeconds.

Read the slot in a single decode pass via a server-side streaming
cursor (Base.logical_slot_peek_changes_stream), making decoding O(N)
while bounding client memory to one LOGICAL_SLOT_CHUNK_SIZE batch.
Validated in production: ~336k changes per index drained at a flat
~8 ms per 5000-row chunk regardless of offset.

CDC correctness is unchanged: PEEK stays non-destructive, the slot is
consumed exactly once at the end with PG_LOGICAL_SLOT_GET_CHANGES
bounded by the same upto_lsn, and the cursor/advisory lock are closed
without consuming the slot on error so a failed run replays safely
(at-least-once with deterministic document IDs).

Also add WAL_PERFORMANCE_LOGGING (default off): structured timing logs
for the WAL catch-up path, plus integration tests characterising the
slot semantics the fix relies on (marker wal_integration, requires a
PostgreSQL 16 logical-decoding database).

[2026-08-25]

Moved the pgsync-onehq project into the official pgsync repository.

This PR consolidates the shared PGSync components, scripts, tests, Docker configuration, and supporting files into the main PGSync repository, while excluding application-specific configuration and resources.

The goal is to keep the PGSync core and reusable functionality centralized, while application-specific schemas, plugins, index settings, and related configuration remain within each application repository.

Included

  • Shared PGSync scripts and runtime components
  • Dockerfile and entrypoint
  • Selective reindex functionality
  • Unit and integration tests
  • Shared supporting documentation and configuration

Excluded

  • Application-specific schemas
  • Application-specific plugins
  • Index settings owned by the application
  • Other application-specific configuration and resources

Risk Level

  • Low
  • Med
  • High

Rollback Plan

Please describe and add steps for rollback if needed.

Screenshots

image

Checklist

  • I linked the change to an approved ticket
  • I have added/updated appropriate tests and evidence
  • I considered security/privacy impact
  • I updated docs/runbook if needed
  • I added a changelog/version entry for this PR

@pabescx pabescx changed the title Moved the pgsync-onehq project into the official pgsync repository. Moved the pgsync-onehq project into the official pgsync repository and Fix O(N^2) reindex WAL catch-up with single-pass slot streaming Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant