Repository navigation
Conversation
pgsync-onehq project into the official pgsync repository.pgsync-onehq project into the official pgsync repository and Fix O(N^2) reindex WAL catch-up with single-pass slot streaming
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
[2026-10-27]
Fix O(N^2) reindex WAL catch-up with single-pass slot streaming
The reindex WAL catch-up paged through the logical slot with
PG_LOGICAL_SLOT_PEEK_CHANGES and a growing SQL OFFSET. The peek
function decodes the whole backlog (restart_lsn..upto_lsn) on every
call and LIMIT/OFFSET are applied only after the set-returning function
materialises its rows, so each page re-decoded the entire backlog:
O(N^2 / chunk_size) work plus repeated temp I/O. On large backlogs this
turned catch-up into hours and tripped the Kubernetes
activeDeadlineSeconds.
Read the slot in a single decode pass via a server-side streaming
cursor (Base.logical_slot_peek_changes_stream), making decoding O(N)
while bounding client memory to one LOGICAL_SLOT_CHUNK_SIZE batch.
Validated in production: ~336k changes per index drained at a flat
~8 ms per 5000-row chunk regardless of offset.
CDC correctness is unchanged: PEEK stays non-destructive, the slot is
consumed exactly once at the end with PG_LOGICAL_SLOT_GET_CHANGES
bounded by the same upto_lsn, and the cursor/advisory lock are closed
without consuming the slot on error so a failed run replays safely
(at-least-once with deterministic document IDs).
Also add WAL_PERFORMANCE_LOGGING (default off): structured timing logs
for the WAL catch-up path, plus integration tests characterising the
slot semantics the fix relies on (marker wal_integration, requires a
PostgreSQL 16 logical-decoding database).
[2026-08-25]
Moved the
pgsync-onehqproject into the officialpgsyncrepository.This PR consolidates the shared PGSync components, scripts, tests, Docker configuration, and supporting files into the main PGSync repository, while excluding application-specific configuration and resources.
The goal is to keep the PGSync core and reusable functionality centralized, while application-specific schemas, plugins, index settings, and related configuration remain within each application repository.
Included
Excluded
Risk Level
Rollback Plan
Please describe and add steps for rollback if needed.
Screenshots
Checklist