Skip to content

fix(ingest): deduplicate events redelivered during rebalance - #603

Open
grootbro wants to merge 5 commits into
Openpanel-dev:mainfrom
grootbro:fix/kafka-rebalance-563
Open

grootbro wants to merge 5 commits into
Openpanel-dev:mainfrom
grootbro:fix/kafka-rebalance-563

Conversation

@grootbro

@grootbro grootbro commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

A rebalance can reject a heartbeat after events have reached Redis but before their Kafka offsets are committed. The old Promise.all also returns while sibling handlers are still running. A second consumer then appends the same events again, even when their producer ids are stable.

This waits for sibling handlers and their durability gate before propagating a batch error. Consumer-produced rows use an atomic Redis enqueue/dedup script. Old envelopes get stable ids from Kafka coordinates; derived session_start rows get separate stable ids. Offsets still resolve only after durability.

Fixes #563.

Reproduction

The regression starts separate Bun worker processes with the real batch handler and EventBuffer, actual Kafka/Redis/ClickHouse, and a controlled synthetic business-event handler. It forces a join during an in-flight batch and records partition/offset independently of the queue. Each case must observe a rebalance and repeat deliveries, then check committed offsets, physical rows, distinct ids and distinct event sequences.

Version / workers Inputs Handler deliveries ClickHouse rows Distinct sequences
Before / 2, producer ids 2,000 3,536 3,536 2,000
After / 2, producer ids 2,000 2,117 2,000 2,000
After / 3, producer ids 2,000 2,147 2,000 2,000
After / 2, legacy envelopes 2,000 2,115 2,000 2,000

The new integration assertion fails on the unchanged base with 3,536 rows. The heartbeat regression also fails on the base. Redis tests cover concurrent writers, retry after a refused enqueue, drained queues, marker expiry, corrupt marker types and project isolation.

Validation

  • 84 focused consumer/handler/event-buffer tests pass; the 3 broker tests also pass when explicitly enabled.
  • Independent Codex review found no actionable P0-P2 defects in the supplied change.
  • Core tsc --noEmit, Biome on all changed TypeScript and git diff --check pass.
  • Recorded runs use native Apache Kafka 3.9.1 (KRaft, initial rebalance delay 0), Redis 7.2.5 with AOF/always fsync and ClickHouse 26.1.3.52. This is a single-node fixture, not the original Redpanda deployment or a full API/session/Postgres run.
  • packages/core/test/kafka-rebalance.md documents the opt-in test and a dedicated Compose stack using prebuilt images. Compose configuration was validated; the image-based stack was not run. No application builds were needed.

Delivery limits / upgrade

Dedup markers expire after one hour: Redis retains one extra key per distinct consumer-produced row in that window. Replays after expiry, lost/evicted Redis state, and the existing ClickHouse-insert → Redis-trim crash window remain at-least-once. During rolling upgrades, old workers still append unconditionally; full protection starts after all consumers upgrade. No migration or environment change is required.

Summary by CodeRabbit

  • Bug Fixes
    • Improved event handling during consumer rebalances by waiting for in-flight work before reporting failures.
    • Prevented duplicate event redeliveries across workers while allowing retries when queue writes are refused. Deduplication markers expire after one hour.
    • Improved identity handling for legacy events without IDs and derived session-start events.
  • Tests
    • Added coverage for redelivery, deduplication, and event identity, plus an opt-in Kafka rebalance integration test.
  • Documentation
    • Added guidance on running rebalance checks and on delivery guarantees and limitations.

Drain sibling handlers before returning a failed Kafka batch and atomically enqueue consumer events with expiring Redis deduplication markers. Preserve legacy envelope identity and derived session-start identity across redelivery.

Self-hosted workers retain one additional Redis key per consumer-produced event for one hour. No schema migration or environment change is required; mixed-version deployments gain full protection after all workers upgrade.
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6a191f33-20fd-4a25-a1dc-148186d12fff
📥 Commits

Reviewing files that changed from the base of the PR and between 0b09797 and f70866c.

📒 Files selected for processing: 3
  • packages/core/src/buffers/event-buffer.ts
  • packages/core/src/modules/ingest/src/consumer.ts
  • packages/core/src/modules/ingest/src/incoming-event-handler.ts
🚧 Files skipped from review as they are similar to previous changes: 3
  • packages/core/src/modules/ingest/src/consumer.ts
  • packages/core/src/modules/ingest/src/incoming-event-handler.ts
  • packages/core/src/buffers/event-buffer.ts

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 6 remain after this review.

📝 Walkthrough

Ingestion assigns stable IDs to legacy and derived events, deduplicates eligible redeliveries in Redis, and settles in-flight batch work before offset resolution. Unit tests and an opt-in integration test cover Redis behavior, rebalance handling, and ClickHouse results.

Changes: Kafka redelivery and rebalance durability

Layer / File(s) Summary
Stable event identities
packages/core/src/modules/ingest/src/consumer.ts, packages/core/src/modules/ingest/src/consumer.test.ts, packages/core/src/modules/ingest/src/incoming-event-handler.ts, packages/core/src/modules/ingest/src/incoming-event-handler.test.ts
Events without IDs receive deterministic IDs based on Kafka delivery coordinates. Derived session_start events receive deterministic IDs based on the triggering event ID and current session ID. Tests cover repeat delivery and producer-supplied IDs.
Redis redelivery deduplication
packages/core/src/buffers/event-buffer.ts, packages/core/src/buffers/event-buffer-redelivery.test.ts
EventBuffer uses a Redis script to validate queue and marker key types, enqueue eligible events, and set project-scoped deduplication markers that expire after one hour. Tests cover retries, expiry, project scoping, and non-redeliverable events.
Drain batch work before offset resolution
packages/core/src/modules/ingest/src/consumer.ts, packages/core/src/modules/ingest/src/consumer.test.ts
The consumer waits for keyed groups to settle and closes the durability window before rethrowing a group failure. It resolves offsets only after a successful flush. Tests cover interruption, sibling work, and unresolved offsets.
End-to-end rebalance test
packages/core/test/kafka-rebalance.compose.yml, packages/core/test/kafka-rebalance-worker.ts, packages/core/src/modules/ingest/src/consumer.rebalance.integration.test.ts, packages/core/test/kafka-rebalance.md
An opt-in integration test runs Kafka workers, records delivery and rebalance traces, and checks buffered events in ClickHouse. The Compose stack and documentation describe test setup, execution, and cleanup.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant KafkaJS
  participant eachBatch
  participant KeyGroups
  participant DurabilityGate
  eachBatch->>KeyGroups: Process grouped messages
  KeyGroups-->>eachBatch: Settle running groups
  eachBatch->>DurabilityGate: Close window and flush
  DurabilityGate-->>eachBatch: Return flush result
  eachBatch->>KafkaJS: Resolve contiguous offsets after successful flush
Loading

Suggested reviewers: lindesvard

Merge Risk: ⚪ Minimal · up to f7086

No concrete merge-blocking issue is established from the supplied evidence; merge after normal checks.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: deduplicating events that are redelivered during Kafka consumer rebalancing.
Linked Issues check Passed [#563] Offsets are not committed before durable effects. In consumer.ts eachBatch, keyed groups run under Promise.allSettled. The durability gate bufferedEventsAreDurable runs after all groups settle.…
Out of Scope Changes check Passed All changed production code maps to #563. consumer.ts changes batch drain ordering, durability ordering, interruption handling and legacy id assignment. event-buffer.ts adds the atomic enqueue/dedup s…
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 8 files.
Full details: Linked Issues check

Explanation

[#563] Offsets are not committed before durable effects. In consumer.ts eachBatch, keyed groups run under Promise.allSettled. The durability gate bufferedEventsAreDurable runs after all groups settle. The code throws the first rejected reason before the ascending offset-resolution loop, so a failed batch resolves no offsets. Sibling handlers finish before the error propagates. EventBuffer.enqueueEvents uses an atomic Lua script that enqueues rows and sets project-scoped dedup markers with a 3600 s TTL in one step. Markers are never written before the row is queued. Legacy envelopes receive a UUID v5 from topic, partition and offset in processMessage. Derived session_start rows receive a stable UUID v5 in incoming-event-handler.ts. The rebalance integration test is opt-in and present in the diff. Its author-reported results were not re-run in this review. Unit tests cover interruption, redelivery, markers and derived IDs. Known limits are at-least-once after marker expiry, Redis state loss and the existing crash window. The PR and documentation state these limits. The session sorting-key issue and EPIPE observation are excluded by the issue and not targeted by the diff.

Full details: Out of Scope Changes check

Explanation

All changed production code maps to #563. consumer.ts changes batch drain ordering, durability ordering, interruption handling and legacy id assignment. event-buffer.ts adds the atomic enqueue/dedup script, project-scoped markers and the readonly queueKey declaration. incoming-event-handler.ts gives derived session_start rows stable ids. The tests, rebalance worker, Compose file and documentation support the same fix. The stacked JSDoc block above processMessage is cosmetic and does not change scope. The diff shows no session sorting-key change and no EPIPE-related change.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@grootbro

Copy link
Copy Markdown
Contributor Author

@coderabbitai run pre-merge checks

@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor
✅ Actions performed

Running Pre-merge checks. Results will be posted in the summary comment once complete.

@grootbro

Copy link
Copy Markdown
Contributor Author

@coderabbitai The latest review covers 0b09797c and reports docstring coverage of 75% across four functions, but does not identify the uncovered function. Please list the exact four function names and paths used in this calculation and identify which one is missing documentation. JSDoc has been added to enqueueEvents, ingestIncomingEvent, the batch/message handlers and test helpers; writeToRedis now also has a description before its return tag. Please check the current head and refresh the pre-merge result if the metric is stale. No code generation or commits are needed.

@coderabbitai

coderabbitai Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -uo pipefail
git rev-parse HEAD
gh pr view 603 --repo Openpanel-dev/openpanel --json headRefOid,baseRefOid,url
gh api --paginate repos/Openpanel-dev/openpanel/issues/603/comments --jq '.[] | select(.user.login | test("coderabbit")) | {id,updated_at,body}'
gh api --paginate repos/Openpanel-dev/openpanel/pulls/603/reviews --jq '.[] | {id,user:.user.login,commit_id,submitted_at,body}'
head_sha=$(gh pr view 603 --repo Openpanel-dev/openpanel --json headRefOid --jq .headRefOid)
gh api "repos/Openpanel-dev/openpanel/commits/$head_sha/check-runs" --jq '.check_runs[] | {id,name,status,conclusion,output}'
for file in AGENTS.md packages/core/AGENTS.md; do
  if [ -f "$file" ]; then printf '\n--- %s ---\n' "$file"; cat "$file"; fi
done
ast-grep outline packages/core/src/buffers/event-buffer.ts --items all
ast-grep outline packages/core/src/modules/ingest/src/consumer.ts --items all
ast-grep outline packages/core/src/modules/ingest/src/incoming-event-handler.ts --items all
fd -H -i 'coderabbit|docstring' . --exclude .git

Length of output: 62911


🏁 Script executed:

#!/bin/bash
set -uo pipefail
if [ -f WORKTREE.md ]; then cat WORKTREE.md; fi
git diff 3a5fbd05f5202a14ce3e224685a50f9cc03d18cb HEAD --stat
git diff --unified=5 3a5fbd05f5202a14ce3e224685a50f9cc03d18cb HEAD -- packages/core/src/buffers/event-buffer.ts packages/core/src/modules/ingest/src/consumer.ts packages/core/src/modules/ingest/src/incoming-event-handler.ts
rg -n -B14 -A14 '(^|[[:space:]])(private async writeToRedis|private async enqueueEvents|async function ingestIncomingEvent|async function processMessage|const eachBatch|async function eachBatch|function createEventsBatchHandler)' packages/core/src/buffers/event-buffer.ts packages/core/src/modules/ingest/src/consumer.ts packages/core/src/modules/ingest/src/incoming-event-handler.ts
ast-grep outline packages/core/src/modules/ingest/src/consumer.rebalance.integration.test.ts --items all
ast-grep outline packages/core/test/kafka-rebalance-worker.ts --items all
ast-grep outline packages/core/src/buffers/event-buffer-redelivery.test.ts --items all
git diff --unified=2 3a5fbd05f5202a14ce3e224685a50f9cc03d18cb HEAD -- packages/core/src/modules/ingest/src/consumer.rebalance.integration.test.ts packages/core/test/kafka-rebalance-worker.ts packages/core/src/buffers/event-buffer-redelivery.test.ts | head -360
gh api repos/Openpanel-dev/openpanel/commits/0b09797c426b0b0b4c7da20d7721786fe83d818f --jq '{sha,commit:.commit.message,files:[.files[]|{filename,patch}]}'

Length of output: 47666


@grootbro I couldn't produce a complete response to this request. Please try again.

You are interacting with an AI system.

@lindesvard

Copy link
Copy Markdown
Contributor

Thanks for digging into this. The redelivery analysis is useful, and the dedup key is right: it only matches true redeliveries, so separate client requests are kept.

We'd like to split this PR, though. In aaf02b1 we removed a per-event Redis marker because of its cost, one command per event and events/s × 120 s keys held in memory. This one brings a marker back with a 1 hour TTL, so about 30× the keys, and the micro-batch enqueue goes from one RPUSH to a Lua script doing several commands per event. The compose template runs Redis with noeviction, so self-hosters would simply hold that memory.

Could you do this instead?

  1. Keep the consumer change as its own PR: waiting for sibling handlers and the durability gate before rethrowing the heartbeat error. We'd take that now.
  2. Try to reduce redelivery at the source. After draining, the handler rethrows without resolving any offsets (consumer.ts), so the whole batch comes back. Resolving the durable contiguous prefix before the rethrow might avoid most of that with no Redis state. We haven't verified that the broker accepts that commit during a rebalance, so please check it with your rebalance test.
  3. Leave the dedup out for now. If kafka_events_reprocessed_total still shows duplicates worth fixing after that, we can look at it again, with a TTL of minutes rather than an hour. Redelivery after a rebalance should arrive well inside that.

One thing to keep in mind if dedup does come back: sessions.ingest and aiEvents.add still run on redelivery, so sessions and ai_events would keep counting the duplicates while events wouldn't.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Duplicate event rows observed during a Kafka consumer-group rebalance with multiple workers

2 participants