Repository navigation
Conversation
Build and golangci-lint failed because emit_test.go referenced histogramCountSpy without defining it. Co-authored-by: Cursor <cursoragent@cursor.com>
Record counts on the existing histogramSpy instead of adding a duplicate spy type. Co-authored-by: Cursor <cursoragent@cursor.com>
Port the metric-specific compatibility fix and MySQL/MariaDB regression tests from #56. Co-authored-by: Bastian Bartmann <bastian.bartmann@shopify.com>
Count row events independently from decoded rows. Retire the processed and filtered row counters while preserving all event counters and consumed logical-row counts. Add parser-to-streamer regressions for skipped payloads, original/changelog rows, update pairs, and transaction coordinates. Adapt test fixtures to the upstream streamer API. (cherry picked from commit 6e43dc4)
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Transaction metrics are miscounted or lost across commit grouping, file-position mode, and reconnects, while table tags introduce unbounded cardinality and malformed labels.
7 open issues
Suppress row-count metrics for incomplete transactions · New Measure commit groups after they close · New Skip delivery-time samples when no rows were forwarded · New Emit transaction metrics in file-position mode · New Flush transaction metrics on XID in GTID mode · New Remove unbounded table tags from migration metrics · New Escape reserved delimiters in table metric tags · New
What changed in this PR
Extends the existing StatsD instrumentation with binlog throughput and transaction diagnostics for monitoring migration health, building on #1701.
Changes:
- Adds row-event counters, logical-row counts, and channel-delivery timing.
- Records transaction sizes and MySQL GTID metadata, with MariaDB transaction-boundary handling.
- Adds regression coverage for table filtering, header-only events, UPDATE pairs, and transaction boundaries.
| File | Description |
|---|---|
go/metrics/emit.go |
Adds binlog counter, histogram, and gauge helpers. |
go/metrics/emit_test.go |
Tests metric values, tags, and nil-safe behavior. |
go/binlog/gomysql_reader.go |
Wires metrics into row processing and transaction boundaries. |
go/binlog/gomysql_reader_test.go |
Adds MySQL and MariaDB GTID metric coverage. |
go/binlog/gomysql_reader_metrics_test.go |
Tests decoded and header-filtered row events across tables and transaction boundaries. |
go/binlog/binlog_dml_event.go |
Adds DML metric tags and logical-row counting helpers. |
go/binlog/binlog_dml_event_test.go |
Tests DML tags and logical-row counts, including UPDATE pairs. |
Review effort: Max (auto)
| return | ||
| } | ||
| emit := gmr.migrationContext.Metrics | ||
| metrics.RecordBinlogTransactionSize(emit, gmr.transactionRowEventTotal, gmr.transactionRowTotal) |
Comment on lines
+153
to
+156
| if event.LastCommitted != gmr.previousLastCommitted { | ||
| metrics.RecordUnfilteredCommitGroupSize(emit, float64(event.SequenceNumber-event.LastCommitted)) | ||
| } | ||
| gmr.previousLastCommitted = event.LastCommitted |
Comment on lines
+222
to
+224
| if relevant { | ||
| metrics.RecordBinlogStreamerBlockedOnOutChannel(emit, time.Since(beforeChannel)) | ||
| } |
| if err != nil { | ||
| return err | ||
| } | ||
| gmr.onGTIDEvent(gtidEvent) |
Comment on lines
+291
to
+293
| if !gmr.migrationContext.UseGTIDs { | ||
| gmr.flushTransactionMetrics() | ||
| } |
Comment on lines
+181
to
+190
| emit.Count("binlog_events", 1, "type:processed", "table:"+tableName) | ||
| } | ||
|
|
||
| // RecordBinlogRowsEventFiltered counts a rows event on an unrelated table. | ||
| // Header-only filtering leaves its row count unknown, so no rows are counted. | ||
| func RecordBinlogRowsEventFiltered(emit Emitter, tableName string) { | ||
| if emit == nil || tableName == "" { | ||
| return | ||
| } | ||
| emit.Count("binlog_events", 1, "type:filtered", "table:"+tableName) |
| if emit == nil || tableName == "" || binlogEventType == "" { | ||
| return | ||
| } | ||
| emit.Count("binlog_events", 1, "type:consumed", "table:"+tableName, "binlog_event_type:"+binlogEventType) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Related issue: #1672
Description
Extends the StatsD instrumentation from #1701 with binlog processing metrics:
Respects header-only filtering: unrelated payloads remain undecoded, and unknown row counts are not emitted. UPDATE before/after images count as one logical row. MySQL and MariaDB GTID handling remain compatible.
Regression coverage checks decoded and header-filtered events, original/changelog tables, UPDATE pairs and transaction boundaries.
script/cibuildreturns with no formatting errors, build errors or unit test errors.