Repository navigation
Skip DSM tag/context building in Kafka consumers when Data Streams is off (dsm - perf quick fix) - #13075
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits intoOct 27, 2026
Conversation
… off The consumer TracingIterators built DataStreamsTags and a DataStreamsContext for every record even with DSM disabled, where the checkpoint and DSM injection are no-ops. Guard the whole DSM block with the same dynamic isDataStreamsEnabled() check the payload-size line already used, so remote-config enablement still takes effect. kafka-consumer-perf-test with DSM off: ~12% less consumer-thread allocation, +6.6% throughput. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dougqh
added
type: bug fix
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
Kafka / producer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics. See unchanged results
|
Kafka / consumer-benchmarkParameters
See matching parameters
SummaryFound 0 performance improvements and 1 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.
See unchanged results
|
dougqh
marked this pull request as ready for review
October 27, 2026 15:00
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
Both Kafka consumer guards preserve DSM-enabled checkpoint and propagation behavior while skipping tag and context construction when DSM is disabled. The completed static review identified no actionable regressions.
🤖 Bits Code Review · Commit fcf4319
KafkaConsumeScenariosBenchmark drains a 500-record poll batch through TracingIterator with a default-configured CoreTracer and no agent. Each record carries the headers this tracer injects on produce, so every record goes through extraction, a consume span and the DSM checkpoint path. dataStreams toggles dd.data.streams.enabled per fork. It measures what the Kafka macro benchmark can't isolate without a broker: ns and bytes per record on the consumer thread. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
More details
Both Kafka consumer guards use the existing dynamic DSM setting, preserving enabled checkpoint and propagation behavior while avoiding tag and context construction when disabled. The completed static review found no actionable regression.
🤖 Bits Code Review · Commit b13e6e3
dougqh
changed the title
This was referenced Oct 27, 2026
Merged
Merged
Merged
ygree
reviewed
Oct 27, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Skips building Data Streams Monitoring (DSM) tags and context in the Kafka consumer
TracingIterators (kafka-clients-0.11andkafka-clients-3.8) when DSM is disabled.Before this change,
startNewRecordSpancalledDataStreamsTags.create(...)(hashed tags, strings) and built aDataStreamsContextfor every consumed record, whether DSM was on or not. Only the payload-size computation checkedisDataStreamsEnabled(). With DSM off, the receivers are already no-ops:NoopPathwayContext, sosetCheckpointrecords nothing and sets no pathway-hash tagDataStreamsPropagator.injectalready returns early when DSM is disabledThe whole DSM block now sits behind
traceConfig().isDataStreamsEnabled(). That's the same dynamic check the payload-size line already used, so turning DSM on through remote config still works.Motivation
kafka-consumer-perf-test(10M records carrying real trace headers, 256m heap, JDK 17, 5 fresh-JVM reps, rotated), DSM off:Kafka consumers amplify per-message tracer cost more than any other workload we've measured. The consumer's own work is about 0.7 µs per record, so small per-record savings show up directly.
The macro numbers above were measured on the combined dense-store + trace-segment build. This PR adds
KafkaConsumeScenariosBenchmark(JMH, no agent or broker needed), which measures the guard alone against master: a 500-record poll batch drained throughTracingIterator, every record carrying this tracer's propagation headers. JDK 17, 3 forks × 5 iterations, M1 Max:dataStreamsfalsefalsetruetrueTo reproduce, cherry-pick the benchmark commit onto the base commit and run on both:
./gradlew :dd-java-agent:instrumentation:kafka:kafka-clients-0.11:jmh \ -Pjmh.includes=KafkaConsumeScenariosBenchmark -Pjmh.profilers=gc -PtestJvm=17Additional Notes
The same pattern appears elsewhere: DSM arguments built eagerly for a call that does nothing when DSM is off. Follow-ups:
setTagsFromIf(Product, ...)), so a disabled product costs one branch by construction instead of relying on a guard at each call site/perf-reviewrule to flag arguments built for a disabled or no-op callDSM and IDM reviewers: mostly an FYI on the pattern, since it affects how this code is likely to evolve.
Jira ticket: APMLP-2115
🤖 Generated with Claude Code