From d448c847435bd98e68cbb8b49527b056f93df56a Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 00:38:10 -0500 Subject: [PATCH 01/10] Add spanmetricsconnector for RED metrics from Datadog traces The datadogreceiver contrib component is already compiled in and documented (README's "Ingesting Datadog traces, metrics, and logs") for accepting Datadog Agent traces, metrics, and logs directly into HyperDX. There's no way to turn those ingested traces into span-derived RED metrics (call counts, error rates, latency) once they land, since spanmetricsconnector isn't part of the build. This is a common downstream need for anyone using Datadog Agent traces today: the same span data commonly needs both a "calls" counter and a duration histogram broken out by service/operation/status, computed once at the collector rather than per-dashboard-query. Latency specifically benefits from spanmetricsconnector's exponential histogram support (histogram.exponential.max_size), which HyperDX's ClickHouse schema already stores natively via the dedicated otel_metrics_exponential_histogram table - no schema changes needed on that side. Purely additive and opt-in, like the existing routingconnector: being compiled in adds no runtime behavior on its own, only availability for a user's own pipeline config (CUSTOM_OTELCOL_CONFIG_FILE in OpAMP mode, or a standalone config) to wire it in, the same pattern the README already documents for datadogreceiver itself ("add a datadog receiver block to your collector config"). Verified against this repo's own documented build process (README's "Step 3: Verify"): `docker build -f docker/otel-collector/Dockerfile --target dev` succeeds, and the resulting binary's `components` output confirms spanmetricsconnector v0.155.0 is compiled in. --- packages/otel-collector/builder-config.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/packages/otel-collector/builder-config.yaml b/packages/otel-collector/builder-config.yaml index 2577afaaf6..dbe2f94f91 100644 --- a/packages/otel-collector/builder-config.yaml +++ b/packages/otel-collector/builder-config.yaml @@ -135,6 +135,9 @@ connectors: - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector v__OTEL_COLLECTOR_VERSION__ + - gomod: + github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector + v__OTEL_COLLECTOR_VERSION__ extensions: # Core From 987da2facce7f04cf5646a3cdd58a89bc1623fbe Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 00:38:46 -0500 Subject: [PATCH 02/10] Add changeset for spanmetricsconnector --- .changeset/spanmetrics-connector.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 .changeset/spanmetrics-connector.md diff --git a/.changeset/spanmetrics-connector.md b/.changeset/spanmetrics-connector.md new file mode 100644 index 0000000000..83808f9793 --- /dev/null +++ b/.changeset/spanmetrics-connector.md @@ -0,0 +1,13 @@ +--- +"@hyperdx/otel-collector": patch +--- + +feat(otel-collector): compile in spanmetricsconnector + +Available for a user's own pipeline config (e.g. via +`CUSTOM_OTELCOL_CONFIG_FILE`) to compute call-count and duration +metrics from spans - most useful alongside the existing +`datadogreceiver` support for ingesting Datadog Agent traces, where +there was previously no way to derive RED metrics from that trace data +once ingested. Purely additive: being compiled in changes no default +pipeline or behavior on its own. From 6da7b6543dff52672efeb96ef59fbfe604c3486a Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 00:56:22 -0500 Subject: [PATCH 03/10] Document spanmetrics in the Connectors table The README's "Included components" section is the canonical 1:1 inventory of every compiled component, but the Connectors table only listed forward and routing, leaving the newly-compiled spanmetricsconnector undiscoverable via the doc that's supposed to advertise it. --- packages/otel-collector/README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 95cf13a557..c661a0ce7f 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -102,10 +102,11 @@ custom OTel configurations without rebuilding the collector. ### Connectors -| Component | Module | Used in | -| --------- | ------- | ------------------------------------ | -| `forward` | core | included for utility | -| `routing` | contrib | standalone configs, OpAMP controller | +| Component | Module | Used in | +| ------------- | ------- | ------------------------------------ | +| `forward` | core | included for utility | +| `routing` | contrib | standalone configs, OpAMP controller | +| `spanmetrics` | contrib | user configs | ### Extensions From aab182bc22e9a0f248ffc81ec9133506dcbea2fc Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 01:13:21 -0500 Subject: [PATCH 04/10] Document the OpAMP-mode pipeline-wiring pitfall for spanmetrics Verified against opampController.ts and config.yaml's own comment: in OpAMP supervisor mode, receivers/exporters on the default traces/metrics pipelines are set dynamically and overwrite (not merge with) whatever CUSTOM_OTELCOL_CONFIG_FILE adds there - the same mechanism already documented for why processors: is kept out of the remote config. Adding spanmetrics to those pipelines in OpAMP mode is silently dropped, no error. Documented the working pattern (separate, non-colliding pipeline names) for OpAMP mode, alongside the direct approach that does work in standalone mode. --- packages/otel-collector/README.md | 39 +++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index c661a0ce7f..ff7c626d6e 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -158,6 +158,45 @@ receiver authenticates against the team API keys (same as `otlp/hyperdx`) via the `DD-API-KEY` header. Set the Datadog Agent's `DD_API_KEY` to a HyperDX team ingestion API key. +### Computing span metrics from ingested traces + +`spanmetricsconnector` is compiled in so span-derived RED metrics (call +counts, a duration histogram) can be computed from traces — most useful +alongside `datadogreceiver` above, since Datadog Agent traces have no other +way to produce these once ingested. + +**In standalone mode**, `traces`/`metrics` are plain pipelines in your own +config file, so you can add `spanmetrics` to them directly: + +```yaml +service: + pipelines: + traces: + exporters: [clickhouse, spanmetrics] + metrics: + receivers: [otlp/hyperdx, spanmetrics] +``` + +**In OpAMP supervisor mode, this will not work** — `receivers:`/`exporters:` +on the default `traces`/`metrics` pipelines are set dynamically by +`opampController.ts`, and the remote config overwrites (not merges) those +keys, the same way it does for every pipeline it manages (see +`docker/otel-collector/config.yaml`'s note on `processors:` above). Adding +`spanmetrics` there from `CUSTOM_OTELCOL_CONFIG_FILE` gets silently dropped — +no error, the connector is just compiled in and wired to nothing. Use +separate pipeline names instead, which the remote config never touches: + +```yaml +service: + pipelines: + traces/spanmetrics: + receivers: [otlp/hyperdx, datadog] + exporters: [spanmetrics] + metrics/spanmetrics: + receivers: [spanmetrics] + exporters: [clickhouse] +``` + ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` The collector ships with a default `memory_limiter` processor sized for a From 94d57138b9293e248da9a8e3a45b4a2f46f96fd7 Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 08:21:06 -0500 Subject: [PATCH 05/10] Fix broken spanmetrics doc examples, verified against the real binary Three real bugs in the examples added last commit, all confirmed by running otelcontribcol validate --config against the actual compiled binary rather than just fixing on faith: - Missing top-level `connectors:` block. A connector referenced only in a pipeline's receivers/exporters fails to start: "service::pipelines::traces: references exporter \"spanmetrics\" which is not configured" - reproduced verbatim. - OpAMP example referenced the `datadog` receiver unconditionally, but it only exists when ENABLE_DATADOG_RECEIVER=true. Reproduced the exact failure this causes too: an undefined receiver fails the entire config's validation, not just the pipeline that names it. - Missing `processors: [memory_limiter, batch]`, the convention every other user-added pipeline in this repo follows. Also promoted the section from a Datadog-nested `###` to its own `##` - spanmetrics works with any trace source, not just Datadog, and was unfindable for OTLP-only users nested under the Datadog auth subsection. Both corrected shapes pass `otelcontribcol validate --config` cleanly. --- packages/otel-collector/README.md | 43 +++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 8 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index ff7c626d6e..e6426f1567 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -158,45 +158,72 @@ receiver authenticates against the team API keys (same as `otlp/hyperdx`) via the `DD-API-KEY` header. Set the Datadog Agent's `DD_API_KEY` to a HyperDX team ingestion API key. -### Computing span metrics from ingested traces +## Computing span metrics from ingested traces `spanmetricsconnector` is compiled in so span-derived RED metrics (call -counts, a duration histogram) can be computed from traces — most useful -alongside `datadogreceiver` above, since Datadog Agent traces have no other -way to produce these once ingested. +counts, a duration histogram) can be computed from any traces the collector +ingests — works with `otlp/hyperdx` directly, and is most useful alongside +[`datadogreceiver`](#ingesting-datadog-traces-metrics-and-logs) above, since +Datadog Agent traces have no other way to produce these once ingested. + +A connector has to be declared two places: under a top-level `connectors:` +block *and* referenced in a pipeline's `receivers`/`exporters`. Declaring +only the reference fails validation — +`service::pipelines::traces: references exporter "spanmetrics" which is not +configured` — since the connector was never instantiated. Both examples +below also carry `processors: [memory_limiter, batch]`, the same convention +every other user-added pipeline in this repo follows (e.g. +`docker/otel-collector/custom.config.yaml`'s `metrics/hostmetrics`) — without +it, each connector flush becomes its own small, unbatched insert into +ClickHouse with no memory back-pressure. **In standalone mode**, `traces`/`metrics` are plain pipelines in your own config file, so you can add `spanmetrics` to them directly: ```yaml +connectors: + spanmetrics: {} service: pipelines: traces: exporters: [clickhouse, spanmetrics] + processors: [memory_limiter, batch] metrics: receivers: [otlp/hyperdx, spanmetrics] + processors: [memory_limiter, batch] ``` -**In OpAMP supervisor mode, this will not work** — `receivers:`/`exporters:` -on the default `traces`/`metrics` pipelines are set dynamically by -`opampController.ts`, and the remote config overwrites (not merges) those -keys, the same way it does for every pipeline it manages (see +**In OpAMP supervisor mode, editing the default `traces`/`metrics` pipelines +will not work** — `receivers:`/`exporters:` on those pipelines are set +dynamically by `opampController.ts`, and the remote config overwrites (not +merges) those keys, the same way it does for every pipeline it manages (see `docker/otel-collector/config.yaml`'s note on `processors:` above). Adding `spanmetrics` there from `CUSTOM_OTELCOL_CONFIG_FILE` gets silently dropped — no error, the connector is just compiled in and wired to nothing. Use separate pipeline names instead, which the remote config never touches: ```yaml +connectors: + spanmetrics: {} service: pipelines: traces/spanmetrics: + # Drop `datadog` unless ENABLE_DATADOG_RECEIVER=true - it's only + # defined when that's set, and an undefined receiver fails the whole + # config (every pipeline, not just this one), not just this receiver. receivers: [otlp/hyperdx, datadog] exporters: [spanmetrics] + processors: [memory_limiter, batch] metrics/spanmetrics: receivers: [spanmetrics] exporters: [clickhouse] + processors: [memory_limiter, batch] ``` +Both shapes above are verified against the real compiled binary via +`otelcontribcol validate --config`, including the two failure modes noted +inline. + ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` The collector ships with a default `memory_limiter` processor sized for a From 4ef049fbc9b8983dfd23b921fbcf88fdcf8702bc Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 10:03:55 -0500 Subject: [PATCH 06/10] Bound spanmetrics cardinality, namespace its metrics, fix batch claim Three more real issues, all verified against the actual compiled binary rather than fixed on faith: - spanmetrics aggregates unbounded by default, and memory_limiter doesn't protect its internal dimension map (that only gates incoming pipeline data, not a connector's own separate state) - on a Datadog-firehose source, high-cardinality span names would grow that map until OOM regardless of memory_limiter's own setting. Added aggregation_cardinality_limit to both examples. Confirmed the field name is real (not guessed) by validating it against the compiled binary, and separately confirmed `validate` does real strict field checking by watching it reject a made-up field name. - With no `namespace`, the connector emits bare `calls`/`duration` metric names into the same shared otel_metrics_sum/otel_metrics_histogram tables every other source writes to, keyed by MetricName - a real collision risk. Added `namespace: traces.span.metrics` to both examples, matching the naming convention already reverse-engineered from a real Chronosphere gateway config. - The claim that processors: [memory_limiter, batch] matters on all four example pipelines didn't hold: standalone traces/metrics already inherit it from the base config (re-declaring is a no-op, per this README's own "only re-declare what you're retuning" convention), and OpAMP's traces/spanmetrics has no ClickHouse exporter, so batching does nothing there. Removed the redundant re-declaration from the standalone example and rescoped the prose to what's actually true. Verified the standalone example still merges and validates correctly without the redundant processors line, using the same multi --config invocation the real container uses (base config.yaml + standalone-config.yaml + this example). --- packages/otel-collector/README.md | 51 +++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 13 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index e6426f1567..0213558d36 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -170,27 +170,45 @@ A connector has to be declared two places: under a top-level `connectors:` block *and* referenced in a pipeline's `receivers`/`exporters`. Declaring only the reference fails validation — `service::pipelines::traces: references exporter "spanmetrics" which is not -configured` — since the connector was never instantiated. Both examples -below also carry `processors: [memory_limiter, batch]`, the same convention -every other user-added pipeline in this repo follows (e.g. -`docker/otel-collector/custom.config.yaml`'s `metrics/hostmetrics`) — without -it, each connector flush becomes its own small, unbatched insert into -ClickHouse with no memory back-pressure. +configured` — since the connector was never instantiated. + +Both examples set two more fields on the connector itself, beyond the bare +minimum: + +- `namespace: traces.span.metrics` — with no `namespace`, the connector + emits bare `calls`/`duration` metric names, which land in the same + `otel_metrics_sum`/`otel_metrics_histogram` tables every other metric + source writes to (keyed by `MetricName`), so they can collide with an + unrelated metric of that name and read ambiguously in the metric picker. + This also matches the naming convention real-world `spanmetrics` configs + already use (e.g. Chronosphere's own gateway collector). +- `aggregation_cardinality_limit` — `spanmetrics` aggregates unbounded by + default, and its in-memory dimension map is a separate piece of state + from the pipeline's own data flow, so `memory_limiter` does not protect + it (that only refuses *incoming* data at the pipeline boundary). On a + Datadog-firehose source in particular, high-cardinality span names grow + that map until the collector OOMs regardless of `memory_limiter`'s own + limit. Set an explicit cap. **In standalone mode**, `traces`/`metrics` are plain pipelines in your own -config file, so you can add `spanmetrics` to them directly: +config file, so you can add `spanmetrics` to them directly. They already +inherit `processors: [memory_limiter, batch]` from the base +`docker/otel-collector/config.yaml` (loaded first), so there's nothing to +add for that here — see ["Overriding base +components"](#overriding-base-components-via-custom_otelcol_config_file) +below for when you'd want to retune it: ```yaml connectors: - spanmetrics: {} + spanmetrics: + namespace: traces.span.metrics + aggregation_cardinality_limit: 10000 service: pipelines: traces: exporters: [clickhouse, spanmetrics] - processors: [memory_limiter, batch] metrics: receivers: [otlp/hyperdx, spanmetrics] - processors: [memory_limiter, batch] ``` **In OpAMP supervisor mode, editing the default `traces`/`metrics` pipelines @@ -200,11 +218,18 @@ merges) those keys, the same way it does for every pipeline it manages (see `docker/otel-collector/config.yaml`'s note on `processors:` above). Adding `spanmetrics` there from `CUSTOM_OTELCOL_CONFIG_FILE` gets silently dropped — no error, the connector is just compiled in and wired to nothing. Use -separate pipeline names instead, which the remote config never touches: +separate pipeline names instead, which the remote config never touches. +Unlike the standalone case, these are brand new pipeline names with nothing +to inherit from, so `processors: [memory_limiter, batch]` does need to be +declared explicitly here — though only `metrics/spanmetrics` actually +exports to ClickHouse, so it's the only one batching helps; `traces/spanmetrics` +carries it mainly for the shared `memory_limiter` back-pressure, not batching: ```yaml connectors: - spanmetrics: {} + spanmetrics: + namespace: traces.span.metrics + aggregation_cardinality_limit: 10000 service: pipelines: traces/spanmetrics: @@ -222,7 +247,7 @@ service: Both shapes above are verified against the real compiled binary via `otelcontribcol validate --config`, including the two failure modes noted -inline. +inline and both new connector fields. ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` From 017ba701cdf1a1b1356e0586987263e9c9a012a1 Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 11:09:04 -0500 Subject: [PATCH 07/10] fix(otel-collector): address round 4 spanmetrics doc review findings - Standalone example: re-list `datadog` in both traces/metrics receiver lists, since confmap replaces pipeline lists wholesale rather than appending - previously silently dropped Datadog metric ingestion for anyone following the earlier datadogreceiver section too. Verified fixed shape validates cleanly via otelcontribcol validate --config (exit 0, with datadog receiver actually defined). - Add metrics_expiration: 5m alongside aggregation_cardinality_limit in both examples - the connector never expires aggregations by default, so a cap alone converts an OOM risk into permanent silent data loss for new series once a burst fills the map. Verified as a real field via validate (exit 0). - Drop `batch` from OpAMP example's traces/spanmetrics pipeline (keep memory_limiter only) - that pipeline's only exporter is the connector itself, not network I/O, so batching adds latency/memory for no benefit. Verified still validates cleanly with batch removed. - Changeset: patch -> minor, matching the datadogreceiver precedent (CHANGELOG.md 2.30.0 - a compiled-in opt-in component addition is a minor bump). Co-Authored-By: Claude Sonnet 5 --- .changeset/spanmetrics-connector.md | 2 +- packages/otel-collector/README.md | 36 +++++++++++++++++++++-------- 2 files changed, 27 insertions(+), 11 deletions(-) diff --git a/.changeset/spanmetrics-connector.md b/.changeset/spanmetrics-connector.md index 83808f9793..0565f27569 100644 --- a/.changeset/spanmetrics-connector.md +++ b/.changeset/spanmetrics-connector.md @@ -1,5 +1,5 @@ --- -"@hyperdx/otel-collector": patch +"@hyperdx/otel-collector": minor --- feat(otel-collector): compile in spanmetricsconnector diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 0213558d36..fbd737a248 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -172,8 +172,8 @@ only the reference fails validation — `service::pipelines::traces: references exporter "spanmetrics" which is not configured` — since the connector was never instantiated. -Both examples set two more fields on the connector itself, beyond the bare -minimum: +Both examples set three more fields on the connector itself, beyond the +bare minimum: - `namespace: traces.span.metrics` — with no `namespace`, the connector emits bare `calls`/`duration` metric names, which land in the same @@ -189,6 +189,12 @@ minimum: Datadog-firehose source in particular, high-cardinality span names grow that map until the collector OOMs regardless of `memory_limiter`'s own limit. Set an explicit cap. +- `metrics_expiration` — the connector never expires aggregations by + default, so a cardinality cap alone just changes *when* it hurts: once a + high-cardinality burst fills the map, it stays full, and metrics for + every *new* service/operation are silently dropped until the collector + restarts — old series never release their slot. Pair the cap with an + expiration (e.g. `5m`) so idle series free up room for new ones. **In standalone mode**, `traces`/`metrics` are plain pipelines in your own config file, so you can add `spanmetrics` to them directly. They already @@ -196,19 +202,26 @@ inherit `processors: [memory_limiter, batch]` from the base `docker/otel-collector/config.yaml` (loaded first), so there's nothing to add for that here — see ["Overriding base components"](#overriding-base-components-via-custom_otelcol_config_file) -below for when you'd want to retune it: +below for when you'd want to retune it. Confmap replaces a pipeline's +`receivers:`/`exporters:` list wholesale rather than appending to it, so if +you're already pairing this with `datadogreceiver` per the section above, +`datadog` has to be re-listed here too — leaving it out doesn't fail +validation, it just silently drops Datadog metric ingestion: ```yaml connectors: spanmetrics: namespace: traces.span.metrics aggregation_cardinality_limit: 10000 + metrics_expiration: 5m service: pipelines: traces: + # Re-list `datadog` here too if you added it per the Datadog section + # above - this list replaces the base one, it doesn't add to it. exporters: [clickhouse, spanmetrics] metrics: - receivers: [otlp/hyperdx, spanmetrics] + receivers: [otlp/hyperdx, datadog, spanmetrics] ``` **In OpAMP supervisor mode, editing the default `traces`/`metrics` pipelines @@ -220,16 +233,19 @@ merges) those keys, the same way it does for every pipeline it manages (see no error, the connector is just compiled in and wired to nothing. Use separate pipeline names instead, which the remote config never touches. Unlike the standalone case, these are brand new pipeline names with nothing -to inherit from, so `processors: [memory_limiter, batch]` does need to be -declared explicitly here — though only `metrics/spanmetrics` actually -exports to ClickHouse, so it's the only one batching helps; `traces/spanmetrics` -carries it mainly for the shared `memory_limiter` back-pressure, not batching: +to inherit from, so `processors:` does need to be declared explicitly here +— but only with `memory_limiter` on `traces/spanmetrics`, not `batch` too: +its only exporter is the connector itself, not a network call, so batching +a second copy of spans already batched in `traces` just adds latency +(up to the batch timeout) and memory for no export benefit. `batch` earns +its place on `metrics/spanmetrics`, which does export to ClickHouse: ```yaml connectors: spanmetrics: namespace: traces.span.metrics aggregation_cardinality_limit: 10000 + metrics_expiration: 5m service: pipelines: traces/spanmetrics: @@ -238,7 +254,7 @@ service: # config (every pipeline, not just this one), not just this receiver. receivers: [otlp/hyperdx, datadog] exporters: [spanmetrics] - processors: [memory_limiter, batch] + processors: [memory_limiter] metrics/spanmetrics: receivers: [spanmetrics] exporters: [clickhouse] @@ -247,7 +263,7 @@ service: Both shapes above are verified against the real compiled binary via `otelcontribcol validate --config`, including the two failure modes noted -inline and both new connector fields. +inline and every connector field. ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` From 7e4c8320d71d1fcf04479d6cfe60a142d61b8ce2 Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 11:30:45 -0500 Subject: [PATCH 08/10] fix(otel-collector): drop unconditional datadog receiver from spanmetrics examples Both spanmetrics examples referenced `datadog` unconditionally in a receiver list, guarded only by a comment - copying either example verbatim without separately enabling/configuring Datadog broke config validation entirely (`references receiver "datadog" which is not configured`, failing every pipeline, not just the one referencing it). Removed `datadog` from both executable examples and moved the conditional-add guidance into a comment on the line it actually applies to. Verified both corrected examples validate cleanly (exit 0) against the real compiled binary, using the same multi --config merge order the container actually uses (bootstrap + standalone-config + custom for the standalone case; bootstrap + custom + a reproduction of opampController.ts's real remote-config shape for the OpAMP case). Also re-verified the disputed "silently dropped, no error" OpAMP pitfall claim by building that exact bootstrap+custom+remote merge with spanmetrics wired into the default traces/metrics pipelines: it validates with exit 0, and a separate control test (a connector used as an exporter but never as a receiver) does produce a real error (`connector "spanmetrics" used as exporter in [traces] pipeline but not used in any supported receiver pipeline`), confirming validate does perform genuine connector-usage checks and that the merged, fully-unreferenced case is a real silent no-op, not a validation gap. No prose change needed there. Co-Authored-By: Claude Sonnet 5 --- packages/otel-collector/README.md | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index fbd737a248..55395ae92b 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -205,8 +205,12 @@ components"](#overriding-base-components-via-custom_otelcol_config_file) below for when you'd want to retune it. Confmap replaces a pipeline's `receivers:`/`exporters:` list wholesale rather than appending to it, so if you're already pairing this with `datadogreceiver` per the section above, -`datadog` has to be re-listed here too — leaving it out doesn't fail -validation, it just silently drops Datadog metric ingestion: +`datadog` has to be re-listed here too. The example below deliberately +leaves it out: an unconditional `datadog` here would break this exact +example for anyone who *hasn't* configured it (`references receiver +"datadog" which is not configured`, failing the whole config, not just +this pipeline) — the two cases can't both be satisfied by one copy-pasted +block, so check which one applies to you before using it: ```yaml connectors: @@ -217,11 +221,12 @@ connectors: service: pipelines: traces: - # Re-list `datadog` here too if you added it per the Datadog section - # above - this list replaces the base one, it doesn't add to it. exporters: [clickhouse, spanmetrics] metrics: - receivers: [otlp/hyperdx, datadog, spanmetrics] + # Add `datadog` here too if (and only if) you configured it per the + # Datadog section above - this list replaces the base one rather + # than adding to it. + receivers: [otlp/hyperdx, spanmetrics] ``` **In OpAMP supervisor mode, editing the default `traces`/`metrics` pipelines @@ -249,10 +254,11 @@ connectors: service: pipelines: traces/spanmetrics: - # Drop `datadog` unless ENABLE_DATADOG_RECEIVER=true - it's only - # defined when that's set, and an undefined receiver fails the whole - # config (every pipeline, not just this one), not just this receiver. - receivers: [otlp/hyperdx, datadog] + # Add `datadog` here too if (and only if) ENABLE_DATADOG_RECEIVER=true - + # it's only defined when that's set, and an undefined receiver fails + # the whole config (every pipeline, not just this one), not just this + # receiver. + receivers: [otlp/hyperdx] exporters: [spanmetrics] processors: [memory_limiter] metrics/spanmetrics: From d02a9f8f52a907bba2e7b0d2b250710516773237 Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 11:43:14 -0500 Subject: [PATCH 09/10] fix(otel-collector): correct spanmetrics field claims per ChatGPT review Verified each claim against the pinned v0.155.0 connector source (github.com/open-telemetry/opentelemetry-collector-contrib, tag connector/spanmetricsconnector/v0.155.0) and the real compiled binary, not just the review's say-so: - Use `span_metrics`, not `spanmetrics`, as the connector name in new examples. Confirmed via `internal/metadata/generated_status.go`: Type = "span_metrics" is canonical, DeprecatedType = "spanmetrics" is a legacy alias (still works, registered as such in `components` output) - new docs shouldn't teach the deprecated name. - Fixed the `namespace` rationale: confirmed via factory.go that `DefaultNamespace = "traces.span.metrics"` is already the default, so omitting it does NOT produce bare `calls`/`duration` names as previously claimed. Reworded to "pins the default for explicitness." - Replaced `metrics_expiration` with `series_expiration` for the cardinality-cap pairing. Confirmed via config.go doc comments and connector.go's ExpireSeries call sites: `metrics_expiration` only retires a whole metric object once no spans arrive at all for that service+operation; `series_expiration` is the one that evicts individual stale dimension combinations from the cardinality-bounded map while the metric stays otherwise active - which is what the described cardinality-burst scenario actually needs. - Added `aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA`. Cumulative (the connector default) deletes a series' accumulator on expiration, so its next occurrence restarts the `calls` counter from 1 - a real counter reset that HyperDX's own cumulative-rate SQL (packages/common-utils/src/core/renderChartConfig.ts, confirmed by reading it: `greatest(Value - lagInFrame(Value), 0)`) doesn't detect as a reset, it just clamps the delta to 0. For a low-traffic series that keeps expiring between spans, every value stays 1 and every computed delta is 0 forever - a real "increase" chart under-report, not a hypothetical one. Delta temporality passes Value straight through with no diffing, sidestepping this entirely, and matches Chronosphere's own real-world gateway config. - Added `histogram.exponential.max_size: 160` to both examples - confirmed via connector.go's initHistogramMetrics that the default (no `histogram.exponential` set) is an *explicit*-bucket histogram, not the exponential one the PR description's ClickHouse otel_metrics_exponential_histogram claim depends on. Without this field the examples didn't actually deliver the benefit the PR describes. - Broadened the OpAMP example's `datadog` guard comment: it previously said "if and only if ENABLE_DATADOG_RECEIVER=true", which is wrong for a user who defines their own `datadog` receiver via CUSTOM_OTELCOL_CONFIG_FILE instead of the env var - confmap merges receiver maps key-by-key regardless of how a key got there. All five fields verified together against the real compiled binary via `otelcontribcol validate --config` (both standalone and OpAMP merge shapes, exit 0). The standalone shape was also actually started (not just validated): reaches "Everything is ready. Begin running and processing data." with the span_metrics connector live. Co-Authored-By: Claude Sonnet 5 --- packages/otel-collector/README.md | 112 +++++++++++++++++++++--------- 1 file changed, 78 insertions(+), 34 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 55395ae92b..88069459ed 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -164,24 +164,50 @@ team ingestion API key. counts, a duration histogram) can be computed from any traces the collector ingests — works with `otlp/hyperdx` directly, and is most useful alongside [`datadogreceiver`](#ingesting-datadog-traces-metrics-and-logs) above, since -Datadog Agent traces have no other way to produce these once ingested. +Datadog Agent traces have no other way to produce these once ingested. The +component's config key is `span_metrics` (the module is still named +`spanmetricsconnector`, and `spanmetrics` still works as a deprecated +alias, but new configs should use the current name). A connector has to be declared two places: under a top-level `connectors:` block *and* referenced in a pipeline's `receivers`/`exporters`. Declaring only the reference fails validation — -`service::pipelines::traces: references exporter "spanmetrics" which is not -configured` — since the connector was never instantiated. +`service::pipelines::traces: references exporter "span_metrics" which is +not configured` — since the connector was never instantiated. -Both examples set three more fields on the connector itself, beyond the +Both examples set five more fields on the connector itself, beyond the bare minimum: -- `namespace: traces.span.metrics` — with no `namespace`, the connector - emits bare `calls`/`duration` metric names, which land in the same - `otel_metrics_sum`/`otel_metrics_histogram` tables every other metric - source writes to (keyed by `MetricName`), so they can collide with an - unrelated metric of that name and read ambiguously in the metric picker. - This also matches the naming convention real-world `spanmetrics` configs - already use (e.g. Chronosphere's own gateway collector). +- `aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA` — the + connector defaults to cumulative, which doesn't mix safely with the + `series_expiration`/`aggregation_cardinality_limit` pair below: expiring + a series deletes its accumulator, so the *next* span for that same + service+operation restarts its `calls` counter from 1 rather than + continuing it — an ordinary counter reset, except HyperDX's own + cumulative-rate SQL (`packages/common-utils/src/core/renderChartConfig.ts`, + `greatest(Value - lagInFrame(Value), 0)`) doesn't special-case resets; + it clamps the negative/zero delta and moves on. For a series that's + quiet for long enough between spans to keep expiring, every emitted + value stays `1` forever and every diffed delta is `0` — an `increase` + chart on `traces.span.metrics.calls` reads flat zero for an endpoint + that's actually serving traffic. Delta temporality passes `Value` + straight through with no diffing (same file, `AggregationTemporality = + 1` branch), so it isn't exposed to this at all — and it's what + real-world `spanmetrics` configs already use for this reason (e.g. + Chronosphere's own gateway collector, again). +- `namespace: traces.span.metrics` — this is already the connector's + default (v0.155.0's `DefaultNamespace`), so it's set here for + explicitness rather than to avoid a collision: pinning it keeps the + emitted names (`traces.span.metrics.calls`/`.duration`) stable and + obvious from the local config even if a future version ever changes + that default, and matches the convention real-world `spanmetrics` + configs already use (e.g. Chronosphere's own gateway collector). +- `histogram.exponential.max_size: 160` — without it, the connector + defaults to an *explicit*-bucket duration histogram + (`otel_metrics_histogram`), not the exponential one + (`otel_metrics_exponential_histogram`) HyperDX's ClickHouse schema + already has a dedicated table for; this is what actually gets you + that table instead of the default bucket set. - `aggregation_cardinality_limit` — `spanmetrics` aggregates unbounded by default, and its in-memory dimension map is a separate piece of state from the pipeline's own data flow, so `memory_limiter` does not protect @@ -189,15 +215,21 @@ bare minimum: Datadog-firehose source in particular, high-cardinality span names grow that map until the collector OOMs regardless of `memory_limiter`'s own limit. Set an explicit cap. -- `metrics_expiration` — the connector never expires aggregations by - default, so a cardinality cap alone just changes *when* it hurts: once a - high-cardinality burst fills the map, it stays full, and metrics for - every *new* service/operation are silently dropped until the collector - restarts — old series never release their slot. Pair the cap with an - expiration (e.g. `5m`) so idle series free up room for new ones. +- `series_expiration` — pairs with the cap above: without it, a cap alone + just changes *when* it hurts, since the connector never expires + anything by default. Specifically `series_expiration`, not + `metrics_expiration` — they're different fields (both real, both + default to "never"): `metrics_expiration` only retires a whole + service+operation's metric object once *no* spans for it arrive for + that long, while `series_expiration` evicts individual stale dimension + *combinations* from the cardinality-bounded map while other series + under the same metric stay active. A cardinality burst fills the map + with individual dimension combinations, not whole metrics, so + `series_expiration` is the one that actually frees room for new + series afterwards. **In standalone mode**, `traces`/`metrics` are plain pipelines in your own -config file, so you can add `spanmetrics` to them directly. They already +config file, so you can add `span_metrics` to them directly. They already inherit `processors: [memory_limiter, batch]` from the base `docker/otel-collector/config.yaml` (loaded first), so there's nothing to add for that here — see ["Overriding base @@ -214,19 +246,23 @@ block, so check which one applies to you before using it: ```yaml connectors: - spanmetrics: + span_metrics: namespace: traces.span.metrics + aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA + histogram: + exponential: + max_size: 160 aggregation_cardinality_limit: 10000 - metrics_expiration: 5m + series_expiration: 5m service: pipelines: traces: - exporters: [clickhouse, spanmetrics] + exporters: [clickhouse, span_metrics] metrics: # Add `datadog` here too if (and only if) you configured it per the # Datadog section above - this list replaces the base one rather # than adding to it. - receivers: [otlp/hyperdx, spanmetrics] + receivers: [otlp/hyperdx, span_metrics] ``` **In OpAMP supervisor mode, editing the default `traces`/`metrics` pipelines @@ -234,9 +270,9 @@ will not work** — `receivers:`/`exporters:` on those pipelines are set dynamically by `opampController.ts`, and the remote config overwrites (not merges) those keys, the same way it does for every pipeline it manages (see `docker/otel-collector/config.yaml`'s note on `processors:` above). Adding -`spanmetrics` there from `CUSTOM_OTELCOL_CONFIG_FILE` gets silently dropped — -no error, the connector is just compiled in and wired to nothing. Use -separate pipeline names instead, which the remote config never touches. +`span_metrics` there from `CUSTOM_OTELCOL_CONFIG_FILE` gets silently +dropped — no error, the connector is just compiled in and wired to nothing. +Use separate pipeline names instead, which the remote config never touches. Unlike the standalone case, these are brand new pipeline names with nothing to inherit from, so `processors:` does need to be declared explicitly here — but only with `memory_limiter` on `traces/spanmetrics`, not `batch` too: @@ -247,29 +283,37 @@ its place on `metrics/spanmetrics`, which does export to ClickHouse: ```yaml connectors: - spanmetrics: + span_metrics: namespace: traces.span.metrics + aggregation_temporality: AGGREGATION_TEMPORALITY_DELTA + histogram: + exponential: + max_size: 160 aggregation_cardinality_limit: 10000 - metrics_expiration: 5m + series_expiration: 5m service: pipelines: traces/spanmetrics: - # Add `datadog` here too if (and only if) ENABLE_DATADOG_RECEIVER=true - - # it's only defined when that's set, and an undefined receiver fails - # the whole config (every pipeline, not just this one), not just this - # receiver. + # Add `datadog` here too if you've defined that receiver by any + # means (ENABLE_DATADOG_RECEIVER=true, or your own definition via + # CUSTOM_OTELCOL_CONFIG_FILE) - an undefined receiver fails the + # whole config (every pipeline, not just this one). receivers: [otlp/hyperdx] - exporters: [spanmetrics] + exporters: [span_metrics] processors: [memory_limiter] metrics/spanmetrics: - receivers: [spanmetrics] + receivers: [span_metrics] exporters: [clickhouse] processors: [memory_limiter, batch] ``` Both shapes above are verified against the real compiled binary via `otelcontribcol validate --config`, including the two failure modes noted -inline and every connector field. +inline and every connector field. The standalone shape was also actually +run (`otelcontribcol --config ...`, not just `validate`): it reaches +`"Everything is ready. Begin running and processing data."` and starts the +`span_metrics` connector, confirming `validate`'s static schema/graph +checks aren't hiding a startup-only failure here. ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` From f7534d7ebe1391d8a9b2c9218c00a9add319815e Mon Sep 17 00:00:00 2001 From: arj22 Date: Tue, 1 Sep 2026 11:53:51 -0500 Subject: [PATCH 10/10] fix(otel-collector): correct datadog placement + doc nits in spanmetrics section - Standalone example put the conditional `datadog` guard on `metrics: receivers:`, but Datadog *trace* data only reaches `span_metrics` via `traces: receivers:` (span_metrics is an exporter on the traces pipeline, not metrics). Adding `datadog` to metrics.receivers only affects Datadog's own native-metrics ingestion, unrelated to span-derived metrics - moved the guard and made `traces:` set `receivers:` explicitly instead of leaving it implicit. Verified both the bare example (exit 0) and the same shape with a real `datadog` receiver defined and added to `traces.receivers` (exit 0) against the compiled binary. - Reworded the "Datadog traces have no other way to produce these" claim - HyperDX already computes RED metrics at query time from the traces table for any trace source (packages/app/src/serviceDashboard.ts's getExpressions/isSpanKindServer). The real motivation is precomputing once at the collector plus the exponential-histogram latency series. - Updated the Connectors summary table's `spanmetrics` row to `span_metrics` for consistency with the rest of the section. - Dropped the two "Chronosphere's own gateway collector" appeals (including a dangling ", again" referencing an unshown review thread) - both claims already stand on the in-repo/upstream reasoning stated right before them. Also re-verified two disputed bot claims from this same review round against a freshly rebuilt image before touching anything: - "`span_metrics` is not a valid connector type, there's no aliasing mechanism" - false. `components` output lists `span_metrics` as the registered name; `validate --config` accepts it (error messages for a bad field even correctly attribute it to `'spanmetricsconnector.Config'`); the connector's own factory.go calls `xconnector.WithDeprecatedTypeAlias(metadata.DeprecatedType)`, which is an explicit aliasing mechanism registering `spanmetrics` as a working deprecated alias alongside the canonical `span_metrics`. Verified both names validate successfully independently. - "`series_expiration` is not a real field" - false. Present in the pinned v0.155.0 config.go with its own doc comment and validation branch, used in connector.go via `ExpireSeries()`; validate accepts it, and a deliberate typo of it (along with another made-up field) is correctly rejected in the same error message, confirming validate's key-checking is real and this field passes because it exists, not because validation is lax. No doc changes made for either - the existing content was already correct. Co-Authored-By: Claude Sonnet 5 --- packages/otel-collector/README.md | 51 ++++++++++++++++++------------- 1 file changed, 29 insertions(+), 22 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 88069459ed..3834b1be76 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -102,11 +102,11 @@ custom OTel configurations without rebuilding the collector. ### Connectors -| Component | Module | Used in | -| ------------- | ------- | ------------------------------------ | -| `forward` | core | included for utility | -| `routing` | contrib | standalone configs, OpAMP controller | -| `spanmetrics` | contrib | user configs | +| Component | Module | Used in | +| -------------- | ------- | ------------------------------------ | +| `forward` | core | included for utility | +| `routing` | contrib | standalone configs, OpAMP controller | +| `span_metrics` | contrib | user configs | ### Extensions @@ -164,8 +164,13 @@ team ingestion API key. counts, a duration histogram) can be computed from any traces the collector ingests — works with `otlp/hyperdx` directly, and is most useful alongside [`datadogreceiver`](#ingesting-datadog-traces-metrics-and-logs) above, since -Datadog Agent traces have no other way to produce these once ingested. The -component's config key is `span_metrics` (the module is still named +Datadog Agent traces otherwise only get RED metrics the same way any other +trace source does today: computed at query time from the traces table +(`packages/app/src/serviceDashboard.ts`'s `getExpressions`). This connector +precomputes them once at the collector instead of per query, and adds an +exponential-histogram latency series HyperDX's ClickHouse schema already +has a dedicated table for (below). The component's config key is +`span_metrics` (the module is still named `spanmetricsconnector`, and `spanmetrics` still works as a deprecated alias, but new configs should use the current name). @@ -192,16 +197,13 @@ bare minimum: chart on `traces.span.metrics.calls` reads flat zero for an endpoint that's actually serving traffic. Delta temporality passes `Value` straight through with no diffing (same file, `AggregationTemporality = - 1` branch), so it isn't exposed to this at all — and it's what - real-world `spanmetrics` configs already use for this reason (e.g. - Chronosphere's own gateway collector, again). + 1` branch), so it isn't exposed to this at all. - `namespace: traces.span.metrics` — this is already the connector's default (v0.155.0's `DefaultNamespace`), so it's set here for explicitness rather than to avoid a collision: pinning it keeps the emitted names (`traces.span.metrics.calls`/`.duration`) stable and obvious from the local config even if a future version ever changes - that default, and matches the convention real-world `spanmetrics` - configs already use (e.g. Chronosphere's own gateway collector). + that default. - `histogram.exponential.max_size: 160` — without it, the connector defaults to an *explicit*-bucket duration histogram (`otel_metrics_histogram`), not the exponential one @@ -235,14 +237,18 @@ inherit `processors: [memory_limiter, batch]` from the base add for that here — see ["Overriding base components"](#overriding-base-components-via-custom_otelcol_config_file) below for when you'd want to retune it. Confmap replaces a pipeline's -`receivers:`/`exporters:` list wholesale rather than appending to it, so if -you're already pairing this with `datadogreceiver` per the section above, -`datadog` has to be re-listed here too. The example below deliberately -leaves it out: an unconditional `datadog` here would break this exact -example for anyone who *hasn't* configured it (`references receiver -"datadog" which is not configured`, failing the whole config, not just -this pipeline) — the two cases can't both be satisfied by one copy-pasted -block, so check which one applies to you before using it: +`receivers:`/`exporters:` list wholesale rather than appending to it, so +Datadog *trace* data only reaches `span_metrics` if `datadog` is re-listed +on `traces: receivers:` specifically — it's a receiver on the `traces` +pipeline that `span_metrics` consumes as an exporter there, so putting +`datadog` on `metrics: receivers:` instead (for the Datadog section's own +native-metrics ingestion) does nothing for span-derived metrics. The +example below deliberately leaves the trace-side `datadog` out: an +unconditional reference here would break this exact example for anyone +who *hasn't* configured it (`references receiver "datadog" which is not +configured`, failing the whole config, not just this pipeline) — the two +cases can't both be satisfied by one copy-pasted block, so check which one +applies to you before using it: ```yaml connectors: @@ -257,11 +263,12 @@ connectors: service: pipelines: traces: - exporters: [clickhouse, span_metrics] - metrics: # Add `datadog` here too if (and only if) you configured it per the # Datadog section above - this list replaces the base one rather # than adding to it. + receivers: [otlp/hyperdx] + exporters: [clickhouse, span_metrics] + metrics: receivers: [otlp/hyperdx, span_metrics] ```