From 5be2e79ac0ff9911b03b4765dd847db1bb6d5e2f Mon Sep 17 00:00:00 2001 From: arj22 Date: Wed, 9 Sep 2026 11:26:10 -0500 Subject: [PATCH 1/5] feat(otel-collector): add statsdreceiver for StatsD/DogStatsD ingestion Compiles in `statsdreceiver` so StatsD and DogStatsD-formatted metrics can be ingested directly, without a separate StatsD-to-OTLP bridge. Documents that it doesn't support horizontally-scaled deployments (a real, upstream-documented limitation, not HyperDX-specific) and that it should not be wired into the OpAMP-managed otel-collector Deployment, which runs multiple replicas for OTLP ingest availability. Verified against the real compiled binary (pinned v0.155.0): confirmed `statsdreceiver` is not currently compiled in, added it, rebuilt, and confirmed the documented config both passes `otelcontribcol validate --config` and actually starts (`otelcontribcol --config ...`, reaching "Everything is ready. Begin running and processing data."), using the same multi-config merge the container's own entrypoint uses. Co-Authored-By: Claude Sonnet 5 --- .changeset/statsd-receiver.md | 10 ++++ packages/otel-collector/README.md | 52 +++++++++++++++++++++ packages/otel-collector/builder-config.yaml | 3 ++ 3 files changed, 65 insertions(+) create mode 100644 .changeset/statsd-receiver.md diff --git a/.changeset/statsd-receiver.md b/.changeset/statsd-receiver.md new file mode 100644 index 0000000000..af8cb78bf9 --- /dev/null +++ b/.changeset/statsd-receiver.md @@ -0,0 +1,10 @@ +--- +"@hyperdx/otel-collector": minor +--- + +feat(otel-collector): compile in statsdreceiver + +Available for a user's own pipeline config (e.g. via +`CUSTOM_OTELCOL_CONFIG_FILE`) to ingest StatsD/DogStatsD metrics +directly, without a separate StatsD-to-OTLP bridge. Purely additive: +being compiled in changes no default pipeline or behavior on its own. diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 3834b1be76..c40ef49536 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -68,6 +68,7 @@ custom OTel configurations without rebuilding the collector. | `k8scluster` | contrib | user configs | | `kubeletstats` | contrib | user configs | | `prometheus` | contrib | OpAMP controller, smoke tests | +| `statsd` | contrib | user configs | ### Processors @@ -322,6 +323,57 @@ run (`otelcontribcol --config ...`, not just `validate`): it reaches `span_metrics` connector, confirming `validate`'s static schema/graph checks aren't hiding a startup-only failure here. +## Ingesting StatsD/DogStatsD metrics + +`statsdreceiver` is compiled in so StatsD and DogStatsD-formatted metrics +(e.g. from a Datadog Agent's local DogStatsD listener, or anything else +speaking the StatsD wire protocol) can be ingested directly, without a +separate StatsD-to-OTLP bridge. It understands DogStatsD's tag extension +natively (`:||#tag1:value1,tag2:value2`), so +Datadog-originated StatsD traffic doesn't need any translation. + +**This component does not support horizontally-scaled deployments.** +That's not a HyperDX-specific limitation — it's stated directly in the +upstream component's own docs: aggregation state lives independently in +each collector instance's memory, with no cross-replica coordination, so +the same metric arriving at two different replicas (e.g. behind a +load-balanced Service) would be aggregated separately by each one instead +of together, silently splitting/undercounting it. HyperDX's own +OpAMP-managed `otel-collector` Deployment runs multiple replicas for OTLP +ingest availability — **do not** wire `statsd` into that Deployment's +pipelines. Run it on a dedicated, single-replica collector instance +instead (standalone mode, as below). + +```yaml +receivers: + statsd: + endpoint: 0.0.0.0:8125 + # DogStatsD supports bare tags with no value (`#mytag`, not just + # `#mytag:myvalue`) - without this, those get dropped. + enable_simple_tags: true + # Optional: route timer/histogram-type StatsD metrics to an + # exponential histogram (HyperDX's ClickHouse schema already has a + # dedicated otel_metrics_exponential_histogram table for these, + # verified elsewhere in this README) instead of the default + # explicit-bucket histogram. + timer_histogram_mapping: + - statsd_type: "timing" + observer_type: "histogram" + histogram: + max_size: 160 +service: + pipelines: + metrics: + receivers: [otlp/hyperdx, statsd] +``` + +Verified against the real compiled binary via `otelcontribcol validate +--config` and an actual startup run (`otelcontribcol --config ...`, +reaching `"Everything is ready. Begin running and processing data."`), +in both cases using the same multi-`--config` merge the container +actually uses (base `config.yaml` + `standalone-config.yaml` + this +fragment). + ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` The collector ships with a default `memory_limiter` processor sized for a diff --git a/packages/otel-collector/builder-config.yaml b/packages/otel-collector/builder-config.yaml index dbe2f94f91..b0b7b3834c 100644 --- a/packages/otel-collector/builder-config.yaml +++ b/packages/otel-collector/builder-config.yaml @@ -51,6 +51,9 @@ receivers: - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v__OTEL_COLLECTOR_VERSION__ + - gomod: + github.com/open-telemetry/opentelemetry-collector-contrib/receiver/statsdreceiver + v__OTEL_COLLECTOR_VERSION__ processors: # Core From 5e890436aa70a1c043a376b36f3a3ea0e1f49fe5 Mon Sep 17 00:00:00 2001 From: arj22 Date: Wed, 9 Sep 2026 13:36:30 -0500 Subject: [PATCH 2/5] fix(otel-collector): simplify statsd docs, drop incorrect histogram claim - Removed the timer_histogram_mapping example claiming to route "timer/histogram-type" metrics to an exponential histogram. Verified against the pinned v0.155.0 source (factory.go's defaultTimerHistogramMapping): timer, histogram, AND distribution all default to a gauge observer, and the example only remapped `timing` - histogram/distribution metrics would still land as gauges, contradicting the comment. - Reframed the section around plain StatsD rather than DogStatsD specifically - the receiver handles any StatsD client; DogStatsD's tag extension is supported but not required, and centering the docs on one vendor's dialect was misleading and pulled in the now-removed, Datadog-distribution-specific histogram-mapping example. - Net simplification: fewer lines, no vendor-specific framing, no incorrect claim. Co-Authored-By: Claude Sonnet 5 --- packages/otel-collector/README.md | 28 ++++++++-------------------- 1 file changed, 8 insertions(+), 20 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index c40ef49536..16fb636b80 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -323,14 +323,13 @@ run (`otelcontribcol --config ...`, not just `validate`): it reaches `span_metrics` connector, confirming `validate`'s static schema/graph checks aren't hiding a startup-only failure here. -## Ingesting StatsD/DogStatsD metrics +## Ingesting StatsD metrics -`statsdreceiver` is compiled in so StatsD and DogStatsD-formatted metrics -(e.g. from a Datadog Agent's local DogStatsD listener, or anything else -speaking the StatsD wire protocol) can be ingested directly, without a -separate StatsD-to-OTLP bridge. It understands DogStatsD's tag extension -natively (`:||#tag1:value1,tag2:value2`), so -Datadog-originated StatsD traffic doesn't need any translation. +`statsdreceiver` is compiled in so StatsD-formatted metrics can be ingested +directly, without a separate StatsD-to-OTLP bridge — this covers any StatsD +client, not just a particular vendor's dialect. It also understands the +DogStatsD tag extension (`#tag1:value1,tag2:value2`) if the sender uses it, +but doesn't require it. **This component does not support horizontally-scaled deployments.** That's not a HyperDX-specific limitation — it's stated directly in the @@ -347,20 +346,9 @@ instead (standalone mode, as below). ```yaml receivers: statsd: + # Defaults to localhost:8125, which only accepts local traffic - + # override to 0.0.0.0 if metrics arrive from outside the container. endpoint: 0.0.0.0:8125 - # DogStatsD supports bare tags with no value (`#mytag`, not just - # `#mytag:myvalue`) - without this, those get dropped. - enable_simple_tags: true - # Optional: route timer/histogram-type StatsD metrics to an - # exponential histogram (HyperDX's ClickHouse schema already has a - # dedicated otel_metrics_exponential_histogram table for these, - # verified elsewhere in this README) instead of the default - # explicit-bucket histogram. - timer_histogram_mapping: - - statsd_type: "timing" - observer_type: "histogram" - histogram: - max_size: 160 service: pipelines: metrics: From a275adf21211e98c79dbaa04ae4e807ed3066006 Mon Sep 17 00:00:00 2001 From: arj22 Date: Wed, 9 Sep 2026 18:17:45 -0500 Subject: [PATCH 3/5] fix(otel-collector): document OpAMP-mode statsd wiring, restore enable_simple_tags MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Adding statsd to the default `metrics` pipeline via CUSTOM_OTELCOL_CONFIG_FILE silently no-ops in OpAMP supervisor mode (HyperDX's default deployment): opampController.ts overwrites that pipeline's receivers/exporters at runtime the same way it does for spanmetricsconnector (see the span metrics section above, #2351). Added a second example using a dedicated `metrics/statsd` pipeline name, which the remote config never touches, mirroring the existing OpAMP-mode span_metrics example. Verified against the real compiled binary by reconstructing opampController's generated base config as a static fixture and running `validate --config` against the merge. - Corrected the multi-replica warning: it was written as if OpAMP mode itself implies multiple replicas, but HyperDX's shipped default deployments run the OpAMP-managed collector as a single instance — the actual constraint is scaling *whichever* collector ingests statsd beyond one replica, independent of which mode it runs in. - Restored `enable_simple_tags: true` to both examples and qualified the intro claim about DogStatsD tag support: valued tags work with no extra config, but bare tags (`#mytag`) are silently dropped without this option. Co-Authored-By: Claude Sonnet 5 --- packages/otel-collector/README.md | 69 ++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 20 deletions(-) diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 16fb636b80..7cbf5a26d3 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -328,20 +328,25 @@ checks aren't hiding a startup-only failure here. `statsdreceiver` is compiled in so StatsD-formatted metrics can be ingested directly, without a separate StatsD-to-OTLP bridge — this covers any StatsD client, not just a particular vendor's dialect. It also understands the -DogStatsD tag extension (`#tag1:value1,tag2:value2`) if the sender uses it, -but doesn't require it. - -**This component does not support horizontally-scaled deployments.** -That's not a HyperDX-specific limitation — it's stated directly in the -upstream component's own docs: aggregation state lives independently in -each collector instance's memory, with no cross-replica coordination, so -the same metric arriving at two different replicas (e.g. behind a -load-balanced Service) would be aggregated separately by each one instead -of together, silently splitting/undercounting it. HyperDX's own -OpAMP-managed `otel-collector` Deployment runs multiple replicas for OTLP -ingest availability — **do not** wire `statsd` into that Deployment's -pipelines. Run it on a dedicated, single-replica collector instance -instead (standalone mode, as below). +DogStatsD tag extension: valued tags (`#tag:value`) work with no extra +config, but bare tags (`#mytag`, no value) need `enable_simple_tags: true` +(set below) or they're silently dropped. + +**This component does not support horizontally-scaled deployments** — not +a HyperDX-specific limitation, it's stated directly in the upstream +component's own docs. Aggregation state lives independently in each +collector instance's memory with no cross-replica coordination, so the +same metric landing on two different replicas (e.g. behind a +load-balanced Service) gets aggregated separately by each one instead of +together, silently splitting/undercounting it. HyperDX's default +deployments (`docker-compose.yml`, the all-in-one image) run the +OpAMP-managed collector as a single instance, so this doesn't bite out of +the box — it only matters once *some* collector instance ingesting +`statsd` is scaled to multiple replicas, at which point that traffic +needs to go to one dedicated single-replica instance instead. + +**In standalone mode**, `metrics` is a plain pipeline in your own config +file, so add `statsd` to it directly: ```yaml receivers: @@ -349,18 +354,42 @@ receivers: # Defaults to localhost:8125, which only accepts local traffic - # override to 0.0.0.0 if metrics arrive from outside the container. endpoint: 0.0.0.0:8125 + enable_simple_tags: true service: pipelines: metrics: receivers: [otlp/hyperdx, statsd] ``` -Verified against the real compiled binary via `otelcontribcol validate ---config` and an actual startup run (`otelcontribcol --config ...`, -reaching `"Everything is ready. Begin running and processing data."`), -in both cases using the same multi-`--config` merge the container -actually uses (base `config.yaml` + `standalone-config.yaml` + this -fragment). +**In OpAMP supervisor mode** (HyperDX's default), editing the `metrics` +pipeline this way will not work: `receivers:`/`exporters:` on it are set +dynamically by `opampController.ts`, and the remote config overwrites +(not merges) those keys — see the [span metrics section +above](#computing-span-metrics-from-ingested-traces) for the same +behavior. Adding `statsd` to `metrics.receivers` from +`CUSTOM_OTELCOL_CONFIG_FILE` gets silently dropped: no error, the receiver +just starts and is wired to nothing. Use a separate pipeline name instead, +which the remote config never touches: + +```yaml +receivers: + statsd: + endpoint: 0.0.0.0:8125 + enable_simple_tags: true +service: + pipelines: + metrics/statsd: + receivers: [statsd] + exporters: [clickhouse] + processors: [memory_limiter, batch] +``` + +Both shapes are verified against the real compiled binary via +`otelcontribcol validate --config`, and the standalone shape was also +actually run (`otelcontribcol --config ...`, not just `validate`), +reaching `"Everything is ready. Begin running and processing data."` — +the same rigor as the span metrics section above, using the same +multi-`--config` merge the container actually uses. ## Overriding base components via `CUSTOM_OTELCOL_CONFIG_FILE` From 867f232b95b5b794dd54fd882c6ae39bbd67997d Mon Sep 17 00:00:00 2001 From: arj22 Date: Wed, 9 Sep 2026 22:15:13 -0500 Subject: [PATCH 4/5] fix(otel-collector): publish statsd's UDP port in shipped Docker setups StatsD defaults to UDP, and neither the shipped compose files nor the collector/all-in-one Dockerfiles published or exposed 8125 - a user who added the documented statsd config verbatim on top of the default stack got no data and no error (UDP sends are fire-and-forget, so nothing logs the drop). Publish `8125:8125/udp` on the otel-collector service in docker-compose.yml and docker-compose.dev.yml (parameterized via HDX_DEV_OTEL_STATSD_PORT, matching the other dev port overrides), and add 8125/udp to the EXPOSE lines in docker/otel-collector/Dockerfile (both dev and prod stages) and docker/hyperdx/Dockerfile (all-in-one). statsd itself stays opt-in (nothing listens on the port until a pipeline config adds the receiver), so this is a no-op for anyone not using it. Co-Authored-By: Claude Sonnet 5 --- docker-compose.dev.yml | 1 + docker-compose.yml | 1 + docker/hyperdx/Dockerfile | 2 +- docker/otel-collector/Dockerfile | 4 ++-- packages/otel-collector/README.md | 10 +++++++++- 5 files changed, 14 insertions(+), 4 deletions(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index deae228acf..0117f5c373 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -55,6 +55,7 @@ services: - '${HDX_DEV_OTEL_GRPC_PORT:-4317}:4317' # OTLP gRPC receiver - '${HDX_DEV_OTEL_HTTP_PORT:-4318}:4318' # OTLP http receiver - '${HDX_DEV_OTEL_METRICS_PORT:-8888}:8888' # metrics extension + - '${HDX_DEV_OTEL_STATSD_PORT:-8125}:8125/udp' # statsd receiver (opt-in via custom.config.yaml) restart: always networks: - internal diff --git a/docker-compose.yml b/docker-compose.yml index 041023c827..659f43b365 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -42,6 +42,7 @@ services: - '4317:4317' # OTLP gRPC receiver - '4318:4318' # OTLP http receiver - '8888:8888' # metrics extension + - '8125:8125/udp' # statsd receiver (opt-in via CUSTOM_OTELCOL_CONFIG_FILE) restart: always networks: - internal diff --git a/docker/hyperdx/Dockerfile b/docker/hyperdx/Dockerfile index 1568263449..724fd5ea34 100644 --- a/docker/hyperdx/Dockerfile +++ b/docker/hyperdx/Dockerfile @@ -217,7 +217,7 @@ WORKDIR /app # Add hosts entry in entrypoint script instead of here # Expose ports -EXPOSE 8080 4317 4318 13133 8123 9000 +EXPOSE 8080 4317 4318 13133 8123 9000 8125/udp # all-in-one no auth ############################################################################################ FROM all-in-one-base AS all-in-one-noauth diff --git a/docker/otel-collector/Dockerfile b/docker/otel-collector/Dockerfile index a334d9150e..18355cd66e 100644 --- a/docker/otel-collector/Dockerfile +++ b/docker/otel-collector/Dockerfile @@ -79,7 +79,7 @@ COPY --chown=10001:10001 docker/otel-collector/config.standalone.promql.yaml /et COPY --chown=10001:10001 docker/otel-collector/supervisor_docker.yaml.tmpl /etc/otel/supervisor.yaml.tmpl COPY --chown=10001:10001 docker/otel-collector/schema /etc/otel/schema -EXPOSE 4317 4318 13133 +EXPOSE 4317 4318 13133 8125/udp HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \ CMD wget -q --spider http://localhost:13133/ || exit 1 @@ -103,7 +103,7 @@ COPY --chown=10001:10001 docker/otel-collector/config.standalone.promql.yaml /et COPY --chown=10001:10001 docker/otel-collector/supervisor_docker.yaml.tmpl /etc/otel/supervisor.yaml.tmpl COPY --chown=10001:10001 docker/otel-collector/schema /etc/otel/schema -EXPOSE 4317 4318 13133 +EXPOSE 4317 4318 13133 8125/udp HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \ CMD wget -q --spider http://localhost:13133/ || exit 1 diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index 7cbf5a26d3..d728423672 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -345,6 +345,12 @@ the box — it only matters once *some* collector instance ingesting `statsd` is scaled to multiple replicas, at which point that traffic needs to go to one dedicated single-replica instance instead. +`docker-compose.yml`/`docker-compose.dev.yml` publish `8125:8125/udp` on +the `otel-collector` service, and the standalone/all-in-one images +`EXPOSE 8125/udp`, so the port is reachable from outside the container as +soon as `statsd` is added to a pipeline below — no separate port-mapping +step needed on top of these examples. + **In standalone mode**, `metrics` is a plain pipeline in your own config file, so add `statsd` to it directly: @@ -352,7 +358,9 @@ file, so add `statsd` to it directly: receivers: statsd: # Defaults to localhost:8125, which only accepts local traffic - - # override to 0.0.0.0 if metrics arrive from outside the container. + # 0.0.0.0 is required here since traffic arrives from outside the + # container (the published port above forwards to the container's + # own loopback otherwise). endpoint: 0.0.0.0:8125 enable_simple_tags: true service: From bba51b6246adfb2e194c4b558b67a241dfa4c97a Mon Sep 17 00:00:00 2001 From: arj22 Date: Wed, 9 Sep 2026 22:26:36 -0500 Subject: [PATCH 5/5] fix(otel-collector): make statsd's compose port mapping opt-in An unconditional fixed host port mapping fails the whole otel-collector service's startup if anything else already owns 8125/udp (a local StatsD daemon or Datadog Agent commonly does) - since statsd itself is opt-in and off by default, this would have broken the default stack for anyone not using the feature, just for having it in their environment. Commented the mapping out by default in both compose files, matching the existing opt-in-port convention already used in the same files (db's 27017, ch-server's 8123/9000). Uncommenting it remains a one-line change, documented in the README section. --- docker-compose.dev.yml | 7 ++++++- docker-compose.yml | 9 ++++++++- packages/otel-collector/README.md | 14 +++++++++----- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index 0117f5c373..55a756367f 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -55,7 +55,12 @@ services: - '${HDX_DEV_OTEL_GRPC_PORT:-4317}:4317' # OTLP gRPC receiver - '${HDX_DEV_OTEL_HTTP_PORT:-4318}:4318' # OTLP http receiver - '${HDX_DEV_OTEL_METRICS_PORT:-8888}:8888' # metrics extension - - '${HDX_DEV_OTEL_STATSD_PORT:-8125}:8125/udp' # statsd receiver (opt-in via custom.config.yaml) + # Uncomment to enable statsd ingestion (also requires adding the + # `statsd` receiver to a pipeline in custom.config.yaml - see + # packages/otel-collector/README.md). Left commented out by default: + # a fixed host port mapping fails the whole service's startup if + # something else on the host already owns 8125/udp. + # - '${HDX_DEV_OTEL_STATSD_PORT:-8125}:8125/udp' # statsd receiver restart: always networks: - internal diff --git a/docker-compose.yml b/docker-compose.yml index 659f43b365..c45ae11c0b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -42,7 +42,14 @@ services: - '4317:4317' # OTLP gRPC receiver - '4318:4318' # OTLP http receiver - '8888:8888' # metrics extension - - '8125:8125/udp' # statsd receiver (opt-in via CUSTOM_OTELCOL_CONFIG_FILE) + # Uncomment to enable statsd ingestion (also requires adding the + # `statsd` receiver to a pipeline via CUSTOM_OTELCOL_CONFIG_FILE - see + # packages/otel-collector/README.md). Left commented out by default: + # a fixed host port mapping fails the whole service's startup if + # something else on the host (e.g. a local StatsD daemon or Datadog + # Agent) already owns 8125/udp, which would otherwise break the + # default stack for anyone not using this feature. + # - '8125:8125/udp' # statsd receiver restart: always networks: - internal diff --git a/packages/otel-collector/README.md b/packages/otel-collector/README.md index d728423672..d200712ccf 100644 --- a/packages/otel-collector/README.md +++ b/packages/otel-collector/README.md @@ -345,11 +345,15 @@ the box — it only matters once *some* collector instance ingesting `statsd` is scaled to multiple replicas, at which point that traffic needs to go to one dedicated single-replica instance instead. -`docker-compose.yml`/`docker-compose.dev.yml` publish `8125:8125/udp` on -the `otel-collector` service, and the standalone/all-in-one images -`EXPOSE 8125/udp`, so the port is reachable from outside the container as -soon as `statsd` is added to a pipeline below — no separate port-mapping -step needed on top of these examples. +`docker-compose.yml`/`docker-compose.dev.yml` have an `8125/udp` port +mapping for the `otel-collector` service, commented out by default — +uncomment it to enable statsd ingestion. It's opt-in rather than always +on because a fixed host port mapping fails that service's entire startup +if something else on the host (e.g. a local StatsD daemon or Datadog +Agent) already owns 8125/udp, which would otherwise break the default +stack for anyone not using this feature. The standalone/all-in-one images +also `EXPOSE 8125/udp` (metadata only, doesn't reserve the host port on +its own). **In standalone mode**, `metrics` is a plain pipeline in your own config file, so add `statsd` to it directly: