Repository navigation
Prometheus, Loki and Alertmanager bind 0.0.0.0 with no authentication #70
Description
.env.example:11 ships BIND_ADDR=0.0.0.0, and the deployed .env on the
monitoring host has the same. Confirmed with ss -ltn on prometheus
(10.0.99.20):
LISTEN 0.0.0.0:9090 prometheus
LISTEN 0.0.0.0:9093 alertmanager
LISTEN 0.0.0.0:3100 loki
LISTEN 0.0.0.0:3000 grafana
UNCONN 0.0.0.0:1514 alloy (syslog)
Only Grafana authenticates. For the other three, anyone who can reach the host
on VLAN 99 can:
- Prometheus — inject arbitrary metrics (
--web.enable-remote-write-receiver)
and force a config reload (--web.enable-lifecycle); - Loki — push arbitrary log lines and issue delete requests
(auth_enabled: false); - Alertmanager — create silences, i.e. switch off any alert.
The last one is the interesting one: silencing an alert is a quiet way to
disable monitoring, and the stack's own record of that action is the thing being
disabled.
This is not currently exploitable from an untrusted segment
Default-deny between VLANs means only VLAN 50 can reach VLAN 99 at all
(docs/adr/0002). So the firewall is the entire control, and there is no
defence in depth behind it — a compromised workstation on the trusted segment
has unauthenticated write access to the metrics and log stores.
.env.example:9-11 already documents that BIND_ADDR can be set to the
management IP. The question is whether the shipped default should be the wide
one.
Options
- Ship
BIND_ADDR=127.0.0.1and publish only Grafana, which is the only service
with a login. Alloy's syslog listener has to stay reachable off-host and is
separately bound. - Or keep it and record the residual in
SECURITY.md, which already carries an
exposure table.
Found while verifying #12.
Activity
fix(observability): narrow to Alertmanager — oracle is a live client (#…
Merge pull request #181 from Gerrrt/gerrrt/monitoring-bind-addr-secur…
Gerrrt commentedon Aug 30, 2026
Closed by #181, deployed to the lab and verified 2026-08-30. Partly fixed, partly accepted — worth spelling out, since the title names three services and only one of them moved.
Fixed: Alertmanager
9093 now binds to
127.0.0.1. This was the sharpest of the three for the reason the issue gives — a silence switches off monitoring, and the record of it lives in the system being switched off.It cost nothing to close. Nothing off-host ever used the port: Grafana proxies Alertmanager over the compose network through an authenticated datasource, so every operator path to a silence already avoided 9093. Verified after deploy:
Not fixed: Prometheus and Loki
Both stay on
${BIND_ADDR}, i.e.0.0.0.0. The option-1 plan in this issue would have broken the lab, and only checking the running host caught it.docs/roadmap.mdclaimed only the monitoring host ran an agent. It was about a day out of date.oraclehad been remote-writing to10.0.99.20:9090and pushing to:3100for ~18 hours — 378 distinct metric names — and it is not a Prometheus scrape target, so those published ports are its only path. Loopback-binding them would have cut its metrics and logs silently, because Alloy retries with backoff and buffers rather than failing loudly.So the write surface this issue describes for those two is real and still open: anything that can route to
10.0.99.20can read every metric and log line, inject metrics through the remote-write receiver, and delete log ranges through Loki's delete API. Firewall default-deny is the only control, exactly as the issue says. That is now recorded as an accepted residual inSECURITY.mdalongside the SNMP and unencrypted-disk entries, rather than carried implicitly.Closing it properly means authentication in front of the ingest ports plus a credential on every agent — separate work, and a new secret to rotate. Not tracked as an issue yet.
Answering the question the issue actually asked
Two findings settle it, and neither was what I expected going in:
scripts/render-config.shregenerates.envfrom.env.exampleon everymake up, so a hand-edit of.envnever survived a deploy.docs/architecture.mdsayingBIND_ADDR"is set in.env" was misleading; fixed.10.0.99.20/24.0.0.0.0and10.0.99.20are the same thing on this machine, and pinning the literal address would only add a way for the stack to fail if it ever moved.The rule that replaced the shared default is in ADR-0012: publish a port when something off-host uses it, and name the client. That also means #88's
Sarumandeploy needs no bind change —oraclealready made Prometheus and Loki published services. Roadmap #88 is corrected accordingly.Gerrrt commentedon Aug 30, 2026
Follow-up opened as #182 — authenticating the ingest ports, which is what would retire the Prometheus/Loki residual above.