Repository navigation
Screenshot capture, and three faults it uncovered - #62
Merged
Merged
Conversation
Three faults, all with the same shape: a component reporting itself healthy
while producing nothing, and no alert able to tell the difference.
1. Alloy's container hostname hijacked DNS for the Prometheus service.
`hostname: ${ALLOY_HOSTNAME}` was the only way to give constants.hostname a
real name, and it also registers that name in Docker's embedded DNS. This
host is called `prometheus`, so the compose network ended up with two A
records for `prometheus` — the metrics store, and the agent. Every
remote_write to http://prometheus:9090/api/v1/write was a coin toss, and the
losing side is `connection refused` from Alloy's own address.
Measured: node and cAdvisor metrics stopped at 02:50 and did not come back
until the container was recreated at 15:29 — twelve and a half hours — while
`up{job="alloy"}` read 1 for every scrape in between, because Alloy's own
/metrics endpoint was never the thing that broke.
config.alloy now reads ALLOY_HOSTNAME from the environment and the container
carries no hostname at all, so it registers nothing. The instance label is
set from the same value in both relabel blocks; it used to come from
constants.hostname by default, and without this it would have quietly
reverted to the container ID that 9405de0 removed.
2. cAdvisor could only see its own cgroup.
`privileged: true` is not enough on cgroup v2: Docker's default namespace
mode is `private`, so the container's own cgroup is the root of what it can
see. cAdvisor reported exactly one series, id="/", and reported it happily.
No per-container metric has existed for at least the last 24 hours, which
makes the whole Docker Containers dashboard and four of the eight rules in
containers.rules.yaml decorative. `cgroup: host` fixes it.
3. That fix, on its own, was worse. With the host's cgroups visible cAdvisor
also enumerates every systemd slice on the machine: 3365 series where 450
are Docker's, and every dashboard and rule here filters on name!="".
`docker_only = true` brings it back to 828.
Verified on the lab host: `prometheus` resolves to one address again, seven
named containers report, node metrics are flowing, and the host/instance
labels are unchanged from what 9405de0 intended.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each of these variables ships with `current: {}` and `options: []`, because a
provisioned dashboard has nowhere to persist a selection — `allowUiUpdates` is
false by design. Grafana therefore has to resolve the option list before it can
expand "All" into a regex, and the panels do not wait for it. With no
`allValue`, `$host` and `$container` and `$instance` interpolate to nothing and
every panel that filters on one returns no data.
The Logs dashboard was the visible casualty: 10 of its 13 panels were empty,
including all three log streams. The only panel that worked was "Lines/s (all
hosts)", which is the one query on the dashboard that does not reference a
variable.
`allValue: ".+"` removes the dependency on the option list entirely — "All"
becomes a fixed regex that means what it says. Not ".*": Loki rejects a
selector whose every matcher is empty-compatible, with a parse error the panel
renders as "No data", which is the same symptom for a different reason.
Confirmed against the running stack: the Logs dashboard now returns 34k lines
across both hosts, and Log volume by level resolves four levels.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Screenshots were the one artefact this repository could not produce from its own tooling. The monitoring host has no browser, no image libraries and no display; Grafana has no built-in PNG export and is HTTPS-only with anonymous access off. `docs/images/README.md` described a manual capture from a browser somewhere else, which is why it had never happened. `make screenshots` starts a headless renderer behind the `capture` compose profile, shoots all five dashboards, and stops it again. The profile keeps it out of `make up` entirely — it is not pulled, not started, and `docker compose ps` still shows six services. Notes for the next person, because none of this is guessable: - The renderer is Go with a headless Chromium since v5, not the Node service the published examples describe. IGNORE_HTTPS_ERRORS, NODE_EXTRA_CA_CERTS and RENDERING_IGNORE_HTTPS_ERRORS are all silently ignored; the flag that works is BROWSER_FLAGS. `grafana-image-renderer server --help` is the real list. - Chromium rejects the lab CA and cannot be handed a PEM — it wants an NSS database — so this accepts any certificate, narrowly: the container lives for the seconds a capture takes, is not published, and only ever talks to Grafana on the internal bridge. - Grafana refuses to START, not to render, if a rendering server is configured and the renderer token is left at its default. GRAFANA_RENDERER_TOKEN is therefore required rather than optional, and guarded in compose so a missing value is a named error at `make up` instead of a crash loop on the dashboard host. - Height is derived per dashboard from its own JSON. A fixed height crops whatever falls below it and looks deliberate; the first run of this produced five images that stopped halfway down. - Grafana answers a failed render with 200 and a PNG of an error message, so the script checks the magic bytes and the size rather than the exit status. The Grafana password and the renderer token are read in-process through scripts/secrets-env.sh and handed to curl via --config, never argv. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 21, 2026
This was referenced Aug 21, 2026
Gerrrt
pushed a commit
that referenced
this pull request
Aug 21, 2026
#114 is the other half of #63's "two ways out". It goes in Monitoring rather than Infrastructure because it is about the observability stack, not the rack, and it records why it is not simply actionable: cAdvisor has only reported correctly since #62, so there are hours of history to size a limit from rather than the 30 days retention implies. The #63 paragraph now points at it, since "so should we set limits?" is the next question a reader has.
This was referenced Aug 21, 2026
This was referenced Aug 31, 2026
Closed
Closed
Closed
Gerrrt
added a commit
that referenced
this pull request
Sep 4, 2026
#107) `${VAR:?message}` in compose.yaml is the declaration that a service cannot start without VAR, so two scripts derive what a stack requires from it: seed-validation-env.sh must satisfy every guard to get `docker compose config` past them, and render-config.sh must ask SOPS for every guarded secret. Both grepped the raw file, which reads comments as well as YAML — so a comment that *mentioned* the guard syntax became a guard, and validation demanded a value for a variable that does not exist: compose.yaml guards VAR, which the validation .env does not set. That was hit while writing #62, and the fix then was to reword the comment, which is the wrong way round: documentation should not have to avoid describing the thing it documents. Read the parsed document instead, where a comment cannot appear. `docker compose config` is not the authority to ask — it resolves the guards rather than reporting them, and it is the thing being satisfied — so this walks every scalar of the PyYAML parse, which check_compose_health.py already depends on. One script, not two copies of the fix. render-config.sh's comment already described its derivation as "the same one seed-validation-env.sh uses"; now it literally is, in scripts/compose-guards.sh, so the two cannot drift and the comment bug is not left standing in the caller the issue did not name — where it is worse, because there it sends a render to SOPS for a key no stack has. PyYAML is asked for and worked around if absent, rather than required: renders happen on deploy hosts, and making one depend on a Python library to read a list of variable names is a worse trade than the fallback, which strips what YAML would treat as a comment and greps what is left. That is right about the case this exists for and can only ever miss a guard hidden behind a `#` inside a quoted string — the old failure mode, not a new false demand. Both callers read the list through command substitution, because a process substitution's exit status is not the loop's: a failure in the helper would otherwise arrive as an empty guard list, and read as "this stack requires nothing". The guard convention is now documented in the header of stacks/observability/ compose.yaml — the documentation the old grep forbade, and a live regression test for this fix: it names `${VAR:?message}`, so a return to grepping the raw file fails `make validate` with the exact error from the issue. Verified: guards for both stacks unchanged (parsed and fallback paths agree with the old grep on both files), a guard added to a compose.yaml is still caught by name, `make validate` passes, and the old grep sees VAR in the new comment while neither new path does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gerrrt
added a commit
that referenced
this pull request
Sep 4, 2026
… talk around (#107) `5625c52` (#62) explains why GRAFANA_RENDERER_TOKEN is required rather than optional, then describes the mechanism as "the required-variable guard below" — a paraphrase, written three days after the raw-grep check landed in `595f0e2` and made naming the syntax fatal. Writing what it means would have demanded a throwaway value for a variable that does not exist before `make validate` would pass. That is the rewording #107 was filed about, and the previous commit is what removes the need for it. There is no earlier text to restore: the rewording happened while the comment was being written, so no commit ever carried the version this puts in its place. Nothing else in either compose.yaml is talking around a guard — the alertmanager note names `${RENDER_UID:-65534}`, and the `:-` form was never matched by the grep. Verified: the stack's guards are unchanged on both extraction paths, seed + `docker compose config` pass, yamllint clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 4, 2026
feat(soc): author stacks/soc — Wazuh and Velociraptor for odin, ahead of the guest (#266, #267)
#434
Merged
11 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #12: the stack is verified clean on the real host, and dashboard
screenshots can now be produced with one command. The images themselves are not
in this PR — see Why no PNGs yet below.
Verified against the lab host
git pull && make validate && make up, then every acceptance check in #12:RestartCount=0on every onedepends_ondeadlock; both startcurl -s localhost:3100/readyreadyInstanceDown{job="loki"}No legacy container-name collision occurred; the
docker rm -fline in theissue was not needed.
Three faults found on the way
Each one had the same shape — healthy component, no data, nothing alerting.
1. Alloy was answering to the name of the server. The host is called
prometheus,render-config.shwritesALLOY_HOSTNAME=$(hostname), andcompose passed that to
hostname:— which registers the name in Docker'sembedded DNS. Two A records for
prometheuson one network, so everyremote_write was a coin toss. Measured: host and container metrics stopped for
twelve and a half hours while
up{job="alloy"}read 1 for every scrape inbetween.
2. cAdvisor could only see its own cgroup.
privileged: trueis not enoughon cgroup v2 — the default namespace mode is
private, so the container's owncgroup is the root it can see. It reported one series,
id="/", and reported ithappily. No per-container metric had existed for at least 24 hours, which
makes the Docker Containers dashboard and four of the eight rules in
containers.rules.yamldecorative.cgroup: hostfixes it;docker_only = truestops the fix costing 2500 series of systemd slices.
3. "All" selected nothing. Provisioned dashboards ship
current: {}andoptions: []becauseallowUiUpdatesis false, so Grafana has to resolve theoption list before it can expand "All" — and the panels do not wait. 10 of the
Logs dashboard's 13 panels were empty.
allValue: ".+"makes "All" a fixedregex. Not
.*: Loki rejects a selector whose every matcher isempty-compatible, and renders the parse error as "No data".
The screenshot pipeline
make screenshotsstarts a headless renderer behind thecapturecomposeprofile, shoots all five dashboards, stops it again.
make upneither pulls norstarts it, and
docker compose psstill shows six services.The commit message carries the things that are not guessable: the renderer is Go
with a headless Chromium since v5 so every
RENDERING_*andNODE_*variable inthe published examples is silently ignored, Grafana refuses to start rather
than to render if the renderer token is left at its default, and Grafana answers
a failed render with 200 and a PNG of an error message.
GRAFANA_RENDERER_TOKENis a new required key insecrets/observability.sops.yaml.Why no PNGs yet
All five render correctly, but the window is
now-24hand fault 1 was onlyfixed today — so the host and Docker dashboards currently picture their own
outage. Re-running
make screenshotstomorrow is the fix, and it is one command.The Logs dashboard also needs a decision before it is published: now that it
works, its Authentication log panel renders
auth.logverbatim — realusernames, real source addresses.
docs/images/README.mdnow says so explicitly.🤖 Generated with Claude Code