Repository navigation
docs(runbook): build the lab guest on Saruman (#262) - #280
Merged
Merged
Conversation
`stacks/lab` is built, validated and deployed nowhere: the guest it runs on does not exist. This is that build, and it settles the three things #262 left open — the name, the address, and the distribution. The distribution is the one that would have bitten quietly. `config.alloy` tails /var/log/auth.log and /var/log/syslog as explicit file sources alongside the journal, and a default Debian 12/13 install ships no rsyslog — neither file exists. Alloy would start, report healthy, and collect nothing from two of its four log sources. That is #62 and #63 again, and it is why this says Ubuntu Server 24.04: it has both files, and it is what `prometheus` and `oracle` already run, so the agent behaves identically on every Linux host here. A VM and not an LXC, because the stack uses `cgroup: host`, `cap_drop: [ALL]` and a Docker socket mount, and each of those has an asterisk under a container that it does not have under a VM. Four `qm create` flags are explained rather than left to be copied: no VLAN tag (vmbr0 IS ImaginationLAN — `Saruman` is single-homed with no trunk, ADR-0007, so a tag would leave the guest with no network), `--balloon 0` (a guest whose job is reporting its own memory should not have that number move for reasons that are not the workload), `--onboot 1` (nothing converges this stack and nothing outside the lab notices if it dies, #257), and `iothread` needing the `-single` controller to do anything. Disk cache left at the default rather than `writeback`, and the reason is in #76's runbook: the Smart Array cache is battery-backed again since 2026-09-02, but `cpqDaAccelWriteCachePercent` still reads 0, unexplained. Leaning on a cache nobody has confirmed is absorbing writes is not a trade to make silently. Two traps get their own callouts because both are easy and both are quiet: the missing-certificate error names `make certs --ca` as its fix, which here would mint a second CA that this stack's Prometheus cannot verify against; and `secrets-init` on the wrong host would give one age key both stacks, which bootstrap.sh now refuses but only helps if the refusal is read. The verification asks for three Loki jobs and says that two is the failure the runbook exists to prevent, rather than a quiet host — the whole point being that a healthy-looking agent is exactly what the wrong distribution produces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of getting
stacks/labactually running, as something the repository keeps rather than a message in a chat window.Settles the three things #262 left open — name, address, distribution — and writes down the reasoning for each.
The distribution is the one that matters
config.alloytails/var/log/auth.logand/var/log/syslogas explicit file sources, alongside the journal. A default Debian 12/13 install ships no rsyslog — journald only, and neither file exists. Alloy would start cleanly, report healthy, and collect nothing from two of its four log sources.That is #62 and #63 again: a collector that runs healthy and produces nothing. So: Ubuntu Server 24.04 LTS, which has both files and is what
prometheusandoraclealready run, keeping the agent identical on every Linux host in the estate.The verification step asks for three Loki jobs and says explicitly that two is the failure this runbook exists to prevent — because a healthy-looking agent is exactly what the wrong choice produces.
Other decisions, with reasons rather than defaults
cgroup: host,cap_drop: [ALL]and a Docker socket mount; each has an asterisk under a container that it does not have under a VM.vmbr0is ImaginationLAN —Sarumanis single-homed with no trunk and no VLAN-aware bridge (ADR-0007) — so atag=would leave the guest with no network at all.--balloon 0. A guest whose job is reporting its own memory should not have that number move for reasons that are not the workload.--onboot 1. Nothing converges this stack and nothing outside the lab notices if it dies (#257); coming back by itself is the cheapest resilience available.writeback. The Smart Array cache is battery-backed again since 2026-09-02 (#76) — but that runbook recordscpqDaAccelWriteCachePercentstill reading0, unexplained. Leaning on a cache nobody has confirmed is absorbing writes is not a trade to make silently.Two quiet traps get callouts
make certs --caas its fix, which is right on a fresh clone and wrong here — it would mint a second CA that this stack’s Prometheus cannot verify against. Copy the leaf; do not issue a new root.secrets-initon the monitoring host would give one age key both stacks.bootstrap.shrefuses, but a refusal only helps if it is read, so the runbook says which machine you are meant to be on and what the refusal means.Verification
make check-docsandmake lintpass. The runbook is indexed from the roadmap entry and the README tree.Nothing was deployed —
Sarumanis not reachable from the monitoring host, which is why this is a runbook and not a change.🤖 Generated with Claude Code