Repository navigation
Every validator is pinned to stacks/observability — a second stack would be checked by nothing #263
Description
Split out of #101, which
recorded that "the Makefile is already stack-parameterised (STACK ?= observability), so the tooling supports it."
Half of that is true, and it is the half that does not matter. STACK ?=
reaches the lifecycle and secrets targets — up, down, render,
secrets-edit, reload. Every validator is pinned to the one directory:
Pinned to stacks/observability |
|
|---|---|
scripts/validate.sh:17 |
STACK="stacks/observability" |
scripts/check_docs.py:72 |
STACK = REPO / "stacks/observability" |
scripts/check_dashboards.py:37,38,56 |
dashboards, datasources, alloy dir |
scripts/check_compose_health.py:63 |
DEFAULT = ... |
scripts/check_loki_rules.sh:28 |
STACK= |
scripts/seed-validation-env.sh:26 |
STACK= |
.github/workflows/ci.yml:18 |
STACK: stacks/observability |
make check-rules is the exception that proves it: it uses $(STACK_DIR) and
so would follow STACK=lab, while make validate — which is what CI runs and
what a contributor runs — would not.
A stacks/lab/ created before this work is a stack CI has never seen. Its
compose is not config-checked, its rules are not promtool-tested, its
dashboards are not checked against its own datasources, its healthcheck
binaries are not probed, and its images are pinned and digest-verified by
nothing. Every guard this repository has built for the estate's stack would be
silently absent for the lab's — which is the shape of #62, #65, #68 and #79 all
over again, in a directory nobody is watching.
Worth deciding
- Iterate, or matrix? A
for stack in stacks/*/insidevalidate.shis
one change and keeps a single CI job. A GitHub Actions matrix gives a red
square per stack and a clearer failure, at the cost of the job list growing
with the estate. - What is genuinely per-stack and what is repo-wide.
check_docs.py
asserts prose against configs and much of it is estate-specific — the SNMP
targets, the ports table, the compute table. Those are not "run it twice"
candidates; splitting repo-wide assertions from per-stack ones is most of the
work here. - What a stack is allowed to lack. The lab has no snmp-exporter,
no blackbox-exporter and no Alertmanager (ADR-0019), so every checker needs
to distinguish "absent by design" from "missing".render-config.shalready
guards each render behind anif [[ -f ... ]]; itsREQUIREDarray does
not, and today demands all four SNMP communities and all four Alertmanager
URLs from any stack.
Done when
make validate and CI both check every directory under stacks/, and adding
an empty one fails loudly rather than passing quietly.
Activity
Gerrrt commentedon Sep 4, 2026
Done in #278 — stacked on #276, since this exists to validate what that adds. All three CI jobs green, and the Validate job's own log shows it iterating both stacks.
Iterate, not matrix. A matrix would put stack knowledge back into
ci.yml, andlint.shalready asserts thatci.yml, theMakefileandvalidate.shdelegate to the same scripts rather than carrying their own copies.scripts/stacks.shis the one definition, read by everything that iterates — including the Python checkers, because a second implementation in Python would be a second definition of what a stack is. A directory understacks/with nocompose.yamlis an error, not a skip, which was the acceptance criterion here.On the third open question — what a stack is allowed to lack — absences are reported as passes, not skips. A skip means "could not check, and therefore proved nothing", and
validate.shcounts them so a run cannot claim to have checked what it did not (#68). "This stack has no Alertmanager" is a verified fact about the tree. The skip count is still 2, the same two as before.Three guards got stronger rather than just wider:
promtoolunit tests now fail.check rulesnever asks whether an expression can be true — ContainerHighMemory cannot fire — no service sets a memory limit #63 was unfireable for months while passing it.reload-config.shcross-check split in two: per-stack for entries a stack declares, plus a--cross-stackmode asserting every entry exists in some stack. "Not in this compose file" stopped meaning "in no stack at all" the moment there were two.ABSENT_BINARIESgot the same treatment.check_docs.pyglobsstacks/*/README.mdinto its prose set, so a version pin in any stack README goes stale loudly (stacks/observability/README.md lists six stale image versions, and CI does not check Markdown #73).pin-digests.shandmake check-digestsnow cover every stack too — the lab's four images were verified against the registry for the first time.Each guard was proved by introducing the fault rather than by reading the code: an empty
stacks/directory, a renamedSERVICESentry, a version pin in the lab README, and a deleted test file. Two bugs in my own new code turned up that way as well —mapfile < <(...)does not carry the subshell's exit status, so the stack-list guard would never have fired, and((n)) && cp ...underset -eexits when n is zero.Merge pull request #278 from Gerrrt/gerrrt/multi-stack-validators-263
Merge pull request #323 from Gerrrt/gerrrt/timers-not-on-the-lab-guest