[Gerrrt]

Closes #76.

The alert was right, and unusable

IloBatteryCondition fired unbroken from 2026-08-18 21:00Z to
2026-08-31 03:00Z — the entire retained window. It was correct the whole time
and still could not be acted on, because it read
cpqHeSysBackupBatteryCondition: one identity-free scalar for the whole battery
sub-system. It could say "degraded" and nothing more, while its summary claimed
to cover the RAID cache — which that metric cannot see. Nothing in the repo
referenced cpqDaAccel* at all.

So the issue's pointer to cpqDaAccel* was aimed at the right subtree and the
wrong rule, and that gap is most of what this PR closes.

The fault, decoded against the repo's own snmp.yaml

Metric	Value	Meaning
cpqHeSysBatteryProductName	HP Smart Storage Batt 96	spare 815983-001
cpqHeSysBatteryCondition	4	failed
cpqHeSysBatteryStatus	13	shutdownPermanentFailure
cpqDaAccelStatus	5	permDisabled
cpqDaAccelErrCode	25	noCapacitorAttached
cpqDaAccelRead/WriteCachePercent	0 / 0	running write-through
cpqDaAccelBadData	2	none — nothing dirty was lost

13 is terminal — not a flat pack, not a seating fault — so "reseat or
replace" was only ever replace. On a Gen9 this pack backs the Smart Array's
flash-backed cache, so it is one fault seen through two MIB subtrees, not two
problems. Everything else on the box reads ok: both PSUs, both physical
drives, the logical drive, ~30 temperature sensors.

One correction worth carrying: write-through is slower, not riskier. It
closes the cache-loss window rather than opening one. The cost is write latency
and IOPS on the hypervisor, which is why the new rule is warning.

Changes

• IloBatteryCondition re-keyed onto the per-battery table. It carries
  cpqHeSysBatteryStatus as the value, so $value is the diagnosis (13)
  rather than a condition code, and group_lefts the spare part number in so
  the page names the part to buy. Name kept, so the silence is not orphaned.
• IloWriteCacheDisabled (new) on cpqDaAccelStatus > 3 — the family that
  was scraped and read by nothing holds the one series stating what the fault
  costs. > 3 not != 3 because 1/2 are what a controller with no cache
  module reports; for: 1h because a pack recharging after a power cut sits in
  write-through legitimately.
• IloHardwareDegraded extended to the controller rollups at > 3, not
  > 2. At > 2 they fire today — measured, exactly three — because a flat
  cache battery degrades all three. That would put three critical alerts on
  the urgent receiver every 4h for a fault two warnings already describe.
• network.test.yaml — the file's first unit tests.
• check_docs.py now guards the unit-test coverage claim.
• 8th amtool route assertion, since nothing proved category=hardware
  reached a receiver at all.
• Hardware health panel extended to the array and battery rollups. Editing
  an existing panel, so the 84-panel count is unchanged.

Verification

./scripts/validate.sh passes end to end, including
amtool config routes test (8 assertions) and
docs OK — 40 Prometheus + 13 Loki rules, 5 dashboards, 84 panels.

Every expression was evaluated against the real retained series before being
written, not just unit-tested: the join returns one series at value 13
carrying cpqHeSysBatterySparePartNum="815983-001", cpqDaAccelStatus > 3
returns one at 5, and the extended IloHardwareDegraded returns empty.

Seven mutations were checked against the new tests and all seven fail,
including dropping the group_left, keying the join on instance alone, and
reverting to the old scalar.

Two things found on the way

• docs/observability.md claimed "Coverage is six rules of 39 ...
  ContainerHighMemory and Watchdog" — six claimed, two named, three more
  covered since blackbox.test.yaml landed. No pattern matched it, which is
  Alert-rule and panel counts are stale in 8 places, including a deploy-runbook check #72's failure mode surviving inside the fix for it. Both numbers are now
  checked and mutation-tested in both directions.
• The residual is recorded in roadmap.md, hardware.md and a new runbook —
  not in SECURITY.md. That table is a security register (credentials, bind
  addresses, an unencrypted disk); a dead battery is not an exposure, and its
  house phrase "not a fix in progress" would be false while a part is on order.

Silence

3ffc313f-7154-459c-9f62-7c9a432bc97e, matching
alertname=~"IloBatteryCondition|IloWriteCacheDisabled" and device="shiva",
until 2026-10-01. Deliberately narrow, so a genuinely new fault on this box
is not suppressed with it — and to be deleted when the pack goes in rather than
left to expire.

Not in this PR

shiva is not being scraped right now: the running prometheus container is
bind-mounted from a deleted worktree, so its file_sd targets directory is
empty. That is local container state, not a repo defect — a make up from a
current checkout restores it. It does mean the rules here are proven by unit
tests and by queries against retained history, not by a live firing alert.

🤖 Generated with Claude Code