[Gerrrt]

The pack is in, and the documents that said failed, spare in transit are now
wrong in the other direction. This moves them to fitted and proven — and
names the two readings that did not move, so "proven" is not read as "finished".

What was verified before anything was written

Saruman was off 22:30–22:58 UTC on 2026-09-02. The Observed column is
the 23:02 UTC scrape, the first after it came back, unchanged at 23:12.

Metric	Failed pack	Runbook expected	Observed
cpqHeSysBatteryCondition	4 failed	2 ok	2
cpqHeSysBatteryStatus	13 shutdownPermanentFailure	1 noError	1
cpqHeSysBatterySerialNumber	6EZBN0CB29N3YZ	different	6EZBN0FB2431YM
cpqDaAccelStatus	5 permDisabled	3 enabled	3
cpqDaAccelErrCode	25 noCapacitorAttached	1	2 — 1 and 2 are both "nothing wrong"; the table said 1
cpqDaAccelBattery	6 notPresent	2 ok	2
cpqDaAccelBackupPowerSource	1 other	—	4 smartbattery
cpqDaAccelBadData	2 none	2 none	2
cpqDaAccelCondition / cpqDaCntlrCondition / cpqDaCntlrBoardCondition	3 / 3 / 3	2	2 / 2 / 2
cpqDaAccelWriteCachePercent / ReadCachePercent	0 / 0	non-zero	0 / 0 — did not move
cpqDaAccelFailedBatteries	1	—	1 — did not move

The serial row is the one the runbook's table lacked and the one that settles
it: every other row could be the old pack reporting differently, and that one
cannot. Both proof rows are added to the runbook.

The silence

bfdfff66-d9c3-4df4-9495-f1f38ebf93c1 reads expired with endsAt
2026-09-02T23:11:53Z instead of its original 2026-10-01, so it was deleted
rather than left to lapse. IloBatteryCondition and IloWriteCacheDisabled
are live again over the new part; nothing is firing for shiva in Prometheus
or Alertmanager.

The order was inverted again — reading 23:02, deletion 23:11, the same
thing the UPS runbook recorded. It cost nothing here for a reason that case did
not have: both rules carry a for: of 30m and 1h, so no notification could
have existed inside nine minutes, suppressed or otherwise. The runbook says so
and still says delete first.

What did not move

• Cache ratio 0 / 0. The controller says the cache is enabled and
  backed by the new pack, and reports no read/write split, where a P440ar with
  a 2 GB module normally reports something like 10/90. Slow iLO refresh, a
  ratio left unset after weeks permanently disabled, or a first charge — ten
  minutes of history cannot tell them apart. If still 0 / 0 after an hour,
  the next step is ssacli ctrl all show config detail on Saruman, which is
  not reachable from the monitoring host. Not alerted on.
• cpqDaAccelFailedBatteries still 1. An octet-string bitmap that did
  not clear when cpqDaAccelBattery went to ok. Latched or cached; unknown.
  Not alerted on, recorded so nobody rediscovers it.

Files

Runbook (status, Observed column, two proof rows, ErrCode corrected, steps 3
and 4 marked done, a "What is still open" section); roadmap (#76 into Done);
network.rules.yaml header comment only — no rule expression changed, they
became correct on their own as designed; docs/observability.md,
docs/hardware.md, docs/images/README.md to past tense; the test file's
comments, since the quiet-pack fixture is now a measured reading too. No
dashboard change, so the panel-count coupling is untouched.

make validate green (two skips, the usual worktree ones).

🤖 Generated with Claude Code

[Copilot]

🟢 Approval recommended

Changes are documentation/comment-only with no rule expression changes, and the two noted nits are purely wording/formatting improvements.

Pull request overview

Updates the HomeLab documentation and Prometheus rule/test commentary to reflect that shiva’s Smart Storage Battery replacement is now fitted and verified, and that the Alertmanager silence referenced in the runbook has been removed.

Changes:

• Update the Smart Storage Battery runbook with post-replacement observations, proof points, and remaining open readings.
• Move issue #76 from “In progress” to “Done” in the roadmap, and update related docs to past tense.
• Refresh Prometheus rule/test comments to reflect that the silence is lifted and that the “quiet” fixture is now a measured post-replacement reading.

File summaries

File	Description
stacks/observability/prometheus/tests/network.test.yaml	Updates test commentary to note the quiet fixture is now measured post-replacement data.
stacks/observability/prometheus/rules/network.rules.yaml	Updates header commentary to record the (now-deleted) silence and current live state of alerts.
docs/runbooks/replace-the-smart-storage-battery.md	Rewrites runbook status and steps with measured “after” readings, proof rows, and an “open” section for non-moving metrics.
docs/roadmap.md	Moves #76 into Done with a dated completion summary and remaining caveats.
docs/observability.md	Updates rules overview text to reflect the battery issue is historical and metrics before 2026-09-02 still show the failed pack.
docs/images/README.md	Updates screenshot narration to past tense and notes the next capture should show the alert quiet.
docs/hardware.md	Updates hardware inventory note to reflect the battery was replaced and calls out the still-open cache ratio observation.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.