Skip to content

IloBatteryCondition has been firing on shiva with no silence and no owner #76

New issue
New issue
Closed
#200
Closed
IloBatteryCondition has been firing on shiva with no silence and no owner#76
#200
Labels
bugSomething isn't working

Description

@Gerrrt
Gerrrt
opened on Aug 21, 2026

IloBatteryCondition has been firing on shiva (10.0.30.10, the HPE iLO BMC
on VLAN 30) continuously, at severity: warning, category: hardware.

Unlike UpsSelfTestFailed — which is silenced until 2026-09-20 with a comment
recording exactly why, and instructions to delete the silence the moment the
pack is fitted — this one has no silence and no tracking issue. It is simply
firing, which over time is how a firing alert stops being read.

On an HPE array controller this is usually the cache/FBWC battery: either
depleted, disconnected, or genuinely failed. If it is the write cache battery
the controller will have dropped back to write-through, which is a real
performance change on the Proxmox host, not just a lamp.

What this needs

One of three outcomes, and the point of the issue is to pick one:

  • Fix it — reseat or replace the cache battery.
  • Accept it — silence it in Alertmanager with an expiry and a comment, the
    way the UPS one is done, and record the residual.
  • Retune it — if this condition is expected on a BMC this old, the rule
    should say so rather than fire forever.

The relevant metric is under cpqDaAccel* in the iLO module; the rule is in
network.rules.yaml.

Found while verifying #12 — the stack is behaving correctly here, the hardware
is not.

Activity

  1. Gerrrt
    added
    bugSomething isn't working
    on Aug 21, 2026
  2. Gerrrt
    added a commit that references this issue on Aug 31, 2026

    Merge pull request #200 from Gerrrt/gerrrt/ilo-battery-condition-shiv…

    a4db4a0
  3. Gerrrt
    closed this as completedin #200on Aug 31, 2026

  4. Gerrrt commented on Aug 31, 2026

    @Gerrrt
    Gerrrt
    on Aug 31, 2026
    OwnerAuthor

    Closed automatically by #200. Recording which of the three outcomes was taken,
    since the issue asked for exactly that.

    Accepted with an expiry — and the rule was retuned, because it was also
    wrong.     Not fixed: the hardware is still broken.

    The alert was a true positive on a permanently failed part. Chassis 0
    battery 1 reads cpqHeSysBatteryStatus 13 (shutdownPermanentFailure) —
    "HP Smart Storage Batt 96", spare 815983-001. 13 is terminal, so the
    issue's "reseat or replace" was only ever replace; there was nothing to try
    before ordering.

    Two corrections to the issue as written:

    • The metric pointer was wrong, and that turned out to be the interesting
      part.       The rule read cpqHeSysBackupBatteryCondition — one identity-free
      scalar — not anything under cpqDaAccel*. Nothing in the repo referenced
      cpqDaAccel* at all, so it fired correctly for twelve days and still could
      not be acted on: which pack, why, and what to order were all in tables no
      rule touched. It now names the part in the alert itself.
    • Write-through is slower, not riskier. It closes the cache-loss window
      rather than opening one, and cpqDaAccelBadData reads 2 (none) — nothing
      dirty was lost when the controller dropped back. The cost is write latency
      and IOPS on the hypervisor, which is why the new rule is warning.

    On a Gen9 this is one fault seen through two MIB subtrees, not two problems:
    the chassis pack backs the Smart Array's flash-backed cache.

    Both alerts were watched from pending into firing against live hardware and
    both reached the default receiver with zero webhook failures before the
    silence went back on, so what is suppressed is known-working rather than
    assumed-working.

    Silence bfdfff66-d9c3-4df4-9495-f1f38ebf93c1, device=shiva, expires
    2026-10-01.     Deliberately narrow, so a genuinely new fault on this box is not
    suppressed with it.

    The residual is the part. Tracked in
    docs/roadmap.md, procedure in
    docs/runbooks/replace-the-smart-storage-battery.md.
    If the pack is not fitted by 2026-10-01 the silence lapses and both alerts come
    back, which is the intent.

  5. Gerrrt
    added 2 commits that reference this issue on Aug 31, 2026

    Merge branch 'main' into gerrrt/backup-verification-scheduling-4008e9

    b0ffa1d

    docs: the installer stick is not in the rack yet, and rewrap #76

    af0a919

  6. Gerrrt commented on Sep 2, 2026

    @Gerrrt
    Gerrrt
    on Sep 2, 2026
    OwnerAuthor

    The residual is no longer the part. Spare 815983-001 fitted 2026-09-02;
    Saruman off 22:30–22:58 UTC.

    First scrape after it came back, 23:02 UTC: chassis 0 battery 1 reads
    cpqHeSysBatteryCondition 2 (ok), cpqHeSysBatteryStatus 1 (noError),
    serial 6EZBN0FB2431YM where the failed pack was 6EZBN0CB29N3YZ — a
    different part being read, not the old one reading differently. The Smart Array
    re-enabled its cache on the same scrape: cpqDaAccelStatus 5 → 3,
    cpqDaAccelBackupPowerSource 1 → 4 (smartbattery), cpqDaAccelBattery
    6 → 2, and the three controller rollups back to 2. cpqDaAccelBadData
    stayed 2 — nothing dirty was lost at either end.

    Silence bfdfff66 deleted at 23:11 UTC rather than left to expire on
    2026-10-01; both alerts are live again over the new part and nothing is firing.

    Two readings did not move. cpqDaAccelWriteCachePercent /
    ReadCachePercent still 0 / 0 ten minutes in, where a P440ar normally
    reports a ratio, and cpqDaAccelFailedBatteries still 1. Neither is alerted
    on. The runbook now names both, with the ssacli check on Saruman that
    resolves the first. Documents flipped in the PR that references this issue.

  7. Gerrrt
    added a commit that references this issue on Sep 2, 2026

    Merge pull request #243 from Gerrrt/gerrrt/hpe-smart-battery-replacem…

    0e4b60e
  8. Gerrrt
    added a commit that references this issue on Sep 18, 2026

    docs(runbooks): the Saruman SSD fit becomes a procedure, and the tray…

    b612b97

  9. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    Gerrrt
    on Sep 20, 2026
    OwnerAuthor

    The ssacli reading, 2026-09-19 22:52 UTC, seventeen days on: branch (a), a blind spot in the iLO, not a fault in the array.

    Cache Ratio: 10% Read / 90% Write
Total Cache Size: 2.0
Total Cache Memory Available: 1.8
Battery Backed Cache Size: 1.8
No-Battery Write Cache: Disabled
Drive Write Cache: Disabled
Battery/Capacitor Count: 1
Battery/Capacitor Status: OK
logicaldrive 1 ... Caching:  Enabled   LD Acceleration Method: Controller Cache

    While cpqDaAccelWriteCachePercent, ReadCachePercent, Memory and ReadMemory all still read 0, and cpqDaLogDrvHasAccel reads other on the cached logical drive. The iLO does not populate those columns on this hardware; the ratio was set and the cache was absorbing writes all along. The fit's fio confirmed it from the other side: the 7.2K mirror completes 4 KiB random writes in 1.3 ms on average at queue depth 1 (741 IOPS), a fifth of them in 50 µs.

    So: no modify cacheratio= was ever needed, no rule may be written on those four columns, and IloWriteCacheDisabled reading cpqDaAccelStatus was right. cpqDaAccelFailedBatteries 1 is still unexplained and still unalerted. The roadmap's Done entry for this issue carries a dated paragraph; the reading and the reasoning are in #527 and #556. Stays closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions