Repository navigation
docs(observability): record the silence actually in force on shiva - #201
Merged
Merged
Conversation
The first silence was deleted so both alerts could be watched notifying for real, which means the UUID written into the rule header, the roadmap entry and the runbook pointed at an expired silence. Replace it with the one now in force, bfdfff66-d9c3-4df4-9495-f1f38ebf93c1 — same matchers, same 2026-10-01 expiry, so the decision is unchanged and only the identifier moves. Record what the deletion bought while it was off. Both alerts were watched through pending into firing and both reached the default receiver with zero webhook failures: IloBatteryCondition at 14:07Z carrying cpqHeSysBatterySparePartNum=815983-001 and $value 13, IloWriteCacheDisabled at 14:37Z at 5. IloHardwareDegraded stayed inactive throughout with cpqDaCntlrCondition reading 3, which is the > 3 threshold doing its job against live hardware rather than a fixture. What is suppressed is now known-working rather than assumed-working, and that is the part worth having in the record — a silence over an unproven rule is the failure mode this repository keeps re-learning. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #200, which merged at
13:53:59Zwhile the verification below wasstill running.
Why this is needed
#200 recorded silence
3ffc313f-7154-459c-9f62-7c9a432bc97ein three places.That silence was deleted at
13:43:50Z— ten minutes before #200 merged — soboth alerts could be watched notifying for real.
maintherefore currentlypoints at an expired silence in the rule header, the roadmap entry and the
runbook. This replaces it with the one now in force,
bfdfff66-d9c3-4df4-9495-f1f38ebf93c1: same matchers, same 2026-10-01expiry, so the accept-with-expiry decision is unchanged and only the identifier
moves.
What the deletion bought
Both alerts were watched from
pendingintofiringagainst live hardware, andboth reached the
defaultreceiver with zero webhook failures:IloBatteryCondition13cpqHeSysBatterySparePartNum=815983-001on the alert,receivers=['default'],silencedBy=[]IloWriteCacheDisabled5receivers=['default'],silencedBy=[]IloHardwareDegradedstayedinactivethroughout withcpqDaCntlrConditionreading
3— the> 3threshold doing its job against real hardware ratherthan a fixture, which is the one thing the unit tests could not prove.
All three rules held
health=ok, so thegroup_leftjoin matches at evaluationtime and does not error.
Incidentally
AlertmanagerNotificationsFailingcleared toinactiveon its ownduring this — it had been firing off pre-restart failures aging out of its
window, and nothing has failed since.
The silence is back on and both alerts now read
state=suppressed,silencedBy=['bfdfff66-d9c3-4df4-9495-f1f38ebf93c1'].Verification
./scripts/validate.shpasses end to end. No reference to the expired3ffc313f…survives anywhere in the tree.🤖 Generated with Claude Code