Skip to content

feat(alerts): add per-group acknowledgments for grouped alerts #2696

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oddegen wants to merge 3 commits into from
+3,397 −271
Closed

feat(alerts): add per-group acknowledgments for grouped alerts #2696

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0174673
Feat/per group alert ack (#2)
oddegen Jul 21, 2026
b5f5efa
feat: refine grouped alert empty state
oddegen Jul 22, 2026
0499409
Merge branch 'main' into main
jordan-simonovski Jul 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changeset/quiet-alert-groups.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
'@hyperdx/api': patch
'@hyperdx/app': patch

@jordan-simonovski jordan-simonovski Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Given this is a new feature being added to the app, I'd say this is a minor change, not a patch bump. See: https://semver.org/

---

Add per-group alert acknowledgments and group resume overrides so grouped alerts can mute or resume notifications for one group without changing other groups or the aggregate alert state.
28 changes: 28 additions & 0 deletions packages/api/openapi.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,26 @@
}
}
},
"AlertSilencedGroup": {
"allOf": [
{
"$ref": "#/components/schemas/AlertSilenced"
},
{
"type": "object",
"required": [
"group"
],
"properties": {
"group": {
"type": "string",
"description": "Exact group key being silenced.",
"example": "service:api"
}
}
}
]
},
"AlertChannelWebhook": {
"type": "object",
"required": [
Expand Down Expand Up @@ -331,6 +351,14 @@
"description": "Silencing metadata.",
"nullable": true
},
"silencedGroups": {
"type": "array",
"nullable": true,
"description": "Stored per-group silences, if any.",
"items": {
"$ref": "#/components/schemas/AlertSilencedGroup"
}
},
"executionErrors": {
"type": "array",
"nullable": true,
Expand Down
93 changes: 93 additions & 0 deletions packages/api/src/controllers/alertHistory.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ type GroupedAlertHistory = {
lastValues: IAlertHistory['lastValues'][];
};

type AlertGroupHistorySummary = {
group: string;
state: AlertState;
history: Omit<IAlertHistory, 'alert' | 'group'>[];
};

function groupStateToOverallState(states: string[]): AlertState {
if (states.includes(AlertState.ALERT)) {
return AlertState.ALERT;
Expand All @@ -44,6 +50,17 @@ function mapGroupedHistories(
}));
}

function mapHistoryEntry(
history: Pick<IAlertHistory, 'createdAt' | 'counts' | 'lastValues' | 'state'>,
): Omit<IAlertHistory, 'alert' | 'group'> {
return {
createdAt: history.createdAt,
counts: history.counts,
lastValues: history.lastValues,
state: history.state,
};
}

/**
* Gets the most recent alert histories for a given alert ID,
* limiting to the given number of entries.
Expand Down Expand Up @@ -131,6 +148,82 @@ export async function getRecentAlertHistoriesBatch(
);
}

export async function getRecentAlertGroupSummaries({
alertId,
interval,
limit,
}: {
alertId: ObjectId;
interval: AlertInterval;
limit: number;
}): Promise<AlertGroupHistorySummary[]> {
const lookbackMs = limit * ALERT_INTERVAL_TO_MINUTES[interval] * 60 * 1000;

const histories = await AlertHistory.find({

@jordan-simonovski jordan-simonovski Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): this can make the alerts page unusable for an entire team, and it gets worse silently as their telemetry grows.

Why this blocks. GET /api/alerts is the alerts page — the first thing someone opens during an incident. This query now runs once per alert with no ceiling on rows returned, and nothing upstream caps how many distinct groups exist: checkAlerts never sets seriesLimit on alert chart configs, so group cardinality is whatever the telemetry happens to contain. One grouped alert on a naturally high-cardinality dimension (pod name, user ID, trace ID) can have thousands of distinct groups in the lookback window, each carrying up to 20 history entries with their own lastValues arrays.

All of that is pulled into the API process heap and serialised into a single response. So the failure isn't "the page is a bit slow" — it's heap pressure on a shared API process and a response large enough to make the alerts page unusable for everyone on the team, at exactly the moment they need it. It also degrades silently and gradually: this will pass review, pass staging, and fall over first in your largest production tenant, where the diagnosis is hardest.

What's happening.

const histories = await AlertHistory.find({
  alert: new ObjectId(alertId),
  createdAt: { $gte: new Date(Date.now() - lookbackMs) },
  group: { $type: 'string' },
}).sort({ group: 1, createdAt: -1 }).lean();

No .limit(), no cap on distinct groups. The {alert, group, createdAt} index makes this cheap to plan — it does nothing to bound the result set. The per-group limit is applied in JS after every document is already in memory. Note the sibling getRecentAlertHistories directly above uses an aggregation for exactly this reason.

Fix. Mirror the sibling with $group/$slice, or cap both dimensions:

const MAX_ALERT_GROUP_SUMMARIES = 50;
const histories = await AlertHistory.find({ ... })
  .sort({ group: 1, createdAt: -1 })
  .limit(MAX_ALERT_GROUP_SUMMARIES * limit)   // backstop; group:1 sort truncates by group
  .lean();
// ...
return Array.from(groupedHistories.values())
  .sort(...)
  .slice(0, MAX_ALERT_GROUP_SUMMARIES);

Prefer an ordering that keeps ALERT-state groups when the cap bites, and return a truncated flag so the UI can say so. Also worth considering: drop groups from the list endpoint entirely and populate it only on GET /alerts/:id.

alert: new ObjectId(alertId),
createdAt: { $gte: new Date(Date.now() - lookbackMs) },
group: { $type: 'string' },
})
.sort({ group: 1, createdAt: -1 })
.lean();

const groupedHistories = new Map<string, AlertGroupHistorySummary>();

for (const history of histories) {
const group = history.group;

if (group == null) {
continue;
}

const existing = groupedHistories.get(group);

if (existing == null) {
groupedHistories.set(group, {
group,
state: history.state,
history: [mapHistoryEntry(history)],
});
continue;
}

if (existing.history.length < limit) {
existing.history.push(mapHistoryEntry(history));
}
}

return Array.from(groupedHistories.values()).sort(
(a, b) =>
b.history[0]!.createdAt.getTime() - a.history[0]!.createdAt.getTime(),
);
}

export async function getRecentAlertGroupSummariesBatch(
alerts: { alertId: ObjectId; interval: AlertInterval }[],
limit: number,
): Promise<Map<string, AlertGroupHistorySummary[]>> {
const queue = new PQueue({ concurrency: ALERT_HISTORY_QUERY_CONCURRENCY });

const entries = await Promise.all(
alerts.map(({ alertId, interval }) =>
queue.add(async () => {
const summaries = await getRecentAlertGroupSummaries({
alertId,
interval,
limit,
});
return [alertId.toString(), summaries] as const;
}),
),
);

return new Map(
entries.filter(
(e): e is [string, AlertGroupHistorySummary[]] => e !== undefined,
),
);
}

/**
* Returns alert firing/recovery transitions (ALERT-boundary crossings) within
* [startTime, endTime] for one alert, for drawing chart annotations. One window
Expand Down
38 changes: 36 additions & 2 deletions packages/api/src/controllers/alerts.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,24 @@ export const getAlertsEnhanced = async (teamId: ObjectId) => {
silenced?: IAlert['silenced'] & {
by: IUser;
};
}>(['savedSearch', 'dashboard', 'createdBy', 'silenced.by']);
silencedGroups?: Array<
Omit<NonNullable<IAlert['silencedGroups']>[number], 'by'> & {
by?: IUser;
}
>;
unsilencedGroups?: Array<
Omit<NonNullable<IAlert['unsilencedGroups']>[number], 'by'> & {
by?: IUser;
}
>;
}>([
'savedSearch',
'dashboard',
'createdBy',
'silenced.by',
'silencedGroups.by',
'unsilencedGroups.by',
]);
};

export const getAlertEnhanced = async (
Expand All @@ -312,7 +329,24 @@ export const getAlertEnhanced = async (
silenced?: IAlert['silenced'] & {
by: IUser;
};
}>(['savedSearch', 'dashboard', 'createdBy', 'silenced.by']);
silencedGroups?: Array<
Omit<NonNullable<IAlert['silencedGroups']>[number], 'by'> & {
by?: IUser;
}
>;
unsilencedGroups?: Array<
Omit<NonNullable<IAlert['unsilencedGroups']>[number], 'by'> & {
by?: IUser;
}
>;
}>([
'savedSearch',
'dashboard',
'createdBy',
'silenced.by',
'silencedGroups.by',
'unsilencedGroups.by',
]);
};

export const deleteAlert = async (id: string, teamId: ObjectId) => {
Expand Down
77 changes: 72 additions & 5 deletions packages/api/src/models/alert.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,23 @@ export interface IAlertError {
message: string;
}

interface IAlertSilence {
by?: ObjectId;
at: Date;
until: Date;
}

export interface IAlertSilencedGroup extends IAlertSilence {
group: string;
}

export interface IAlertUnsilencedGroup {
group: string;
by?: ObjectId;
at: Date;
parentSilencedAt: Date;
}

// follow 'ms' pkg formats
export type AlertInterval =
| '1m'
Expand Down Expand Up @@ -79,11 +96,9 @@ export interface IAlert {
tileId?: string;

// Silenced
silenced?: {
by?: ObjectId;
at: Date;
until: Date;
};
silenced?: IAlertSilence;
silencedGroups?: IAlertSilencedGroup[];
unsilencedGroups?: IAlertUnsilencedGroup[];

// Multi-window alerting: fire only after N violations in M consecutive windows
numConsecutiveWindows?: number | null;
Expand Down Expand Up @@ -218,6 +233,58 @@ const AlertSchema = new Schema<IAlert>(
required: false,
},
},
silencedGroups: {
type: [
{
_id: false,
group: {
type: String,
required: true,
},
by: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
required: false,
},
at: {
type: Date,
required: true,
},
until: {
type: Date,
required: true,
},
},
],
required: false,
default: undefined,
},
unsilencedGroups: {
type: [
{
_id: false,
group: {
type: String,
required: true,
},
by: {
type: mongoose.Schema.Types.ObjectId,
ref: 'User',
required: false,
},
at: {
type: Date,
required: true,
},
parentSilencedAt: {
type: Date,
required: true,
},
},
],
required: false,
default: undefined,
},
executionErrors: {
type: [
{
Expand Down
Loading
Loading