Alert notification sample events ignore saved-search filters — alerts show unrelated log lines as evidence #3059

What happened?

A saved search scoped via the filter bar (the filters array) instead of the where clause evaluates its alerts correctly — the threshold count respects the filters — but the sample events rendered into the notification body are queried without the filters. The alert fires saying "1 lines found" and then shows five arbitrary, unrelated log lines from the time window as evidence (in our case: Kubernetes reloader/info noise with empty service fields, while the actually-matched event was a service error).

Expected: the sample lines come from the same row set the alert counted — the sample query should apply savedSearch.filters the same way evaluation does.

Cause: the two paths build their queries differently. Alert evaluation
(packages/api/src/tasks/checkAlerts/index.ts, getChartConfigFromAlert)
passes filters: savedSearch.filters into the shared buildSearchChartConfig
— per its own comment, so all paths assemble configs "identically". The sample
fetch (packages/api/src/tasks/checkAlerts/template.ts) hand-builds its
chartConfig from savedSearch.where/whereLanguage only; filters is never
referenced in that file. Routing the sample fetch through the same shared
builder would fix it.

Steps to reproduce

1. Create a saved search on a log source with an empty where, scoped entirely via filter-bar conditions, e.g. ServiceName = 'my-service', ResourceAttributes['deployment.environment.name'] = 'production', SeverityText IN ('error','fatal')
2. Attach an alert (at least 1 line within 5 minutes) with a generic or incident.io webhook channel
3. Wait for one matching event; the alert fires with the correct count ("1 lines found")
4. Inspect the delivered webhook body: the code fence contains up to 5 unfiltered lines from the window instead of the matched event

How are you running HyperDX?

ClickStack on ClickHouse Cloud (managed, current version as of Sept 2026 — ~2.37 era).

Where does it show up?

In the delivered alert notification body (webhook payload description / incident.io alert description) — both generic and incident.io webhook types. The search UI itself displays correct results for the same conditions.

Logs

Sample from a real firing (redacted): alert scoped to ServiceName = 'customer-idp' AND ResourceAttributes['deployment.environment.name'] = 'production' AND SeverityText IN ('error','fatal'); exactly 1 matching error existed in the window (verified by querying ClickHouse directly), yet the body contained:

"2026-09-02T11:18:39.000000000Z","","warn","","","Back-off restarting failed container reloader-reloader in pod reloader-…"
"2026-09-02T11:18:39.105628798Z","","info","","","[WARNING] No files matching import glob pattern: custom/*.override"

Note the empty service fields — none of these rows match the search's filters. Moving the same conditions into the where clause makes the samples correct, which is our current workaround.