Skip to content

fix(alerts): the signal-event rules count the host, not the data point - #377

Merged
rldyourmnd merged 1 commit into
mainfrom
fix/signal-alerts-count-the-host-not-the-data-point
Sep 2, 2026
Merged

fix(alerts): the signal-event rules count the host, not the data point#377
rldyourmnd merged 1 commit into
mainfrom
fix/signal-alerts-count-the-host-not-the-data-point

Conversation

@rldyourmnd

Copy link
Copy Markdown
Contributor

The OTel signal-event streams carry a start_time label that changes with each counter run, so an expression that does not aggregate returns one series per run rather than one per host. Three rules read those streams and none aggregated: alert_evaluation_failed, audit_suppression_burst, kernel_workqueue_hog.

Measured on 2026-09-02: sixteen failed alert evaluations in two minutes reached the channel as three series of 4, 4 and 3 — three identical pages for one incident, re-sent every eleven minutes for the hour the window remembers. The same arithmetic under-counts in the other direction: three failures spread across three runs sum to one each and never cross the threshold, and in the two cumulative rules a counter that restarts hides half its increase.

All three now sum by (host_name), which is also the label the notification prints as its subject, so the count is the host's and each host produces one message. A walking test refuses any signal-event rule that does not aggregate; the sustained-delta test now asserts the property (no outer subquery, no or, no scalar collapse, delta summed by host) instead of matching a literal prefix.

Audited the same way against a live query: every other rule already groups by host_name, scale_set or error_class.

https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp

The OTel signal-event streams carry a start_time label that changes with
each counter run, so an expression that does not aggregate returns one
series per run rather than one per host. Three rules read those streams and
none aggregated.

Measured on 2026-09-02: sixteen failed alert evaluations in two minutes
reached the channel as three series of 4, 4 and 3 -- three identical pages
for one incident, re-sent every eleven minutes for the hour the window
remembers. The same arithmetic silently under-counts: three failures spread
across three runs sum to one each and never cross the threshold, and in
audit_suppression_burst and kernel_workqueue_hog a counter that restarts
hides half its increase.

All three now sum by (host_name), which is also the label the notification
prints as its subject. A walking test refuses a signal-event rule that does
not aggregate, and the sustained-delta test asserts the property rather than
the literal prefix it used to match.

Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp
@rldyourmnd
rldyourmnd enabled auto-merge September 2, 2026 15:01
@rldyourmnd
rldyourmnd merged commit a43040d into main Sep 2, 2026
10 checks passed
@rldyourmnd
rldyourmnd deleted the fix/signal-alerts-count-the-host-not-the-data-point branch September 2, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant