fix(alerts): the signal-event rules count the host, not the data point - #377
Merged
rldyourmnd merged 1 commit intoSep 2, 2026
Merged
Conversation
The OTel signal-event streams carry a start_time label that changes with each counter run, so an expression that does not aggregate returns one series per run rather than one per host. Three rules read those streams and none aggregated. Measured on 2026-09-02: sixteen failed alert evaluations in two minutes reached the channel as three series of 4, 4 and 3 -- three identical pages for one incident, re-sent every eleven minutes for the hour the window remembers. The same arithmetic silently under-counts: three failures spread across three runs sum to one each and never cross the threshold, and in audit_suppression_burst and kernel_workqueue_hog a counter that restarts hides half its increase. All three now sum by (host_name), which is also the label the notification prints as its subject. A walking test refuses a signal-event rule that does not aggregate, and the sustained-delta test asserts the property rather than the literal prefix it used to match. Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp
rldyourmnd
enabled auto-merge
September 2, 2026 15:01
rldyourmnd
deleted the
fix/signal-alerts-count-the-host-not-the-data-point
branch
September 2, 2026 15:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The OTel signal-event streams carry a
start_timelabel that changes with each counter run, so an expression that does not aggregate returns one series per run rather than one per host. Three rules read those streams and none aggregated:alert_evaluation_failed,audit_suppression_burst,kernel_workqueue_hog.Measured on 2026-09-02: sixteen failed alert evaluations in two minutes reached the channel as three series of 4, 4 and 3 — three identical pages for one incident, re-sent every eleven minutes for the hour the window remembers. The same arithmetic under-counts in the other direction: three failures spread across three runs sum to one each and never cross the threshold, and in the two cumulative rules a counter that restarts hides half its increase.
All three now
sum by (host_name), which is also the label the notification prints as its subject, so the count is the host's and each host produces one message. A walking test refuses any signal-event rule that does not aggregate; the sustained-delta test now asserts the property (no outer subquery, noor, no scalar collapse, delta summed by host) instead of matching a literal prefix.Audited the same way against a live query: every other rule already groups by
host_name,scale_setorerror_class.https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp