From 4fa0045477583d19a1beb1b583152c039baa950e Mon Sep 17 00:00:00 2001 From: rldyourmnd Date: Wed, 2 Sep 2026 17:39:37 +0500 Subject: [PATCH] fix(alerts): the slow-burn ticket can reach its own threshold queue_wait_slow_burn asked for a queued wait above 600 s held for fifteen minutes. Measured against a full day of the metric -- 2026-09-02, with one project's dispatched release CI bursting the priority pools and jobs waiting up to 906 s -- that would have fired zero times on every scale set. The queued wait does not stay above ten minutes for a quarter of an hour: the provider assigns the intent first and the rest of the wait belongs to lifecycle_assigned_stall. The rule also contradicted its own summary, which places it "between the two-minute objective and the five-minute page". A ticket must hold at least ten minutes -- the package's own policy for the class -- so the threshold comes down to where a ten-minute hold can reach: 300 s, the five-minute mark the summary names. On that same day it fires once. The measured grid is in the rule's comment so the next person can re-derive it rather than trust it. Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp --- config/observability-rules.yaml | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/config/observability-rules.yaml b/config/observability-rules.yaml index 20567c3..e9a0994 100644 --- a/config/observability-rules.yaml +++ b/config/observability-rules.yaml @@ -581,6 +581,25 @@ rules: summary: A durable provider-create circuit is blocking an entire scale set. action: Preserve the retry journal, fix and verify the provider cause, then use exact CAS recovery while GARM is stopped. recovery: Terminal circuit count remains zero and a fresh job reaches provider create and runner online. + # Measured against a full day of the real metric (2026-09-02, a day with one + # project's dispatched release CI bursting the priority pools): the rule as + # written -- above 600 s held for fifteen minutes -- would have fired zero + # times on every scale set, while jobs waited up to 906 s. The queued wait + # simply does not stay above ten minutes for a quarter of an hour; the + # provider assigns the intent first, and the rest of the wait belongs to + # lifecycle_assigned_stall. A ticket that cannot fire is not a quiet fleet, + # it is an unwatched one. + # + # Episodes in that same day, by (threshold, minutes held): (300, 5) = 18, + # (300, 10) = 1, (420, 5) = 4, (450, 5) = 3, (540, 5) = 2, (600, 5) = 0, + # and zero for anything held the fifteen minutes this rule asked for. + # + # A ticket must hold at least ten minutes -- that is the rule package's own + # policy for the class, and it is right -- so the threshold has to come down + # to where a ten-minute hold can reach. 300 s is also the boundary this + # rule's own summary names: "between the two-minute objective and the + # five-minute page". Held ten minutes it fires once on a day like that one, + # which is what a slow-burn ticket should cost. - id: queue_wait_slow_burn severity: ticket query_language: promql @@ -602,9 +621,9 @@ rules: # always receives a matrix. expression: max by (scale_set) (gha_fleet_queue_oldest_queued_wait_seconds_by_scale_set) * (max by (scale_set) (gha_fleet_queue_oldest_queued_wait_seconds_by_scale_set) < bool 1800) operator: ">" - threshold: 600 + threshold: 300 evaluation_seconds: 300 - hold_seconds: 900 + hold_seconds: 600 destination_ref: fleet_oncall enabled: true owner: fleet-performance