feat(alerts): health that flaps below the page's window is visible - #374
Merged
Conversation
fleet_platform_unhealthy pages when health is down for two minutes, and it reads `max` over that window. A reconciler that fails and is restarted by its timer takes health down for one minute at a time, so the healthy minute beside each failed one kept the page silent: on 2026-09-02 the warm pool reconcilers failed eleven times between 09:35Z and 11:00Z and no rule fired. fleet_health_flapping counts the unhealthy minutes over half an hour and raises a ticket at three. A fleet that recovers by itself every minute is not an outage; it is a defect with a heartbeat, and it now has a name. Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fleet_platform_unhealthypages when health is down for two minutes and readsmax by (host_name)over that window, so a one-minute dip beside a healthy minute never fires it. On 2026-09-02 the warm-pool reconcilers failed eleven times between 09:35Z and 11:00Z (each failure taking services-host health down for a minute) and no rule fired at all; I found them by querying the metric by hand.fleet_health_flappingcounts the unhealthy minutes over half an hour and raises a ticket at three. A fleet that recovers by itself every minute is not an outage, it is a defect with a heartbeat. The cause of those eleven dips is fixed separately in #373; this is the rule that would have shown them.No provider-compiled package changes.
https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp