feat(observer): the fleet measures the wait it is judged by - #379
Merged
Merged
Conversation
Every wait metric the fleet exported stopped when the provider assigned the intent: gha_fleet_queue_oldest_queued_wait_seconds counts only intents still queued, and the rest of the wait -- assignment, boot, runner registration -- belonged to a different gauge with a different threshold. On 2026-09-02 that gauge peaked at 597 s on three scale sets while GitHub's own job records showed waits up to 906 s, so no rule could see the number the fleet is actually judged by. The observer now closes the measurement: for every intent that entered running within the last fifteen minutes it takes the whole wait, from the immutable first-queued stamp to the moment the runner reported running, and exports the sample count, median, ninetieth percentile and maximum, plus the percentile per scale set. queue_started_wait_slow_burn tickets when that percentile stays above the five-minute objective for ten minutes, and its action points at the two phase rules so the reader knows which half grew. Claude-Session: https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp
rldyourmnd
enabled auto-merge
September 2, 2026 16:32
rldyourmnd
deleted the
feat/the-fleet-measures-the-wait-it-is-judged-by
branch
September 2, 2026 16:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every wait metric the fleet exported stopped when the provider assigned the intent.
gha_fleet_queue_oldest_queued_wait_secondscounts only intents still queued; assignment, boot and runner registration belong to a different gauge with a different threshold. On 2026-09-02 that gauge peaked at 597 s on three scale sets while GitHub's own job records showed waits up to 906 s — no rule could see the number the fleet is judged by, andqueue_wait_slow_burncould not fire because the wait it watches ends early.The observer now closes the measurement. For every intent that entered
runningwithin the last fifteen minutes it takes the whole wait — from the immutable first-queued stamp to the moment the runner reported running — and exports:gha_fleet_queue_started_wait_samplesgha_fleet_queue_started_wait_median_secondsgha_fleet_queue_started_wait_p90_secondsgha_fleet_queue_started_wait_max_secondsgha_fleet_queue_started_wait_p90_seconds_by_scale_setqueue_started_wait_slow_burntickets when that percentile stays above the five-minute objective for ten minutes, and its action points atqueue_wait_slow_burnandlifecycle_assigned_stallso the reader knows which half of the wait grew before anyone adds capacity.An empty window reads zero: no jobs started is not a long wait. The still-queued gauge is untouched, and a test pins both behaviours together.
https://claude.ai/code/session_0128syXKxAGCfJGRDxUUNQXp