fix(console): choose the statistic by the question — mean for capacity, median for typical, p95 for the tail - #114
fix(console): choose the statistic by the question — mean for capacity, median for typical, p95 for the tail#114harper-joseph wants to merge 1 commit into
Conversation
… v0.6.0 You are right that p95 is the wrong default, and none of this is an SLA. The payload carries mean, median and p95; they answer different questions, and the console was using one of them nearly everywhere. CAPACITY IS A MEAN. The queue view tiled and charted render p95 directly under a note saying "renders/hour = concurrency ÷ time". Throughput follows the average service time, not the tail, so that number argued away a third of the fleet: measured on prod just now, p95 16.0s against a mean of 11.0s. The tile is the mean and says what it is for; the p95 rides in the subtitle, where a widening gap between the two is the tell that some renders are pathological rather than the whole fleet being slower. The mean is also the only statistic that merges exactly across combos, buckets and nodes — a merged median or p95 is a count-weighted average of percentiles, which is not the percentile of the pooled population. SERVE TIME WAS TWO POPULATIONS IN ONE NUMBER, which is worse than the wrong percentile. Cache hits run 2ms and origin proxies 371ms; pooled at the current 54/46 split that is "462ms p95", a figure that improves when offload improves and says nothing about how fast either path is. The tile now reports the cache-hit median with the origin-served median beside it. STALENESS IS JUDGED ON THE MEDIAN. A page's age walks from zero to its interval and is re-rendered, so an evenly refreshed corpus sits at median 0.50x with its p95 already at ~0.95x — a p95 threshold has no headroom and flags a healthy fleet as behind. The median crossing 1.0 means something unambiguous instead: half the cache serves were past due. Live numbers on the whole corpus: median 0.35x, p95 0.87x. The per-route table now uses the same statistic as the tile, which it did not: 0.34x on the tile and 0.89x in the table for the same route, with nothing on screen to reconcile them. And the ratio now defers to the verdicts that outrank it. hit/swr/stale are decided per request against each page's OWN expiry; this panel can only divide by the cadence the ROUTE configures, and those differ whenever a target's interval comes from its stored sitemap changefreq — which no metric exposes. When the ratio says "behind" and the verdicts say nothing was past due, the divisor is what is wrong, and the panel says so instead of letting a config gap read as a fleet failure. Silent on this deployment, where every prerender route sets a renderInterval. Origin cost per verdict leads with the median (its p95 is in the tooltip): "what does a miss cost" is a typical-case question. Verified by rendering every changed panel against a real 6h window pulled from a prod node, not only against fixtures. The test fixture now carries a median distinct from its p95, because with a flat distribution no test could tell which statistic a panel had used — which is the entire subject of this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request updates the prerender management console to use more appropriate statistical metrics (mean, median, and p95) across various panels. Specifically, it switches to using the median for page age, staleness, and serve times, and the mean for render capacity, while keeping p95 to represent tail latency. It also adds a warning note when freshness verdicts contradict the calculated staleness ratio, updates the corresponding tests, and improves documentation. A review comment suggests ensuring that ageTailP95 is explicitly validated as a finite number before division to prevent incorrect formatting when the value is null or undefined.
| ? el('span', { | ||
| cls: ratio > 1 ? 'pill warn' : 'mono', | ||
| text: fmtRatio(ratio), | ||
| title: `p95 ${fmtRatio(cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN)}`, | ||
| }) |
There was a problem hiding this comment.
If ageTailP95 is null or undefined (which can happen if weighted returns null), the expression ageTailP95 / cadence.interval will evaluate to 0 in JavaScript (since null / number === 0). This causes fmtRatio to format it as "0.00×" instead of showing a missing value ("—").
To prevent this, we should explicitly check if ageTailP95 is a finite number before performing the division. Since metrics or database values can sometimes be surfaced as BigInt/Long, always coerce the value to a Number first before calling Number.isFinite to avoid false negatives.
| ? el('span', { | |
| cls: ratio > 1 ? 'pill warn' : 'mono', | |
| text: fmtRatio(ratio), | |
| title: `p95 ${fmtRatio(cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN)}`, | |
| }) | |
| ? el('span', { | |
| cls: ratio > 1 ? 'pill warn' : 'mono', | |
| text: fmtRatio(ratio), | |
| title: 'p95 ' + fmtRatio(Number.isFinite(Number(ageTailP95)) && cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN), | |
| }) |
References
- When validating database column values (which may be surfaced as BigInt/Long) using Number.isFinite, always coerce the value to a Number first, as Number.isFinite returns false for BigInts.
Right on both counts, and checking it against a live 6h window turned up more than the question
asked.
hdb_analyticscarries mean, median and p95; they answer different questions, and thisconsole was reaching for p95 nearly everywhere.
What the live numbers said
The rule this settles on
time. It is also the only statistic that merges exactly across combos, buckets and nodes; a
merged median or p95 is a count-weighted average of percentiles, which is not the percentile of
the pooled population.
healthy middle: the blob-read incident ran a cohort of cache hits at 13.6s while the median
stayed at 2.3ms. Nothing here is an SLA.
Changes
Capacity was computed from a tail. The queue view tiled and charted render p95 directly
under a note reading "renders/hour = concurrency ÷ time". On today's numbers that argues away a
third of the fleet (16.0s vs a mean of 11.0s). The tile is now the mean and says what it is for;
the p95 sits in the subtitle, where a widening gap between the two is the tell that some renders
are pathological rather than the fleet being uniformly slower.
Serve time was two populations in one number — worse than the wrong percentile. 2ms cache hits
and 371ms origin proxies pooled at the current 54/46 split give "462ms", a figure that improves
when offload improves and describes neither path. Now: cache-hit median, with the origin-served
median beside it.
Staleness is judged on the median. A page's age walks from zero to its interval and is
re-rendered, so an evenly refreshed corpus sits at median 0.50× with its p95 already at ~0.95×
— a p95 threshold has no headroom and calls a healthy fleet behind. The median crossing 1.0 is
unambiguous: half the cache serves were past due. The panel now states those baselines, so the
number is interpretable without doing the reasoning.
The per-route table used a different statistic than the tile above it — 0.34× on the tile and
0.89× in the table for the same route, with nothing on screen to reconcile them. Both are the
median now; each row's p95 is in its tooltip.
The ratio now defers to the verdicts that outrank it.
hit/swr/staleare decided perrequest against each page's own expiry. This panel can only divide by the cadence the route
configures, and those differ whenever a target's interval comes from its stored sitemap
changefreq— which no metric exposes. When the ratio says "behind" while the verdicts saynothing was past due, the divisor is what is wrong, and the panel says so rather than letting a
config gap read as a fleet failure. Silent on this deployment, where every prerender route sets a
renderInterval.Origin cost per verdict leads with the median; its p95 moved to the tooltip.
Verification
Rendered every changed panel against a real 6h window pulled from a prod node, not only
fixtures:
Console suite 115 → 121. The fixture now carries a median distinct from its p95: with a flat
distribution no test could tell which statistic a panel had used, which is the whole subject here.
Lint + format clean, plugin suite 715 unaffected. Console only.
Follow-up worth its own PR (plugin side)
Charts can only draw mean and p95 —
bucketize()builds per-bucketmeansandp95sarraysbut keeps
medianas a scalar, so a "typical" trend line is impossible today and the tiles carrythe median alone. Adding
mediansto the per-bucket arrays is a small change inutil/analyticsRead.jsplus the console's merge, but it needs a plugin release and aprerender-cluster deploy.