Skip to content

fix(console): choose the statistic by the question — mean for capacity, median for typical, p95 for the tail - #114

Open
harper-joseph wants to merge 1 commit into
mainfrom
fix/statistic-per-panel
Open

fix(console): choose the statistic by the question — mean for capacity, median for typical, p95 for the tail#114
harper-joseph wants to merge 1 commit into
mainfrom
fix/statistic-per-panel

Conversation

@harper-joseph

Copy link
Copy Markdown
Contributor

"I'm guessing p95 is too high for something like response time? It's not an SLA right?"

Right on both counts, and checking it against a live 6h window turned up more than the question
asked. hdb_analytics carries mean, median and p95; they answer different questions, and this
console was reaching for p95 nearly everywhere.

What the live numbers said

population median p95 mean
cache-hit serves 1.8ms 2.7ms 1.8ms
origin-proxied serves 371ms 1006ms 468ms
all bot serves — what the tile showed 171ms 462ms 215ms
render time 10.6s 16.0s 11.0s
page age ÷ route cadence 0.34× 0.87×

The rule this settles on

  • mean — capacity, and anything that must add up. Throughput is concurrency ÷ mean service
    time. It is also the only statistic that merges exactly across combos, buckets and nodes; a
    merged median or p95 is a count-weighted average of percentiles, which is not the percentile of
    the pooled population.
  • median — "what does a crawler normally get": serve time, origin cost, served age.
  • p95 — the tail, and only the tail. It earns its place where a pathology hides behind a
    healthy middle: the blob-read incident ran a cohort of cache hits at 13.6s while the median
    stayed at 2.3ms. Nothing here is an SLA.

Changes

Capacity was computed from a tail. The queue view tiled and charted render p95 directly
under a note reading "renders/hour = concurrency ÷ time". On today's numbers that argues away a
third of the fleet (16.0s vs a mean of 11.0s). The tile is now the mean and says what it is for;
the p95 sits in the subtitle, where a widening gap between the two is the tell that some renders
are pathological rather than the fleet being uniformly slower.

Serve time was two populations in one number — worse than the wrong percentile. 2ms cache hits
and 371ms origin proxies pooled at the current 54/46 split give "462ms", a figure that improves
when offload improves
and describes neither path. Now: cache-hit median, with the origin-served
median beside it.

Staleness is judged on the median. A page's age walks from zero to its interval and is
re-rendered, so an evenly refreshed corpus sits at median 0.50× with its p95 already at ~0.95×
— a p95 threshold has no headroom and calls a healthy fleet behind. The median crossing 1.0 is
unambiguous: half the cache serves were past due. The panel now states those baselines, so the
number is interpretable without doing the reasoning.

The per-route table used a different statistic than the tile above it — 0.34× on the tile and
0.89× in the table for the same route, with nothing on screen to reconcile them. Both are the
median now; each row's p95 is in its tooltip.

The ratio now defers to the verdicts that outrank it. hit/swr/stale are decided per
request against each page's own expiry. This panel can only divide by the cadence the route
configures, and those differ whenever a target's interval comes from its stored sitemap
changefreq — which no metric exposes. When the ratio says "behind" while the verdicts say
nothing was past due, the divisor is what is wrong, and the panel says so rather than letting a
config gap read as a fleet failure. Silent on this deployment, where every prerender route sets a
renderInterval.

Origin cost per verdict leads with the median; its p95 moved to the tooltip.

Verification

Rendered every changed panel against a real 6h window pulled from a prod node, not only
fixtures:

Serve time · cache hit  2ms   median · p95 3ms · origin-served 371ms
Staleness               0.35× median · p95 0.87×
/product/prd-  48.0h cadence  age median 16.3h  0.34×
/              1.0h  cadence  age median 30m    0.50×   ← textbook for an evenly refreshed route
contradiction detector: silent (correct — the routes set their own intervals)

Console suite 115 → 121. The fixture now carries a median distinct from its p95: with a flat
distribution no test could tell which statistic a panel had used, which is the whole subject here.
Lint + format clean, plugin suite 715 unaffected. Console only.

Follow-up worth its own PR (plugin side)

Charts can only draw mean and p95bucketize() builds per-bucket means and p95s arrays
but keeps median as a scalar, so a "typical" trend line is impossible today and the tiles carry
the median alone. Adding medians to the per-bucket arrays is a small change in
util/analyticsRead.js plus the console's merge, but it needs a plugin release and a
prerender-cluster deploy.

… v0.6.0

You are right that p95 is the wrong default, and none of this is an SLA. The payload
carries mean, median and p95; they answer different questions, and the console was
using one of them nearly everywhere.

CAPACITY IS A MEAN. The queue view tiled and charted render p95 directly under a note
saying "renders/hour = concurrency ÷ time". Throughput follows the average service
time, not the tail, so that number argued away a third of the fleet: measured on prod
just now, p95 16.0s against a mean of 11.0s. The tile is the mean and says what it is
for; the p95 rides in the subtitle, where a widening gap between the two is the tell
that some renders are pathological rather than the whole fleet being slower. The mean
is also the only statistic that merges exactly across combos, buckets and nodes — a
merged median or p95 is a count-weighted average of percentiles, which is not the
percentile of the pooled population.

SERVE TIME WAS TWO POPULATIONS IN ONE NUMBER, which is worse than the wrong
percentile. Cache hits run 2ms and origin proxies 371ms; pooled at the current 54/46
split that is "462ms p95", a figure that improves when offload improves and says
nothing about how fast either path is. The tile now reports the cache-hit median with
the origin-served median beside it.

STALENESS IS JUDGED ON THE MEDIAN. A page's age walks from zero to its interval and is
re-rendered, so an evenly refreshed corpus sits at median 0.50x with its p95 already at
~0.95x — a p95 threshold has no headroom and flags a healthy fleet as behind. The
median crossing 1.0 means something unambiguous instead: half the cache serves were
past due. Live numbers on the whole corpus: median 0.35x, p95 0.87x. The per-route
table now uses the same statistic as the tile, which it did not: 0.34x on the tile and
0.89x in the table for the same route, with nothing on screen to reconcile them.

And the ratio now defers to the verdicts that outrank it. hit/swr/stale are decided per
request against each page's OWN expiry; this panel can only divide by the cadence the
ROUTE configures, and those differ whenever a target's interval comes from its stored
sitemap changefreq — which no metric exposes. When the ratio says "behind" and the
verdicts say nothing was past due, the divisor is what is wrong, and the panel says so
instead of letting a config gap read as a fleet failure. Silent on this deployment,
where every prerender route sets a renderInterval.

Origin cost per verdict leads with the median (its p95 is in the tooltip): "what does a
miss cost" is a typical-case question.

Verified by rendering every changed panel against a real 6h window pulled from a prod
node, not only against fixtures. The test fixture now carries a median distinct from
its p95, because with a flat distribution no test could tell which statistic a panel
had used — which is the entire subject of this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the prerender management console to use more appropriate statistical metrics (mean, median, and p95) across various panels. Specifically, it switches to using the median for page age, staleness, and serve times, and the mean for render capacity, while keeping p95 to represent tail latency. It also adds a warning note when freshness verdicts contradict the calculated staleness ratio, updates the corresponding tests, and improves documentation. A review comment suggests ensuring that ageTailP95 is explicitly validated as a finite number before division to prevent incorrect formatting when the value is null or undefined.

Comment on lines +1163 to +1167
? el('span', {
cls: ratio > 1 ? 'pill warn' : 'mono',
text: fmtRatio(ratio),
title: `p95 ${fmtRatio(cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN)}`,
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If ageTailP95 is null or undefined (which can happen if weighted returns null), the expression ageTailP95 / cadence.interval will evaluate to 0 in JavaScript (since null / number === 0). This causes fmtRatio to format it as "0.00×" instead of showing a missing value ("—").

To prevent this, we should explicitly check if ageTailP95 is a finite number before performing the division. Since metrics or database values can sometimes be surfaced as BigInt/Long, always coerce the value to a Number first before calling Number.isFinite to avoid false negatives.

Suggested change
? el('span', {
cls: ratio > 1 ? 'pill warn' : 'mono',
text: fmtRatio(ratio),
title: `p95 ${fmtRatio(cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN)}`,
})
? el('span', {
cls: ratio > 1 ? 'pill warn' : 'mono',
text: fmtRatio(ratio),
title: 'p95 ' + fmtRatio(Number.isFinite(Number(ageTailP95)) && cadence.interval > 0 ? ageTailP95 / cadence.interval : NaN),
})
References
  1. When validating database column values (which may be surfaced as BigInt/Long) using Number.isFinite, always coerce the value to a Number first, as Number.isFinite returns false for BigInts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant