From de60642a544c9d59c9d830ada123c3401d6354de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Tom=C3=A1s=20Gr=C3=BCner?= <47506558+MegaRedHand@users.noreply.github.com> Date: Thu, 6 Aug 2026 15:05:01 -0300 Subject: [PATCH] fix(metrics): widen lean_block_building_time_seconds buckets past 1s The leanMetrics bucket set for this histogram tops out at 1s, but block builds on our devnets routinely take 2-3s. Every sample therefore landed in `+Inf`, and `histogram_quantile` had no finite bucket left to interpolate in, so it returned the upper bound of the last finite bucket. The panel read as a flat 1s line regardless of how long builds actually took. Buckets now span 0.1s to 8s, the same range as `lean_block_proposal_attestation_build_phase_seconds`, whose phases this metric encloses. The tradeoff is sub-100ms resolution: attestation-free blocks skip the prover and build in single-digit milliseconds, and those now all collapse into the first bucket. The phase histogram still resolves that end, and it is not the range we need to watch. This deviates from the leanMetrics spec, so both the histogram and docs/metrics.md carry a note explaining why, to keep it from being "corrected" back to a set that cannot measure the thing. Divergent buckets do not affect other clients: Prometheus stores each bucket as its own series, and every boundary that was shared before (0.1 through 1) is still present. The leanMetrics dashboard panel for this metric queries per-instance rather than `sum by (le)` across clients, so each histogram stays internally consistent. --- crates/blockchain/src/metrics.rs | 5 ++++- docs/metrics.md | 8 +++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/crates/blockchain/src/metrics.rs b/crates/blockchain/src/metrics.rs index 41b70bd7..81d00c29 100644 --- a/crates/blockchain/src/metrics.rs +++ b/crates/blockchain/src/metrics.rs @@ -460,12 +460,15 @@ static LEAN_BLOCK_BUILDING_PAYLOAD_AGGREGATION_TIME_SECONDS: std::sync::LazyLock .unwrap() }); +// Widened past the leanMetrics bucket set: block builds regularly exceed its top bound, +// which collapsed every sample into `+Inf` and pinned the reported quantiles to that +// bound. The range mirrors the phase timings this metric encloses. static LEAN_BLOCK_BUILDING_TIME_SECONDS: std::sync::LazyLock = std::sync::LazyLock::new(|| { register_histogram!( "lean_block_building_time_seconds", "Time taken to build a block", - vec![0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1.0] + vec![0.1, 0.25, 0.5, 0.75, 1.0, 2.0, 4.0, 8.0] ) .unwrap() }); diff --git a/docs/metrics.md b/docs/metrics.md index 017dc9ea..10f274a3 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -36,7 +36,7 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le |--------|-------|-------|-------------------------|--------|---------|-----------| | `lean_block_aggregated_payloads` | Histogram | Number of `aggregated_payloads` in a block | On block production | | 1, 2, 4, 8, 16, 32, 64, 128 | ✅ | | `lean_block_building_payload_aggregation_time_seconds` | Histogram | Time taken to build `aggregated_payloads` during block building | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 3, 4 | ✅ | -| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1 | ✅ | +| `lean_block_building_time_seconds` | Histogram | Time taken to build a block | On block production | | 0.1, 0.25, 0.5, 0.75, 1, 2, 4, 8 | ✅ | | `lean_block_building_success_total` | Counter | Successful block builds | On block production | | | ✅ | | `lean_block_building_failures_total` | Counter | Failed block builds (error building the block, signing the block root, or processing it locally) | On block production failure | | | ✅ | | `lean_block_proposal_attestation_build_phase_seconds` | Histogram | Phase-level time in block-proposal attestation selection | On block production | phase=select_payloads,compact,stf_simulate | 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 4, 8 | ✅ | @@ -45,6 +45,12 @@ The exposed metrics follow [the leanMetrics specification](https://github.com/le | `lean_block_proposal_attestation_data_selected` | Histogram | Distinct `AttestationData` entries in the proposal block body | On block production | | 0, 1, 2, 4, 8, 16, 32 | ✅ | | `lean_block_proposal_aggregates_selected` | Histogram | Aggregated signature proofs in the proposal result after compaction | On block production | | 0, 1, 2, 4, 8, 16, 32, 64, 128 | ✅ | +> `lean_block_building_time_seconds` intentionally deviates from the leanMetrics bucket +> set, which tops out at 1s. Real builds on our devnets routinely run past that, so every +> sample landed in `+Inf` and `histogram_quantile` reported a flat 1s ceiling. The range +> now covers the same span as the `lean_block_proposal_attestation_build_phase_seconds` +> phases it contains. + ## Fork-Choice Metrics | Name | Type | Usage | Sample collection event | Labels | Buckets | Supported |