chore(release): prepare v0.5.0 and address pre-release review findings

.env.example

@@ -24,7 +24,7 @@ OTLP_AUTH_TOKEN=local-dev-token
 
 # Where Grafana delivers alert notifications (self-hosted ntfy, hosted ntfy, Slack, …).
 # For self-hosted ntfy with basic auth publisher:
-#   ALERT_WEBHOOK_URL=http://grafana:password@ntfy:80/alerts?tpl=1&title={{.CommonAnnotations.summary}}&message={{.CommonAnnotations.description}}
+#   ALERT_WEBHOOK_URL=http://grafana:password@ntfy:80/alerts?tpl=1&title={{or .commonAnnotations.summary .groupLabels.alertname}}&message={{or .commonAnnotations.description .commonAnnotations.summary "Alert firing"}}
 # Not optional: an empty value drops every alert silently while the heartbeat
 # below keeps reporting healthy. With the tunnel overlay active, `just up`
 # refuses to start without it.

.github/CONTRIBUTING.md

@@ -28,7 +28,7 @@ Python environment for the citation hooks itself). `just check` runs the static
 validates every config, `just smoke` starts a throwaway copy of the stack and sends it
 telemetry, and `just fmt` formats the YAML. CI runs `just lint`, `just validate` and
 `just smoke`, plus `just infra-validate` when `infra/` changes. The
-[Checks](../README.md#checks) table in the README lists every recipe.
+[Verification commands](../docs/RUNBOOK.md#verification-commands) table in the runbook lists every recipe.
 
 ## Write commits and pull requests
 

CHANGELOG.md

@@ -6,6 +6,17 @@ Notable changes to this stack. Format follows
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-11-01
+
+### Upgrade
+
+- Re-vendor `templates/run_scheduled.sh` on spokes for accurate cancellation and SIGINT reporting (exit code 130).
+- Re-vendor `templates/alloy/compose.telemetry.yml` on spokes for Alloy `v1.21.2`.
+- Hub operators:
+  - If using Cloudflare Tunnel, run `tofu apply` in `infra/` to provision the new tunnel ingress route and CNAME for `ntfy`.
+  - Run `just up` to create the `ntfy_data` persistent volume and launch the Blackbox exporter and `ntfy` containers.
+  - In `.env`, ensure `ALERT_WEBHOOK_URL` uses `.commonAnnotations` rather than `.CommonAnnotations` so alert titles and messages render correctly in webhook payloads.
+
 ### Added
 
 - Dynamic file-based service discovery for blackbox uptime probing (`config/targets/probes.yaml`
@@ -46,13 +57,16 @@ Notable changes to this stack. Format follows
 ### Changed
 
 - Streamlined repository documentation and onboarding guides around researcher workflows, extracting host deployment guards and S3 storage migration into `docs/RUNBOOK.md` and refreshing dashboard screenshots.
+- Renamed covered research project identifier from `relab` to `r9lab` across `projects.yaml` and dashboard filters.
 - `bootstrap.sh` creates only a `watchdog` healthchecks.io check per
   project/environment, since that check reports the other scheduled jobs.
   The `backup` and `restore-check` checks it used to create were never
   pinged; delete them at healthchecks.io. `HC_JOBS` still overrides the list.
 
 ### Fixed
 
+- Stat panels in Service Health dashboard explicitly request instant metrics to prevent Prometheus query timeouts on 30-day time ranges.
+- Webhook alert templates use lowercase `.commonAnnotations` to prevent template render errors and ntfy HTTP 400 Bad Request delivery drops.
 - Replaced the dead upstream `nvidia_gpu_exporter` tag link in the GPU dashboard
   with unified stack navigation.
 - Moved `Container Restarts (1h)` to the top of the Containers section in Host &
@@ -405,3 +419,11 @@ First tagged release: the stack is runnable, demoable, and validated in CI.
 Initial stack: OTel Collector → Loki / Tempo / Prometheus → Grafana wired
 for cross-signal correlation, exposed via Cloudflare Tunnel, with RELab
 dashboards.
+
+[Unreleased]: https://github.com/CMLPlatform/monitoring/compare/v0.5.0...HEAD
+[0.5.0]: https://github.com/CMLPlatform/monitoring/compare/v0.4.0...v0.5.0
+[0.4.0]: https://github.com/CMLPlatform/monitoring/compare/v0.3.0...v0.4.0
+[0.3.0]: https://github.com/CMLPlatform/monitoring/compare/v0.2.0...v0.3.0
+[0.2.0]: https://github.com/CMLPlatform/monitoring/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/CMLPlatform/monitoring/compare/v0.0.1...v0.1.0
+[0.0.1]: https://github.com/CMLPlatform/monitoring/releases/tag/v0.0.1

CITATION.cff

@@ -11,3 +11,14 @@ authors:
     orcid: "https://orcid.org/0009-0006-6953-909X"
 repository-code: "https://github.com/CMLPlatform/monitoring"
 license: MIT
+version: 0.5.0
+date-released: "2026-11-01"
+keywords:
+  - "observability"
+  - "opentelemetry"
+  - "grafana"
+  - "loki"
+  - "tempo"
+  - "prometheus"
+  - "research-infrastructure"
+  - "docker-compose"

dashboards/service-health.json

@@ -113,6 +113,7 @@
       "targets": [
         {
           "expr": "sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))",
+          "instant": true,
           "legendFormat": "Requests",
           "refId": "A"
         }
@@ -168,6 +169,7 @@
       "targets": [
         {
           "expr": "(100 * sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__rate_interval])) / sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))) or vector(0)",
+          "instant": true,
           "legendFormat": "Error Rate",
           "refId": "A"
         }
@@ -232,6 +234,7 @@
       "targets": [
         {
           "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))",
+          "instant": true,
           "legendFormat": "p95",
           "refId": "A"
         }
@@ -295,6 +298,7 @@
       "targets": [
         {
           "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))",
+          "instant": true,
           "legendFormat": "p50",
           "refId": "A"
         }
@@ -358,6 +362,7 @@
       "targets": [
         {
           "expr": "100 * (1 - ((sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__range])) or vector(0)) / (sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__range])) > 0)))",
+          "instant": true,
           "legendFormat": "Availability",
           "refId": "A"
         }
@@ -423,6 +428,7 @@
       "targets": [
         {
           "expr": "100 * (1 - (((sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__range])) or vector(0)) / (sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__range])) > 0)) / 0.005))",
+          "instant": true,
           "legendFormat": "Budget Remaining",
           "refId": "A"
         }

docs/RUNBOOK.md

@@ -59,25 +59,8 @@ legitimately grows into its ceiling.
 ## Alert reference
 
 All alerts evaluate in Grafana (`config/grafana/alerting/`) and deliver to
-`ALERT_WEBHOOK_URL` (with `Watchdog` pinging `HEARTBEAT_URL`):
-
-| Alert | Severity | Trigger condition | Triage action |
-| --- | --- | --- | --- |
-| `Watchdog` | none | Fires permanently | Dead-man's switch heartbeat. If external pings stop, the host or Grafana is down. |
-| `AlertDeliveryFailing` | critical | Grafana cannot deliver alerts for 10m | Check webhook URL connectivity and webhook receiver logs. |
-| `TargetDown` | critical | Prometheus cannot scrape an internal service for 2m | Run `docker compose ps` and `docker compose logs <service>`. |
-| `OtelExportFailures` | warning | Collector dropping/failing exports for 5m | Check downstream receiver logs (Loki, Tempo, Prometheus); check disk/memory. |
-| `ProjectTelemetrySilent` | critical | Onboarded spoke stopped sending telemetry for 15m | Inspect spoke agent/host; check Cloudflare tunnel or network connectivity. |
-| `ProjectsUncovered` | warning | A project is sending telemetry but missing from `projects.yaml` | Add the project/environment to `config/grafana/alerting/projects.yaml`. |
-| `ContainerRestarting` | warning | Spoke or hub container restarted > 3 times in 1h | Check container crashloop logs (`docker logs --tail 100 <container>`). |
-| `ContainerOOMKilled` | warning | Container killed by Linux kernel OOM killer | Check container memory ceiling in compose file or spoke allocation. |
-| `HighErrorRate` | warning | Application HTTP 5xx error rate > 5% for 5m | Inspect service RED dashboard, drill into error logs and Tempo traces. |
-| `HighLatencyP99` | warning | Application p99 latency > 2s for 5m | Check DB connection pool usage, slow routes on RED dashboard, and trace spans. |
-| `HostDiskSpaceLow` | warning | Host disk usage > 80% for 15m | Free space, prune docker images, or shorten retention in Loki/Tempo. |
-| `HostDiskFilling` | warning | Linear forecast predicts disk full within 3 days | Check for runaway log generation or Prometheus metric cardinality explosion. |
-| `HostMemoryHigh` | warning | Host RAM usage > 90% for 15m | Check container working sets on Host & Containers dashboard. |
-| `PrometheusCardinalityHigh` | warning | Active timeseries > 30,000 for 30m | Inspect label explosion using Prometheus TSDB status API. |
-| `PrometheusCardinalitySpike` | warning | > 5,000 new timeseries within 30m | High churn in metric labels; identify which service introduced non-cardinal labels. |
+`ALERT_WEBHOOK_URL` (with `Watchdog` pinging `HEARTBEAT_URL`). See the complete
+[Alert catalog](#alert-catalog) below for all 18 provisioned rules and triage procedures.
 
 ## Backup and restore
 
@@ -120,7 +103,7 @@ on the host.
   run `cd infra && tofu apply` to refresh the token data source, then read
   the new value with `tofu output -raw tunnel_token`. To rotate from code
   instead, run `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring`.
-  That builds a new tunnel and repoints both CNAMEs at it, and ingestion and
+  That builds a new tunnel and repoints all three CNAMEs (grafana, otel, ntfy) at it, and ingestion and
   Grafana are unreachable for about a minute. Either way, put the new token
   in `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`.
 - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`,
@@ -162,7 +145,7 @@ credential.
 
 ## Alert catalog
 
-Eighteen alert rules guard the stack and reporting spokes across six folders.
+Eighteen alert rules guard the stack and reporting spokes across seven alert groups in the Stack alerts folder.
 The PromQL threshold evaluates instant expressions; a rule fires when its
 comparison stays true for the duration of `for`:
 
@@ -184,8 +167,8 @@ comparison stays true for the duration of `for`:
 | `PrometheusCardinalitySpike` | capacity | warning | 15m | Head series spike by >5,000 in 30 minutes. |
 | `ContainerRestarting` | container-lifecycle | warning | 10m | Container restarts >3 times within 1 hour. |
 | `ContainerOOMKilled` | container-lifecycle | warning | 0m | Container terminated by host kernel OOM killer. |
-| `ProjectTelemetrySilent` | spoke | warning | 15m | Spoke service stopped shipping OTel telemetry. |
-| `ProjectUnscheduledRestart` | spoke | warning | 5m | Spoke container restarted unexpectedly. |
+| `ProjectTelemetrySilent` | spoke | critical | 15m | Spoke service stopped shipping OTel telemetry. |
+| `ProjectsUncovered` | _coverage | warning | 5m | Telemetry received from unlisted project; register in projects.yaml. |
 
 ## Alert delivery
 
@@ -247,7 +230,7 @@ Point `ALERT_WEBHOOK_URL` in `.env` to the local ntfy instance using native
 message templating (`?tpl=1`):
 
 ```sh
-ALERT_WEBHOOK_URL=http://grafana:<grafana-secret>@ntfy:80/alerts?tpl=1&title={{.CommonAnnotations.summary}}&message={{.CommonAnnotations.description}}
+ALERT_WEBHOOK_URL=http://grafana:<grafana-secret>@ntfy:80/alerts?tpl=1&title={{or .commonAnnotations.summary .groupLabels.alertname}}&message={{or .commonAnnotations.description .commonAnnotations.summary "Alert firing"}}
 ```
 
 Apply the change to Grafana:
@@ -332,6 +315,7 @@ When the tunnel overlay is active, `just up` validates critical security setting
 - **`GRAFANA_ROOT_URL`**: Must match the `https://` tunnel hostname for Grafana.
 - **`GRAFANA_COOKIE_SECURE=true`**: Required when Grafana runs behind an HTTPS tunnel.
 - **`ALERT_WEBHOOK_URL`**: Cannot be empty; empty URLs drop all fired alerts.
+- **`NTFY_BASE_URL`**: Must be an https:// URL when exposing ntfy via tunnel.
 - **`GRAFANA_JWT_AUTH=true`**: Requires both `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` set.
 - **File permissions**: Refuses to start if `.env`, `infra/terraform.tfvars`, or Terraform state files are readable by other users (mode must be `0600`).
 - **`HEARTBEAT_URL`**: Emits a warning if unset (the stack runs without a dead-man's switch).
@@ -345,6 +329,7 @@ When the tunnel overlay is active, `just up` validates critical security setting
 | `just check` | Runs `just lint` and `just validate`. |
 | `just smoke` | Boots an isolated copy of the stack and asserts ingestion, storage, and query APIs end-to-end. |
 | `just restore-check` | Verifies backup snapshot creation and restoration against the smoke stack. |
+| `just probe-add` | Registers an outside-in HTTP/S probe target in `config/targets/probes.yaml`. |
 | `just hooks` | Installs git pre-commit and pre-push hooks (gitleaks, citation checks, check). |
 
 ## Storage architecture and S3 migration
@@ -355,6 +340,7 @@ Telemetry persists to local Docker volumes managed by `docker compose`:
 - `tempo_data`: Tempo trace blocks.
 - `prometheus_data`: Prometheus time-series metrics.
 - `grafana_data`: Dashboard definitions and alert state.
+- `ntfy_data`: SQLite user database and message cache for ntfy.
 - `otel_queue`: Ephemeral collector export buffer (skipped during backups).
 
 Create volume snapshots using `just backup`, which creates a timestamped archive in `backups/`.

templates/run_scheduled.sh

@@ -64,14 +64,20 @@ ping_status() {
 # shellcheck disable=SC2329  # invoked via the TERM/INT traps below
 on_terminate() {
     local sig="$1"
+    local exit_code=143
+    local reason="job killed (likely a systemd timeout)"
+    if [[ "$sig" == "INT" ]]; then
+        exit_code=130
+        reason="job cancelled"
+    fi
     [[ -z "${job_pid:-}" ]] || kill -TERM -- "-$job_pid" 2>/dev/null
-    echo "run_scheduled: received SIG${sig}; job killed (likely a systemd timeout)" >>"$output_file"
+    echo "run_scheduled: received SIG${sig}; ${reason}" >>"$output_file"
     cat "$output_file"
     if [[ -n "$ping_url" ]]; then
-        ping_status 143
+        ping_status "$exit_code"
     fi
     rm -f "$output_file"
-    exit 143
+    exit "$exit_code"
 }
 trap 'on_terminate TERM' TERM
 trap 'on_terminate INT' INT

templates/web/README.md

@@ -20,7 +20,7 @@ Add your endpoint and project details before loading the snippet:
   window.__OTEL_PROJECT__ = "cml-gallery";
   window.__OTEL_SERVICE__ = "cml-gallery";
   window.__OTEL_ENV__ = "prod";
-  window.__OTEL_TOKEN__ = "<otlp-auth-token>"; // Required when collector enforces bearer auth
+  window.__OTEL_TOKEN__ = ""; // Leave blank when using an edge proxy or unauthenticated endpoint
 </script>
 ```
 
@@ -31,5 +31,5 @@ Paste the contents of [`rum.html`](rum.html) into your layout `<head>` or before
 ## Ingestion and Security
 
 - **CORS origins:** The collector HTTP receiver permits cross-origin requests from `https://cmlplatform.github.io` and Leiden University domains (`https://*.leidenuniv.nl`).
-- **Credential safety:** Do not embed cluster-wide admin secrets in client HTML. Route telemetry through an edge proxy that injects the Authorization header, or provision a dedicated ingestion token.
+- **Credential safety:** Do not embed collector bearer secrets in public client HTML. For public apps, either omit `window.__OTEL_TOKEN__` and let an edge proxy (such as Cloudflare Workers or Nginx) inject `Authorization: Bearer ...`, or provision a scoped low-privilege token.
 - **Log queries:** Logs are indexed in Loki by `service_name`, `project`, and `env`, viewable on the Grafana **Logs** dashboard.

templates/web/rum.html

@@ -118,7 +118,7 @@
   window.addEventListener("error", function(evt) {
     sendLog(evt.message || "Unhandled JavaScript Error", "ERROR", {
       error_type: "uncaught_exception",
-      filename: evt.filename || "unknown",
+      filename: evt.filename ? evt.filename.split("?")[0] : "unknown",
       lineno: evt.lineno || 0,
       colno: evt.colno || 0,
       stack: evt.error ? evt.error.stack || "" : ""