diff --git a/.env.example b/.env.example index 9e288ba..7a668ef 100644 --- a/.env.example +++ b/.env.example @@ -24,7 +24,7 @@ OTLP_AUTH_TOKEN=local-dev-token # Where Grafana delivers alert notifications (self-hosted ntfy, hosted ntfy, Slack, …). # For self-hosted ntfy with basic auth publisher: -# ALERT_WEBHOOK_URL=http://grafana:password@ntfy:80/alerts?tpl=1&title={{.CommonAnnotations.summary}}&message={{.CommonAnnotations.description}} +# ALERT_WEBHOOK_URL=http://grafana:password@ntfy:80/alerts?tpl=1&title={{or .commonAnnotations.summary .groupLabels.alertname}}&message={{or .commonAnnotations.description .commonAnnotations.summary "Alert firing"}} # Not optional: an empty value drops every alert silently while the heartbeat # below keeps reporting healthy. With the tunnel overlay active, `just up` # refuses to start without it. diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index d59dcd6..8d7f139 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -28,7 +28,7 @@ Python environment for the citation hooks itself). `just check` runs the static validates every config, `just smoke` starts a throwaway copy of the stack and sends it telemetry, and `just fmt` formats the YAML. CI runs `just lint`, `just validate` and `just smoke`, plus `just infra-validate` when `infra/` changes. The -[Checks](../README.md#checks) table in the README lists every recipe. +[Verification commands](../docs/RUNBOOK.md#verification-commands) table in the runbook lists every recipe. ## Write commits and pull requests diff --git a/CHANGELOG.md b/CHANGELOG.md index 73c8088..e48ae65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,17 @@ Notable changes to this stack. Format follows ## [Unreleased] +## [0.5.0] - 2026-11-01 + +### Upgrade + +- Re-vendor `templates/run_scheduled.sh` on spokes for accurate cancellation and SIGINT reporting (exit code 130). +- Re-vendor `templates/alloy/compose.telemetry.yml` on spokes for Alloy `v1.21.2`. +- Hub operators: + - If using Cloudflare Tunnel, run `tofu apply` in `infra/` to provision the new tunnel ingress route and CNAME for `ntfy`. + - Run `just up` to create the `ntfy_data` persistent volume and launch the Blackbox exporter and `ntfy` containers. + - In `.env`, ensure `ALERT_WEBHOOK_URL` uses `.commonAnnotations` rather than `.CommonAnnotations` so alert titles and messages render correctly in webhook payloads. + ### Added - Dynamic file-based service discovery for blackbox uptime probing (`config/targets/probes.yaml` @@ -46,6 +57,7 @@ Notable changes to this stack. Format follows ### Changed - Streamlined repository documentation and onboarding guides around researcher workflows, extracting host deployment guards and S3 storage migration into `docs/RUNBOOK.md` and refreshing dashboard screenshots. +- Renamed covered research project identifier from `relab` to `r9lab` across `projects.yaml` and dashboard filters. - `bootstrap.sh` creates only a `watchdog` healthchecks.io check per project/environment, since that check reports the other scheduled jobs. The `backup` and `restore-check` checks it used to create were never @@ -53,6 +65,8 @@ Notable changes to this stack. Format follows ### Fixed +- Stat panels in Service Health dashboard explicitly request instant metrics to prevent Prometheus query timeouts on 30-day time ranges. +- Webhook alert templates use lowercase `.commonAnnotations` to prevent template render errors and ntfy HTTP 400 Bad Request delivery drops. - Replaced the dead upstream `nvidia_gpu_exporter` tag link in the GPU dashboard with unified stack navigation. - Moved `Container Restarts (1h)` to the top of the Containers section in Host & @@ -405,3 +419,11 @@ First tagged release: the stack is runnable, demoable, and validated in CI. Initial stack: OTel Collector → Loki / Tempo / Prometheus → Grafana wired for cross-signal correlation, exposed via Cloudflare Tunnel, with RELab dashboards. + +[Unreleased]: https://github.com/CMLPlatform/monitoring/compare/v0.5.0...HEAD +[0.5.0]: https://github.com/CMLPlatform/monitoring/compare/v0.4.0...v0.5.0 +[0.4.0]: https://github.com/CMLPlatform/monitoring/compare/v0.3.0...v0.4.0 +[0.3.0]: https://github.com/CMLPlatform/monitoring/compare/v0.2.0...v0.3.0 +[0.2.0]: https://github.com/CMLPlatform/monitoring/compare/v0.1.0...v0.2.0 +[0.1.0]: https://github.com/CMLPlatform/monitoring/compare/v0.0.1...v0.1.0 +[0.0.1]: https://github.com/CMLPlatform/monitoring/releases/tag/v0.0.1 diff --git a/CITATION.cff b/CITATION.cff index d805f24..c84e625 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -11,3 +11,14 @@ authors: orcid: "https://orcid.org/0009-0006-6953-909X" repository-code: "https://github.com/CMLPlatform/monitoring" license: MIT +version: 0.5.0 +date-released: "2026-11-01" +keywords: + - "observability" + - "opentelemetry" + - "grafana" + - "loki" + - "tempo" + - "prometheus" + - "research-infrastructure" + - "docker-compose" diff --git a/dashboards/service-health.json b/dashboards/service-health.json index e15cc07..6ba2327 100644 --- a/dashboards/service-health.json +++ b/dashboards/service-health.json @@ -113,6 +113,7 @@ "targets": [ { "expr": "sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))", + "instant": true, "legendFormat": "Requests", "refId": "A" } @@ -168,6 +169,7 @@ "targets": [ { "expr": "(100 * sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__rate_interval])) / sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval]))) or vector(0)", + "instant": true, "legendFormat": "Error Rate", "refId": "A" } @@ -232,6 +234,7 @@ "targets": [ { "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", + "instant": true, "legendFormat": "p95", "refId": "A" } @@ -295,6 +298,7 @@ "targets": [ { "expr": "histogram_quantile(0.5, sum by (le) (rate(http_server_request_duration_seconds_bucket{project=\"$project\", env=\"$env\", job=\"$service\"}[$__rate_interval])))", + "instant": true, "legendFormat": "p50", "refId": "A" } @@ -358,6 +362,7 @@ "targets": [ { "expr": "100 * (1 - ((sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__range])) or vector(0)) / (sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__range])) > 0)))", + "instant": true, "legendFormat": "Availability", "refId": "A" } @@ -423,6 +428,7 @@ "targets": [ { "expr": "100 * (1 - (((sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\", http_response_status_code=~\"5..\"}[$__range])) or vector(0)) / (sum(rate(http_server_request_duration_seconds_count{project=\"$project\", env=\"$env\", job=\"$service\"}[$__range])) > 0)) / 0.005))", + "instant": true, "legendFormat": "Budget Remaining", "refId": "A" } diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index c9dca26..4ebfba7 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -59,25 +59,8 @@ legitimately grows into its ceiling. ## Alert reference All alerts evaluate in Grafana (`config/grafana/alerting/`) and deliver to -`ALERT_WEBHOOK_URL` (with `Watchdog` pinging `HEARTBEAT_URL`): - -| Alert | Severity | Trigger condition | Triage action | -| --- | --- | --- | --- | -| `Watchdog` | none | Fires permanently | Dead-man's switch heartbeat. If external pings stop, the host or Grafana is down. | -| `AlertDeliveryFailing` | critical | Grafana cannot deliver alerts for 10m | Check webhook URL connectivity and webhook receiver logs. | -| `TargetDown` | critical | Prometheus cannot scrape an internal service for 2m | Run `docker compose ps` and `docker compose logs `. | -| `OtelExportFailures` | warning | Collector dropping/failing exports for 5m | Check downstream receiver logs (Loki, Tempo, Prometheus); check disk/memory. | -| `ProjectTelemetrySilent` | critical | Onboarded spoke stopped sending telemetry for 15m | Inspect spoke agent/host; check Cloudflare tunnel or network connectivity. | -| `ProjectsUncovered` | warning | A project is sending telemetry but missing from `projects.yaml` | Add the project/environment to `config/grafana/alerting/projects.yaml`. | -| `ContainerRestarting` | warning | Spoke or hub container restarted > 3 times in 1h | Check container crashloop logs (`docker logs --tail 100 `). | -| `ContainerOOMKilled` | warning | Container killed by Linux kernel OOM killer | Check container memory ceiling in compose file or spoke allocation. | -| `HighErrorRate` | warning | Application HTTP 5xx error rate > 5% for 5m | Inspect service RED dashboard, drill into error logs and Tempo traces. | -| `HighLatencyP99` | warning | Application p99 latency > 2s for 5m | Check DB connection pool usage, slow routes on RED dashboard, and trace spans. | -| `HostDiskSpaceLow` | warning | Host disk usage > 80% for 15m | Free space, prune docker images, or shorten retention in Loki/Tempo. | -| `HostDiskFilling` | warning | Linear forecast predicts disk full within 3 days | Check for runaway log generation or Prometheus metric cardinality explosion. | -| `HostMemoryHigh` | warning | Host RAM usage > 90% for 15m | Check container working sets on Host & Containers dashboard. | -| `PrometheusCardinalityHigh` | warning | Active timeseries > 30,000 for 30m | Inspect label explosion using Prometheus TSDB status API. | -| `PrometheusCardinalitySpike` | warning | > 5,000 new timeseries within 30m | High churn in metric labels; identify which service introduced non-cardinal labels. | +`ALERT_WEBHOOK_URL` (with `Watchdog` pinging `HEARTBEAT_URL`). See the complete +[Alert catalog](#alert-catalog) below for all 18 provisioned rules and triage procedures. ## Backup and restore @@ -120,7 +103,7 @@ on the host. run `cd infra && tofu apply` to refresh the token data source, then read the new value with `tofu output -raw tunnel_token`. To rotate from code instead, run `tofu apply -replace=cloudflare_zero_trust_tunnel_cloudflared.monitoring`. - That builds a new tunnel and repoints both CNAMEs at it, and ingestion and + That builds a new tunnel and repoints all three CNAMEs (grafana, otel, ntfy) at it, and ingestion and Grafana are unreachable for about a minute. Either way, put the new token in `CLOUDFLARE_TUNNEL_TOKEN` in `.env`, then `just up`. - **Grafana admin password:** change `GRAFANA_ADMIN_PASSWORD` in `.env`, @@ -162,7 +145,7 @@ credential. ## Alert catalog -Eighteen alert rules guard the stack and reporting spokes across six folders. +Eighteen alert rules guard the stack and reporting spokes across seven alert groups in the Stack alerts folder. The PromQL threshold evaluates instant expressions; a rule fires when its comparison stays true for the duration of `for`: @@ -184,8 +167,8 @@ comparison stays true for the duration of `for`: | `PrometheusCardinalitySpike` | capacity | warning | 15m | Head series spike by >5,000 in 30 minutes. | | `ContainerRestarting` | container-lifecycle | warning | 10m | Container restarts >3 times within 1 hour. | | `ContainerOOMKilled` | container-lifecycle | warning | 0m | Container terminated by host kernel OOM killer. | -| `ProjectTelemetrySilent` | spoke | warning | 15m | Spoke service stopped shipping OTel telemetry. | -| `ProjectUnscheduledRestart` | spoke | warning | 5m | Spoke container restarted unexpectedly. | +| `ProjectTelemetrySilent` | spoke | critical | 15m | Spoke service stopped shipping OTel telemetry. | +| `ProjectsUncovered` | _coverage | warning | 5m | Telemetry received from unlisted project; register in projects.yaml. | ## Alert delivery @@ -247,7 +230,7 @@ Point `ALERT_WEBHOOK_URL` in `.env` to the local ntfy instance using native message templating (`?tpl=1`): ```sh -ALERT_WEBHOOK_URL=http://grafana:@ntfy:80/alerts?tpl=1&title={{.CommonAnnotations.summary}}&message={{.CommonAnnotations.description}} +ALERT_WEBHOOK_URL=http://grafana:@ntfy:80/alerts?tpl=1&title={{or .commonAnnotations.summary .groupLabels.alertname}}&message={{or .commonAnnotations.description .commonAnnotations.summary "Alert firing"}} ``` Apply the change to Grafana: @@ -332,6 +315,7 @@ When the tunnel overlay is active, `just up` validates critical security setting - **`GRAFANA_ROOT_URL`**: Must match the `https://` tunnel hostname for Grafana. - **`GRAFANA_COOKIE_SECURE=true`**: Required when Grafana runs behind an HTTPS tunnel. - **`ALERT_WEBHOOK_URL`**: Cannot be empty; empty URLs drop all fired alerts. +- **`NTFY_BASE_URL`**: Must be an https:// URL when exposing ntfy via tunnel. - **`GRAFANA_JWT_AUTH=true`**: Requires both `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` set. - **File permissions**: Refuses to start if `.env`, `infra/terraform.tfvars`, or Terraform state files are readable by other users (mode must be `0600`). - **`HEARTBEAT_URL`**: Emits a warning if unset (the stack runs without a dead-man's switch). @@ -345,6 +329,7 @@ When the tunnel overlay is active, `just up` validates critical security setting | `just check` | Runs `just lint` and `just validate`. | | `just smoke` | Boots an isolated copy of the stack and asserts ingestion, storage, and query APIs end-to-end. | | `just restore-check` | Verifies backup snapshot creation and restoration against the smoke stack. | +| `just probe-add` | Registers an outside-in HTTP/S probe target in `config/targets/probes.yaml`. | | `just hooks` | Installs git pre-commit and pre-push hooks (gitleaks, citation checks, check). | ## Storage architecture and S3 migration @@ -355,6 +340,7 @@ Telemetry persists to local Docker volumes managed by `docker compose`: - `tempo_data`: Tempo trace blocks. - `prometheus_data`: Prometheus time-series metrics. - `grafana_data`: Dashboard definitions and alert state. +- `ntfy_data`: SQLite user database and message cache for ntfy. - `otel_queue`: Ephemeral collector export buffer (skipped during backups). Create volume snapshots using `just backup`, which creates a timestamped archive in `backups/`. diff --git a/templates/run_scheduled.sh b/templates/run_scheduled.sh index 3812c1e..1d08fd4 100755 --- a/templates/run_scheduled.sh +++ b/templates/run_scheduled.sh @@ -64,14 +64,20 @@ ping_status() { # shellcheck disable=SC2329 # invoked via the TERM/INT traps below on_terminate() { local sig="$1" + local exit_code=143 + local reason="job killed (likely a systemd timeout)" + if [[ "$sig" == "INT" ]]; then + exit_code=130 + reason="job cancelled" + fi [[ -z "${job_pid:-}" ]] || kill -TERM -- "-$job_pid" 2>/dev/null - echo "run_scheduled: received SIG${sig}; job killed (likely a systemd timeout)" >>"$output_file" + echo "run_scheduled: received SIG${sig}; ${reason}" >>"$output_file" cat "$output_file" if [[ -n "$ping_url" ]]; then - ping_status 143 + ping_status "$exit_code" fi rm -f "$output_file" - exit 143 + exit "$exit_code" } trap 'on_terminate TERM' TERM trap 'on_terminate INT' INT diff --git a/templates/web/README.md b/templates/web/README.md index e701797..d0e61d6 100644 --- a/templates/web/README.md +++ b/templates/web/README.md @@ -20,7 +20,7 @@ Add your endpoint and project details before loading the snippet: window.__OTEL_PROJECT__ = "cml-gallery"; window.__OTEL_SERVICE__ = "cml-gallery"; window.__OTEL_ENV__ = "prod"; - window.__OTEL_TOKEN__ = ""; // Required when collector enforces bearer auth + window.__OTEL_TOKEN__ = ""; // Leave blank when using an edge proxy or unauthenticated endpoint ``` @@ -31,5 +31,5 @@ Paste the contents of [`rum.html`](rum.html) into your layout `` or before ## Ingestion and Security - **CORS origins:** The collector HTTP receiver permits cross-origin requests from `https://cmlplatform.github.io` and Leiden University domains (`https://*.leidenuniv.nl`). -- **Credential safety:** Do not embed cluster-wide admin secrets in client HTML. Route telemetry through an edge proxy that injects the Authorization header, or provision a dedicated ingestion token. +- **Credential safety:** Do not embed collector bearer secrets in public client HTML. For public apps, either omit `window.__OTEL_TOKEN__` and let an edge proxy (such as Cloudflare Workers or Nginx) inject `Authorization: Bearer ...`, or provision a scoped low-privilege token. - **Log queries:** Logs are indexed in Loki by `service_name`, `project`, and `env`, viewable on the Grafana **Logs** dashboard. diff --git a/templates/web/rum.html b/templates/web/rum.html index 7e2f3d3..4aa1ebb 100644 --- a/templates/web/rum.html +++ b/templates/web/rum.html @@ -118,7 +118,7 @@ window.addEventListener("error", function(evt) { sendLog(evt.message || "Unhandled JavaScript Error", "ERROR", { error_type: "uncaught_exception", - filename: evt.filename || "unknown", + filename: evt.filename ? evt.filename.split("?")[0] : "unknown", lineno: evt.lineno || 0, colno: evt.colno || 0, stack: evt.error ? evt.error.stack || "" : ""