diff --git a/CHANGELOG.md b/CHANGELOG.md index 2df99dd..73c8088 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -45,6 +45,7 @@ Notable changes to this stack. Format follows ### Changed +- Streamlined repository documentation and onboarding guides around researcher workflows, extracting host deployment guards and S3 storage migration into `docs/RUNBOOK.md` and refreshing dashboard screenshots. - `bootstrap.sh` creates only a `watchdog` healthchecks.io check per project/environment, since that check reports the other scheduled jobs. The `backup` and `restore-check` checks it used to create were never diff --git a/README.md b/README.md index 507e35f..e1dc364 100644 --- a/README.md +++ b/README.md @@ -6,54 +6,36 @@ [![Grafana LGTM](https://img.shields.io/badge/Grafana-Loki%20%7C%20Tempo%20%7C%20Prometheus-f46800?logo=grafana&logoColor=white)](compose.yml) [![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE) -Central monitoring for CML's research software. One host runs Grafana, Loki, -Tempo, Prometheus, and an OpenTelemetry Collector. Projects send logs, traces, -and metrics here over OTLP (the OpenTelemetry protocol), and Grafana shows -them side by side. +Central monitoring for CML research software. A single host runs Grafana, Loki, +Tempo, Prometheus, and an OpenTelemetry Collector. Research applications send +metrics, traces, and logs over standard OTLP (OpenTelemetry protocol), and Grafana +displays them side by side. -## Try it in one command - -You need Docker with the [Compose plugin](https://docs.docker.com/compose/install/) -and [`just`](https://github.com/casey/just#installation). Everything else runs -in containers. - -```sh -cp .env.example .env -just demo -``` - -This starts the stack plus a small FastAPI service under constant load -(`compose.demo.yml`). The service is auto-instrumented and fails one request -in ten, so the error panels and the error-rate alert have something to show. -Give it a minute, then open Grafana at -(admin / change-me): +## Who reports here -- **Dashboards → Service Health (RED)**: request rate, error rate, and - latency. The dots on the latency panel are exemplars. Click one to open the - trace behind that measurement. -- **Dashboards → Logs**: log volume by service and level, an error feed, and - a live tail. -- **Alerting → Alert rules**: the rules Grafana evaluates. `HighErrorRate` - trips on the demo service after five minutes. +| Project | Environment | Telemetry | Host | Spoke overlay | +| --- | --- | --- | --- | --- | +| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/prod` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` | +| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/staging` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` | +| [CML Gallery](https://github.com/CMLPlatform/cml-gallery) | Planned | Probes, RUM | GitHub Pages | `templates/web/rum.html` | -![Service Health (RED) dashboard](docs/img/service-health.png) +The alert marker in `config/grafana/alerting/projects.yaml` guards active reporting. +The `ProjectsUncovered` rule fires if an unlisted spoke begins transmitting. -`just demo-down` removes the demo services. `just demo-destroy` removes the -whole demo stack. +## Connect your project -The demo runs under its own compose project on its own port, so it never -touches a stack already running on the host. The same holds for `just smoke` -on :3001. +Connecting a service takes about five minutes without application code changes. +See [docs/ONBOARDING.md](docs/ONBOARDING.md) for step-by-step recipes: -## How it works +1. **Web services (FastAPI/Python):** Add standard OTel environment variables to emit traces, RED metrics, and correlated logs. +2. **Static websites and portals:** Register an outside-in uptime probe (`just probe-add`) or drop in the browser Real User Monitoring snippet ([`templates/web/rum.html`](templates/web/rum.html)) for Core Web Vitals. +3. **Project hosts and VMs:** Run `./bootstrap.sh ` to deploy a Grafana Alloy agent for container logs and host resources. -Everything enters through one gateway, the OpenTelemetry Collector. A project -configures a single endpoint, and a storage backend can be swapped later -without touching any application. +## Architecture ```mermaid flowchart LR - apps["Project apps"] -->|"OTLP over HTTPS
(gRPC/HTTP direct on private nets)"| cf["Cloudflare Tunnel
(production, optional)"] + apps["Project apps & sites"] -->|"OTLP over HTTPS
(HTTP/protobuf)"| cf["Cloudflare Tunnel
(production)"] user["Browser"] -->|HTTPS| cf cf --> otel cf --> grafana @@ -67,160 +49,53 @@ flowchart LR end ``` -Solid arrows write telemetry. Dotted arrows are Grafana reading at query -time. Locally there is no tunnel, and Grafana is at `localhost:3000` for -`just up` or `localhost:3002` for `just demo`. +The stack operates on a hub-and-spoke model: -The stack runs on a single host. [ADR 0001](docs/adr/0001-observability-stack.md) -records why, and the alternatives. [ADR 0002](docs/adr/0002-hub-and-spoke-observability.md) -describes the hub-and-spoke design that serves multiple projects. +- **Single gateway:** Everything enters through the OpenTelemetry Collector on one port. +- **Unified alerting:** Grafana evaluates all alert rules across both metrics and logs. No separate Alertmanager daemon. +- **Outside-in detection:** External probers and dead-man's switches are the only alarms whose default state is firing. Silence indicates failure, not health. -### Who reports here +Architecture decisions and trade-offs are documented in [ADR 0001 (Stack Selection)](docs/adr/0001-observability-stack.md) and [ADR 0002 (Hub-and-Spoke Design)](docs/adr/0002-hub-and-spoke-observability.md). -| Project | Environment | Telemetry | Host | Spoke overlay | -| --- | --- | --- | --- | --- | -| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/prod` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` | -| [R9lab](https://github.com/CMLPlatform/relab) | `r9lab/staging` | Logs, Traces, Metrics | CML VM | `templates/compose.telemetry.yml` | -| [CML Gallery](https://github.com/CMLPlatform/cml-gallery) | Planned | Logs, Metrics | CML Host | `templates/compose.telemetry.yml` | - -The public alert marker in `config/grafana/alerting/projects.yaml` guards active -reporting; `ProjectsUncovered` trips if an unlisted spoke begins transmitting. +## Try it in one command -## Run it for real +Prerequisites: Docker with [Compose plugin](https://docs.docker.com/compose/install/) and [`just`](https://github.com/casey/just#installation). ```sh -cp .env.example .env # set GRAFANA_ADMIN_PASSWORD -just up -just check # validate every config in the repo +cp .env.example .env +just demo ``` -Grafana: . +This launches the core monitoring stack and a demo FastAPI service under simulated load on port 3002: -`COMPOSE_FILE` in `.env` names the overlays a host runs. In production, set -`COMPOSE_FILE=compose.yml:compose.tunnel.yml`. `just up`, `just restart`, `just backup`, -`just restore`, and plain `docker compose logs` or `ps` then act on that set. +- Open Grafana at (`admin` / `change-me`). +- View **Dashboards → Service Health (RED)** for request rates, error rates, and latency traces. +- Stop and clean up: `just demo-destroy`. -With the tunnel overlay active, `just up` refuses to start until five settings -are real: a generated `OTLP_AUTH_TOKEN`, a changed `GRAFANA_ADMIN_PASSWORD`, -`GRAFANA_ROOT_URL` pointing at the tunnel hostname, `GRAFANA_COOKIE_SECURE=true`, -and a non-empty `ALERT_WEBHOOK_URL`. It also refuses `GRAFANA_JWT_AUTH=true` -without `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD`, and a `.env`, -`infra/terraform.tfvars` or OpenTofu state file that its group or other users -can read. -An empty `HEARTBEAT_URL` only warns. +![Service Health dashboard](docs/img/service-health.png) -The Cloudflare edge is code too. The tunnel, its hostnames, DNS, and the -Access rule that puts an email one-time PIN in front of Grafana live in -`infra/` as an OpenTofu configuration. Applying it produces the -`CLOUDFLARE_TUNNEL_TOKEN` the overlay needs. The bootstrap steps are at the -top of [infra/main.tf](infra/main.tf). +## Operating the stack -### Checks +Host deployment, secret rotation, pre-flight environment checks, alert matrices, and backups are documented in [docs/RUNBOOK.md](docs/RUNBOOK.md). -| Recipe | What it does | -| --- | --- | -| `just lint` | Static checks: compose files, rendered alert templates, YAML, workflows, shell, Python, OpenTofu formatting, dashboard JSON, git history for secrets | -| `just validate` | Runs each stack config through the exact image the stack uses | -| `just check` | `lint` plus `validate` | -| `just smoke` | Boots an isolated copy of the stack and asserts it works end to end | -| `just restore-check` | Rehearses backup and restore on the smoke stack | -| `just hooks` | Installs the git hooks: gitleaks, a CITATION.cff schema check and citesync at commit, a Conventional Commits check on the message, `check` at push, `infra-validate` at push when `infra/` changed | - -Every `just` check runs in a pinned container. Nothing is installed on the host. -The citation hooks are the exception: prek builds their Python environments itself. - -`just smoke` covers what has no offline validator. It boots the stack in -production shape with JWT auth on, then asserts that every dashboard and -alert rule provisioned, that the contact points carry the URLs from the -environment, that Grafana refuses a forged token, that every scrape target is -up, and that a metric and a log posted through the collector come back out of -Prometheus and Loki with their labels. `SMOKE_ALERTS=1 just smoke` also stops -a scrape target and waits for `TargetDown` to fire, about three minutes more. -The assertions are in [scripts/smoke.sh](scripts/smoke.sh). - -CI runs `lint` on one job and `validate` plus `smoke` (with the alert round -trip) on another, on every pull request. - -## Sending telemetry from a project - -You need the OTLP endpoint, the bearer token, and a few naming conventions. -[docs/ONBOARDING.md](docs/ONBOARDING.md) has copy-paste templates for -applications. [templates/README.md](templates/README.md) covers the host -agent that ships container logs and host metrics. - -> [!WARNING] -> Never publish ports 4317/4318 to the internet. The compose file binds them to -> `127.0.0.1`. The tunnel is the way in. - -## Alerting - -Grafana evaluates and delivers the rules in `config/grafana/alerting/`: -telemetry silent per project, a project sending telemetry with no rule file, -container crash-looping or OOM-killed, scrape target down, OTel export -failures, alert delivery failing, error rate above 5%, p99 latency above 2s, -host memory above 90%, disk above 80%, disk projected full within 3 days, -Prometheus active series above 30k, and 5,000 new series in 30 minutes. There is no Alertmanager. Grafana rules can query -Loki as well as Prometheus, and one engine means one answer to "who gets -told". - -Notifications go to the webhook in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so -on). One rule, `Watchdog`, fires permanently and posts to `HEARTBEAT_URL` -every five minutes. Point that at a dead man's switch such as healthchecks.io, -which alerts when the pings stop. That is the only way to notice the host -itself dying. - -Set both variables. An unset `ALERT_WEBHOOK_URL` drops every alert while the -heartbeat keeps reporting healthy. With the tunnel overlay, `just up` refuses -to start without it. - -## Storage - -Everything persists to local Docker volumes, captured by `just backup`. The -one exception is `otel_queue`, the collector's on-disk export queue. It holds -seconds of in-flight telemetry and is not worth restoring. - -When local disk stops fitting, Loki and Tempo can move to any S3-compatible -object store. The appendix of ADR 0001 documents that change. - -## Layout +In production, run: ```sh -compose.yml # core services -compose.tunnel.yml # production overlay: Cloudflare Tunnel -compose.demo.yml # demo overlay: sample telemetry source -compose.sandbox.yml # isolation overlay for `just demo` and `just smoke` -demo/ # the demo FastAPI service -scripts/smoke.sh # what `just smoke` asserts against the booted stack -config/ - otel-collector.yaml # ingestion gateway (OTLP in → Loki/Tempo/Prometheus out) - loki.yaml # logs - tempo.yaml # traces - prometheus.yaml # metrics - grafana/ # provisioned datasources, dashboards, and alerting -dashboards/ # JSON dashboards; Grafana loads them from here -docs/ # runbook, onboarding, ADRs, screenshots -infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS -templates/ # files a project host vendors: agent config, overlays +# Set COMPOSE_FILE=compose.yml:compose.tunnel.yml in .env +just up +just check ``` -`dashboards/*.json` is the source of truth. The directory is mounted read-only -and UI saves are disabled, so a change made in the browser lasts until the -page reloads. Edit the JSON and provisioning picks it up within 30 seconds. To -keep something built interactively, export it as JSON and paste it into the -file. - ## Documentation -| Document | What it covers | +| Document | Purpose | | --- | --- | -| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connecting an application: endpoint, token, copy-paste templates | -| [templates/README.md](templates/README.md) | Onboarding a project host: the agent, GPU hosts, removing a project | -| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Operations: rotating secrets, disk pressure, backup and restore | -| [docs/adr/](docs/adr/) | Why the stack looks like this | -| [CHANGELOG.md](CHANGELOG.md) | Release history | -| [CONTRIBUTING.md](.github/CONTRIBUTING.md) | Issues, pull requests, checks, commit style | -| [SECURITY.md](.github/SECURITY.md) | Reporting a vulnerability, and what is in scope | -| [Code of Conduct](.github/CODE_OF_CONDUCT.md) | How we treat each other, and how to report a problem | - -Licensed under the [MIT License](LICENSE). To cite the stack, use GitHub's -"Cite this repository" button, which reads [CITATION.cff](CITATION.cff). +| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connect applications, static sites, and browser frontends | +| [templates/README.md](templates/README.md) | Deploy host agent (Grafana Alloy) for host metrics and container logs | +| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Host operations, pre-flight guards, alerts, and troubleshooting | +| [docs/adr/](docs/adr/) | Architectural Decision Records (ADR 0001, ADR 0002) | +| [CHANGELOG.md](CHANGELOG.md) | Release history and upgrade notes | +| [CONTRIBUTING.md](.github/CONTRIBUTING.md) | Contribution workflow, conventions, and checks | +| [SECURITY.md](.github/SECURITY.md) | Security policy and vulnerability disclosure | + +Licensed under the [MIT License](LICENSE). To cite the stack, see [CITATION.cff](CITATION.cff). diff --git a/docs/ONBOARDING.md b/docs/ONBOARDING.md index a48c311..024632c 100644 --- a/docs/ONBOARDING.md +++ b/docs/ONBOARDING.md @@ -1,49 +1,56 @@ -# Sending telemetry from your application - -You need an endpoint, a token, and a few naming conventions. Then pick the -template that matches how your application runs. A Python/FastAPI service -needs no code changes. - -## The endpoint - -| | | -| --- | --- | -| Production (via tunnel) | `https://otel.`, OTLP **HTTP** (`http/protobuf`) only | -| Private network / same host | `:4317` (gRPC) or `:4318` (HTTP) | -| Auth | `Authorization: Bearer ` (ask the stack operator) | - -The tunnel routes HTTPS to the collector's HTTP receiver only. Set -`OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` when you send through it. gRPC -works on private paths only (VPN, WireGuard, or the same Docker network). -Never expose 4317/4318 directly. - -## The conventions - -- **`service.name`** is required: one stable name per deployable unit - (`r9lab-api`, not `r9lab-api-prod-2`). Dashboards key on it. -- **`project`** and **`env`** are both required resource attributes: `project` - is the name `bootstrap.sh` was run with, `env` is `prod`, `staging`, or - `dev`. Together they are what every alert and dashboard filters on, and a - sender that omits them is counted as `unknown` and alerts as an uncovered - project. -- **Keep labels low-cardinality.** Prometheus turns every distinct label - value into a series. User IDs, request IDs, and timestamps belong in the - log line or in span attributes, not in resource attributes or metric labels. -- **Loki indexes only the identity labels**: `service.name`, `department`, - `project`, `env`, `host.name` (the list is in `config/loki.yaml`). Every - other attribute is structured metadata. Select the stream first, then - filter: `{service_name="my-service", env="prod"} | service_instance_id="..."`. - -The Service Health dashboard and the error-rate alert read the standard HTTP -server metrics (`http_server_request_duration_seconds`). The -auto-instrumentation below emits them. - -## Template 1: Python/FastAPI, zero code changes +# Connecting a Project + +Choose your project type to find the right onboarding recipe: + +| Project Type | What it monitors | Setup Time | Recipe | +| --- | --- | --- | --- | +| **Python / FastAPI Web App** | Traces, RED metrics, error logs | 5 minutes | [Template 1: Web Services](#template-1-python--fastapi-services) | +| **CLI / Batch Script / Other Lang** | Custom traces, metrics, logs | 5 minutes | [Template 2: General Applications](#template-2-general-applications-any-language) | +| **Static Website / Portal** | Availability, HTTP 2xx latency, TLS expiry | 1 minute | [Template 3: Static & Edge Sites](#template-3-static--edge-websites-outside-in) | +| **Browser Frontend / Web App** | Core Web Vitals (LCP, CLS), client errors | 2 minutes | [Template 4: Browser RUM](#template-4-browser-real-user-monitoring-rum) | +| **Entire VM or Server Host** | Host CPU/RAM/Disk, container lifecycle, stdout | 10 minutes | [Host Agent: Grafana Alloy](#host-agent-container-logs-and-host-metrics) | + +--- + +## Endpoint and Credentials + +Request the active endpoint URL and ingest token from the monitoring administrator: + +| Path | Ingestion Target | Protocol | Notes | +| --- | --- | --- | --- | +| **Production (via tunnel)** | `https://otel.` | OTLP HTTP (`http/protobuf`) | Uses Cloudflare Tunnel | +| **Private network / same host** | `:4317` (gRPC) or `:4318` (HTTP) | OTLP gRPC or HTTP | Direct internal networking | +| **Authentication** | `Authorization: Bearer ` | Header | Enforced at gateway | + +> [!WARNING] +> Never expose ports 4317 or 4318 directly to the public internet. The compose stack binds them to `127.0.0.1`. + +--- + +## Required Naming Conventions + +Every signal requires three identity labels to route correctly and avoid alerting rules tripping: + +1. **`service.name`**: One stable identifier per deployable unit (e.g. `r9lab-api`). Dashboards filter on this name. +2. **`project`**: The project identifier registered with the hub (e.g. `r9lab`). +3. **`env`**: Environment tier: `prod`, `staging`, or `dev`. + +Keep metric labels low-cardinality. Put user identifiers, request IDs, and query parameters inside log messages or span attributes, never in resource attributes or metric labels. + +--- + +## Template 1: Python / FastAPI Services + +Auto-instrument a Python application with zero code changes. + +### 1. Install dependencies ```sh pip install opentelemetry-distro opentelemetry-exporter-otlp opentelemetry-instrumentation-fastapi ``` +### 2. Export environment variables + ```sh export OTEL_SERVICE_NAME=my-service export OTEL_RESOURCE_ATTRIBUTES=project=my-project,env=prod @@ -52,29 +59,36 @@ export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true export OTEL_SEMCONV_STABILITY_OPT_IN=http +``` + +### 3. Run the service +```sh opentelemetry-instrument uvicorn app:app --host 0.0.0.0 --port 8000 ``` -That gives traces, RED metrics, and logs with trace context, with no OTel -code in the app. A working example is this repo's [`demo/`](../demo/) service -plus [`compose.demo.yml`](../compose.demo.yml). +This exports traces, RED metrics, and correlated logs to the hub. A working reference is available in [`demo/`](../demo/) and [`compose.demo.yml`](../compose.demo.yml). + +--- -## Template 2: any language, plain OTLP +## Template 2: General Applications (Any Language) -Every OpenTelemetry SDK reads the same four environment variables: +Any standard OpenTelemetry SDK reads these four environment variables: ```sh OTEL_SERVICE_NAME=my-service +OTEL_RESOURCE_ATTRIBUTES=project=my-project,env=prod OTEL_EXPORTER_OTLP_ENDPOINT=https://otel.example.org OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer " -OTEL_RESOURCE_ATTRIBUTES=project=my-project,env=prod ``` -## Template 3: Static & Edge Websites (Zero Code) +Configure your application SDK to use standard OTLP/HTTP export. -For static sites (GitHub Pages, Cloudflare Pages, documentation portals), add outside-in -uptime probing via Prometheus Blackbox Exporter: +--- + +## Template 3: Static & Edge Websites (Outside-In) + +For documentation portals, static pages (GitHub Pages, Cloudflare Pages), or edge sites without a backend server, register an external probe: ```sh just probe-add [tier] @@ -86,38 +100,39 @@ Example for a public gallery: just probe-add cml-gallery prod https://cmlplatform.github.io/cml-gallery/ edge ``` -Prometheus re-reads `config/targets/probes.yaml` automatically without a restart, -tracking HTTP 2xx availability, response latency, and TLS certificate expiration. -Outages trigger the `BlackboxProbeFailed` alert without false `ProjectTelemetrySilent` alarms. +Prometheus reloads `config/targets/probes.yaml` automatically without a restart. The blackbox exporter probes HTTP 2xx availability, response latency, and TLS certificate expiration. Downtime trips the `BlackboxProbeFailed` alert without false telemetry warnings. + +--- ## Template 4: Browser Real User Monitoring (RUM) -For browser-side performance (Core Web Vitals) and error tracking on static or single-page sites, -include the zero-dependency snippet from [`templates/web/rum.html`](../templates/web/rum.html): +For client-side performance and error tracking on web applications or static portals, include the lightweight snippet from [`templates/web/rum.html`](../templates/web/rum.html): ```html ``` -The OTel Collector's HTTP receiver includes preconfigured CORS headers allowing cross-origin -vitals and error logs from `https://cmlplatform.github.io` and `https://*.leidenuniv.nl`. +What it captures: + +- **Core Web Vitals:** Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS). +- **Client Exceptions:** Unhandled `window.onerror` exceptions and unhandled promise rejections. +- **Log Streams:** Emits structured OTLP/HTTP JSON (`/v1/logs`) indexed into Loki. + +The collector permits cross-origin requests from `https://cmlplatform.github.io` and Leiden University domains (`*.leidenuniv.nl`). See [`templates/web/README.md`](../templates/web/README.md) for details. + +--- + +## Host Agent: Container Logs and Host Metrics -> [!NOTE] -> For public client sites, avoid embedding shared administrative secrets in client-side HTML. -> Route browser telemetry through a reverse proxy or edge function that injects the Authorization -> header, or configure a dedicated ingestion token. +Applications cannot monitor host CPU, memory, disk saturation, or container crash loops. -## Container logs and host metrics +A shared Grafana Alloy agent runs on each project host to collect system metrics and Docker logs. Run `./bootstrap.sh ` on the monitoring host to generate the project configuration and alerts. -An application cannot report other containers' stdout, host resources, or its -own crash loops. One Grafana Alloy agent per host ships those, over the same -endpoint and token. Run `./bootstrap.sh ` on the monitoring -host and follow what it prints. Details are in -[templates/README.md](../templates/README.md). +Complete host agent deployment steps are documented in [templates/README.md](../templates/README.md). diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md index 1e50735..c9dca26 100644 --- a/docs/RUNBOOK.md +++ b/docs/RUNBOOK.md @@ -314,6 +314,61 @@ cd infra && tofu apply `infra/generate-imports.sh > infra/imports.tf`, which reads the live objects back out of the Cloudflare API and adopts them into a fresh state. +## Production host deployment and pre-flight guards + +In production, run the tunnel overlay alongside the core compose file: + +```sh +cp .env.example .env +# Edit .env and set COMPOSE_FILE=compose.yml:compose.tunnel.yml +just up +just check +``` + +When the tunnel overlay is active, `just up` validates critical security settings before starting: + +- **`OTLP_AUTH_TOKEN`**: Refuses the default `local-dev-token`. Generate one with `openssl rand -hex 32`. +- **`GRAFANA_ADMIN_PASSWORD`**: Refuses `change-me`. Set a strong password. +- **`GRAFANA_ROOT_URL`**: Must match the `https://` tunnel hostname for Grafana. +- **`GRAFANA_COOKIE_SECURE=true`**: Required when Grafana runs behind an HTTPS tunnel. +- **`ALERT_WEBHOOK_URL`**: Cannot be empty; empty URLs drop all fired alerts. +- **`GRAFANA_JWT_AUTH=true`**: Requires both `CF_ACCESS_TEAM_DOMAIN` and `CF_ACCESS_AUD` set. +- **File permissions**: Refuses to start if `.env`, `infra/terraform.tfvars`, or Terraform state files are readable by other users (mode must be `0600`). +- **`HEARTBEAT_URL`**: Emits a warning if unset (the stack runs without a dead-man's switch). + +### Verification commands + +| Command | Action | +| --- | --- | +| `just lint` | Runs static checks (compose, alert templates, YAML, workflows, shell, Python, JSON). | +| `just validate` | Validates each service config with its respective tool container. | +| `just check` | Runs `just lint` and `just validate`. | +| `just smoke` | Boots an isolated copy of the stack and asserts ingestion, storage, and query APIs end-to-end. | +| `just restore-check` | Verifies backup snapshot creation and restoration against the smoke stack. | +| `just hooks` | Installs git pre-commit and pre-push hooks (gitleaks, citation checks, check). | + +## Storage architecture and S3 migration + +Telemetry persists to local Docker volumes managed by `docker compose`: + +- `loki_data`: Loki log chunks. +- `tempo_data`: Tempo trace blocks. +- `prometheus_data`: Prometheus time-series metrics. +- `grafana_data`: Dashboard definitions and alert state. +- `otel_queue`: Ephemeral collector export buffer (skipped during backups). + +Create volume snapshots using `just backup`, which creates a timestamped archive in `backups/`. + +### Migrating to S3 object storage + +When local disk space no longer accommodates Loki or Tempo retention: + +1. Provision S3-compatible buckets for logs and traces (e.g. AWS S3, Cloudflare R2, MinIO). +2. Update `config/loki.yaml` storage schema to use `s3` object storage. +3. Update `config/tempo.yaml` storage block with bucket and endpoint configuration. +4. Restart services with `just restart loki` and `just restart tempo`. +See the appendix of [ADR 0001](adr/0001-observability-stack.md) for detailed configuration snippets. + ## Cutting a release A tag is the deployable unit: `bootstrap.sh` pins every spoke's vendored diff --git a/docs/img/service-health.png b/docs/img/service-health.png index a79f364..92fb1eb 100644 Binary files a/docs/img/service-health.png and b/docs/img/service-health.png differ diff --git a/docs/img/stack-health.png b/docs/img/stack-health.png index ac68421..3c06c83 100644 Binary files a/docs/img/stack-health.png and b/docs/img/stack-health.png differ diff --git a/templates/web/README.md b/templates/web/README.md index b66f83f..e701797 100644 --- a/templates/web/README.md +++ b/templates/web/README.md @@ -1,19 +1,18 @@ # Frontend Real User Monitoring (RUM) Spoke Template -This template provides lightweight, zero-dependency browser observability for -static websites, documentation portals, and edge-served web apps (e.g., GitHub Pages, -Cloudflare Pages). +Lightweight, zero-dependency browser telemetry for static websites, documentation portals, and edge web applications (GitHub Pages, Cloudflare Pages). ## What it captures -1. **Core Web Vitals**: Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS). -2. **Unhandled JavaScript Errors**: `window.onerror` exceptions and unhandled promise rejections. -3. **Traceable Logs**: Shipped as standard OTLP/HTTP JSON (`/v1/logs`) directly to the central - OpenTelemetry Collector. +1. **Core Web Vitals:** Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS). +2. **Unhandled errors:** `window.onerror` exceptions and unhandled promise rejections. +3. **Structured logs:** Shipped as standard OTLP/HTTP JSON (`/v1/logs`) to the central collector. -## Usage in Astro / HTML +## Integration -1. Define your endpoint and project configuration before the snippet: +### 1. Set configuration variables + +Add your endpoint and project details before loading the snippet: ```html ``` -2. Copy the contents of [`rum.html`](rum.html) into your layout `` or before ``. +### 2. Include the script + +Paste the contents of [`rum.html`](rum.html) into your layout `` or before ``. -## Ingestion & Security +## Ingestion and Security -- **CORS Allowed Origins:** The collector's HTTP receiver (`0.0.0.0:4318`) allows cross-origin - POST requests from `https://cmlplatform.github.io` and `https://*.leidenuniv.nl`. -- **Credential Hygiene:** For public client apps, avoid embedding shared cluster-wide secrets. - Route telemetry through an edge proxy that injects the auth header, or supply a dedicated - scoped ingest token. -- **Log Indexing:** Logs are indexed in Loki by `service_name`, `project`, and `env`, viewable - on the **Logs** dashboard. +- **CORS origins:** The collector HTTP receiver permits cross-origin requests from `https://cmlplatform.github.io` and Leiden University domains (`https://*.leidenuniv.nl`). +- **Credential safety:** Do not embed cluster-wide admin secrets in client HTML. Route telemetry through an edge proxy that injects the Authorization header, or provision a dedicated ingestion token. +- **Log queries:** Logs are indexed in Loki by `service_name`, `project`, and `env`, viewable on the Grafana **Logs** dashboard.