🎃 Add an opt-in Prometheus /metrics endpoint for tool calls #1003

Part of the October Challenge #997.

Why

Operators who self-host AnythingMCP want tool-call rates, error rates and latency in Grafana, next to the rest of their stack. Today they only have the Pino tool_call log lines, which they have to parse themselves, and the database (ToolInvocation). Our ROADMAP lists "per-tool latency histograms, error-rate dashboards". Most self-hosted gateways (Kong, Traefik, n8n) ship a Prometheus endpoint.

Self-hosted vs Cloud

• Self-hosted: off by default. When METRICS_ENABLED=true, the backend starts a second, separate HTTP listener for /metrics (it does not use the main app port 4000). Available in Community; no licence gate.
• Cloud (DEPLOYMENT_MODE=cloud): the operator (us) uses the same code. Two extra rules: the listener refuses to start without METRICS_TOKEN (it logs an error and metrics stay off), and per-tool/per-connector labels are always off. No series may carry an organization id, connector name, tool name or user. This prevents cross-tenant leaks and keeps cardinality bounded with thousands of workspaces. Per-org usage already exists in the database (AuditService.getBreakdowns).
• Who can use it: whoever operates the instance (env vars). There are no app roles, no dashboard UI and no org scoping, because metrics are instance-wide by design. That is why they must never be reachable through the public port.

What to build

1. Library: use prom-client. Recommendation and reasons: @opentelemetry/exporter-prometheus would hook into the existing NodeSDK (tracing.ts). But that SDK only starts when OTEL_EXPORTER_OTLP_ENDPOINT is set, its 0.x packages must move in lockstep with @opentelemetry/sdk-node (^0.222.0), and Node runtime metrics need extra packages. prom-client gives collectDefaultMetrics() (event loop lag, heap, GC) for free and is trivial to unit-test (await register.metrics() returns a string). Wrap it in a small facade (src/common/metrics.ts: recordToolCall(...)), so a later switch to OTel metrics touches one file.
2. Env vars: METRICS_ENABLED (default false), METRICS_PORT (default 9464), METRICS_HOST (default 0.0.0.0, because in Docker the Prometheus container must reach it over the compose network; the port is not published in docker-compose.yml), METRICS_TOKEN (optional on self-hosted, required on Cloud, compared with timingSafeEqual), METRICS_TOOL_LABELS (default false, ignored on Cloud with a warning).
3. Listener: a plain node:http server started from main.ts after app.listen(), closed on shutdown. It serves only GET /metrics. Any other path returns 404, and a wrong or missing token returns 401.
4. Metrics (prefix anythingmcp_):
   • tool_calls_total{connector_type,status} counter (status = success|error)
   • tool_call_duration_seconds{connector_type,status} histogram (buckets roughly 0.05 to 60 s; tool calls can take 20 s or more)
   • tool_call_upstream_errors_total{connector_type,upstream_status_class} (4xx|5xx|none)
   • with METRICS_TOOL_LABELS=true (self-hosted only), add connector_id and tool labels. Use the connector id (cuid), not the name.
   • gauges mirrored from the existing processGauges (toolCallsInFlight, mcpRequestsInFlight, registeredTools…), using a collect() callback
   • collectDefaultMetrics()

Where to look

• packages/backend/src/mcp-server/dynamic-mcp-tools.ts:130: logToolCall() is called on both the success and the error path and already has durationMs, status, upstreamStatus and connectorType. Record the metrics here.
• packages/backend/src/common/process-vitals.ts:246: ProcessGauges / processGauges.snapshot(), existing in-process gauges to expose.
• packages/backend/src/tracing.ts:26: the existing opt-in OTel setup (pattern for an env-gated, no-op-by-default module).
• packages/backend/src/main.ts:160: app.listen(port). Start the metrics listener after it.
• packages/backend/src/health/sentry-verify.controller.ts:27: sameSecret(), a constant-time token comparison to reuse.
• packages/backend/src/common/deployment.service.ts:10: isCloud().
• deploy/cloud/Caddyfile:91 and docs/deployment.md:221: /health* is proxied publicly on Cloud and self-hosted. That is why metrics must not live under /health or on port 4000.

Acceptance criteria

• With METRICS_ENABLED unset, no listener starts and prom-client default metrics are not collected.
• GET :9464/metrics returns Prometheus text format, and the counters and histogram move after a tool call.
• With METRICS_TOKEN set, a missing or wrong Authorization: Bearer returns 401.
• With DEPLOYMENT_MODE=cloud and no token, the listener does not start. With METRICS_TOOL_LABELS=true on Cloud, no tool/connector_id label appears.
• Unit tests in src/common/metrics.spec.ts (labels, Cloud rules, token check). Run with cd packages/backend && npx jest src/common/metrics.
• Docs: new section "Prometheus metrics (opt-in)" in docs/operations/observability.md, with a sample scrape config, plus the env vars in the table in docs/deployment.md (around line 338).

Out of scope

• Grafana dashboards or alert rules (a follow-up issue).
• Pushing metrics via OTLP.
• Per-organization metrics or any UI.
• Changing docker-compose.cloud.yml or the Cloud Caddyfile (maintainers wire scraping on Cloud).

Size

M (1–2 days)

How to claim

Comment "I'd like to work on this" and we'll assign you. Rules in #997.