[Iamsdt]

Summary

This PR adds a metrics block to 10xgraph.json for request, task and LLM metrics. Metrics are kept separate from AI tracing: HTTP spans never reach an AI tracing backend. It also passes the new core Phoenix and OTLP trace backends through to the API, and gives 10xgraph init a full setup mode that can scaffold a background worker.

What changed

Metrics (core/config/metrics.py, new)

• New metrics block in 10xgraph.json, with export set to "otlp" (push) or "prometheus" (scrape):

  "metrics": {
    "enabled": true,
    "export": "otlp",
    "otlp": {"endpoint": "http://collector:4318"},
    "prometheus": {"path": "/metrics", "worker_port": 9464},
    "latency_buckets_ms": [50, 100, 250, 500, 1000, 2500, 5000]
  }

• The block is validated with pydantic: unknown keys are refused, the buckets must be ascending, export is required when enabled, and the path cannot be /.
• The API and the worker each set up their own MeterProvider. The worker serves Prometheus on worker_port.
• The Prometheus scrape route is public, like /ping, and skips the auth guard, the Host check and the rate limit. Startup fails if another route matches the scrape path. Scrape requests are left out of the HTTP metrics.
• HTTP spans go to a no-op tracer unless the legacy OTEL_ENABLED path is on. FastAPI's own built-in telemetry is turned off on FastAPI versions that have it, so request spans never reach Phoenix, LangSmith, Logfire or OTLP.
• observability.otlp.use_metrics_endpoint sends AI traces to the same OTLP endpoint as metrics. It warns if that endpoint is a Prometheus OTLP URL, since Prometheus accepts metrics only.
• On shutdown, both the API and the worker send their last metrics and AI spans, with a 5-second limit. This runs last, so a slow collector delays nothing else.
• New extra: 10xgraph-api[metrics].

AI tracing

• The observability block now also accepts phoenix and otlp, using the core 1.0 helpers. Secrets come from env vars (PHOENIX_API_KEY, OTEL_EXPORTER_OTLP_HEADERS).
• The worker now sets up the same observability as the API.

Jobs

• MonitoringMiddleware now publishes task events to event publishers only, using publish_event(..., only="events"). Before, AI tracing backends received them too and showed each task as a graph run.
• A task failure is now logged at ERROR and counted, but no longer published as an event.
• The task duration metric is renamed from tenxgraph.task.duration_ms to tenxgraph.task.duration.

10xgraph init

• --mode quick|full. The old --template stays as an alias.
• New flags for full mode: --storage (memory, sqlite, postgres, postgres-memory), --evals/--no-evals, --tests/--no-tests, --ag-ui/--no-ag-ui, --otel/--no-otel, and --worker (none, redis, rabbitmq, kafka).
• --worker scaffolds worker/tasks.py with an example task.
• The production template now defaults to memory storage and JWT auth, and its .env.example covers Phoenix and metrics.
• init.py is split into init_context.py and init_prompts.py.

Config editor

• New Telemetry page for the metrics and observability blocks, with schema and validation.

Fix

• Shutdown now finds the checkpointer the same way the startup check and /health/ready do, through readiness_targets. Before, it looked it up by a string key the graph never binds.

Example

• examples/observability/: a local stack (OpenTelemetry Collector, Prometheus, Grafana with a 10xGraph dashboard, Redis) bound to 127.0.0.1. It is for local use only.

Behavior changes to note

• The tenxgraph.task.duration_ms metric is now tenxgraph.task.duration.
• Task failures are no longer published as events.
• Without OTEL_ENABLED, the API produces no HTTP spans. FastAPI's built-in telemetry is off.
• The production init template now defaults to memory storage and JWT.

Dependencies

• uv.lock pins core main at 0b65e3b, which includes PR #341 (Phoenix and OTLP tracing, PublisherKind, publish_event(only=...)).
• The pyproject.toml floor is still 10xgraph>=1.0.0rc4. publish_event(only=...) exists only on core main, so the floor has to move to the next core rc at release.

Tests

• New: test_metrics.py, test_metrics_real_packages.py, test_init_full.py, test_init_worker.py, test_worker.py, and config editor tests.
• Updated: middleware, init and storage tests.
• uv run pytest -m "not integration" against the updated lock: 3208 passed, 14 skipped.

[codecov]

Codecov Report

❌ Patch coverage is 94.53552% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tenxgraph_api/cli/commands/init_prompts.py	60.00%	6 Missing and 6 partials ⚠️
...xgraph_api/src/app/core/config/setup_middleware.py	89.70%	5 Missing and 2 partials ⚠️
tenxgraph_api/cli/commands/init.py	91.42%	5 Missing and 1 partial ⚠️
tenxgraph_api/cli/commands/init_context.py	98.54%	1 Missing and 1 partial ⚠️
tenxgraph_api/src/app/core/config/metrics.py	98.85%	2 Missing ⚠️
tenxgraph_api/cli/config_editor/validation.py	97.87%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!