Distributed AI Inference

An inference platform that serves embedding and reranking models across many replicas and stays predictable when demand exceeds capacity. A Go gateway authenticates tenants, charges per-tenant quotas, sheds load it can't serve in time, splits large requests across replicas and routes around failed ones. Python model workers batch requests dynamically on ONNX Runtime. It runs on Docker Compose or Kubernetes, and CI tests it on a real kind cluster.

Stack: Go 1.25 (gateway, load generator) · Python 3.13 · FastAPI · ONNX Runtime (bge-small-en-v1.5 embeddings, MiniLM cross-encoder reranker) · Prometheus · Grafana · Docker · Kubernetes (Kustomize, KEDA, HPA, NetworkPolicy, Pod Security) · kind · GitHub Actions

The problem

Model serving usually breaks in the same few ways:

• Overload turns into timeouts. Requests queue in front of the model until every one of them misses its deadline. The server is 100% busy and serves almost nothing.
• One tenant takes everyone down. A nightly indexing job floods the cluster, and the search box times out for real users.
• Replicas fail and deploy all the time. Each restart becomes a burst of client errors.
• Large requests can't use the cluster. A request with 500 texts runs on one replica while the others sit idle.

What it does

• Admission control sized with Little's law. The gateway admits only as much work as the replicas can finish within the latency budget, and refuses the rest within a millisecond (503 + Retry-After). Admitted requests keep their latency under any load.
• Tenant quotas and priorities.
  • Each tenant has a token bucket counted in texts, not requests, so batching can't dodge a quota.
  • Batch-tier tenants may fill only 70% of capacity and are shed first.
• Deadlines end to end. X-Timeout-Ms becomes a deadline that travels to the workers, which drop expired work before running the model.
• Fan-out. Large requests are split into chunks and sent to different replicas in parallel, and the results are merged in order.
• Failure handling.
  • Replicas are routed by power of two choices on outstanding work.
  • A failed call is retried once on another replica.
  • A replica that keeps failing is ejected for a while.
  • DNS discovery follows scaling and rollouts.
• Dynamic batching in each worker: concurrent requests share one model call, with a bounded queue.
• Kubernetes-ready.
  • Headless services for discovery; readiness tied to a loaded, warmed model.
  • Graceful drain on both sides.
  • PodDisruptionBudgets, NetworkPolicies and the restricted Pod Security profile.
  • KEDA autoscaling on queue depth rather than CPU.

Who it is for: teams that serve models to several applications from shared infrastructure, such as embeddings for search, reranking for RAG, or classification APIs, and need latency guarantees and isolation rather than best effort. It is also a reference for the serving patterns (admission control, deadlines, batching, isolation) that make that work.

Results

python bench/run.py runs every scenario against the Compose stack with an open-loop load generator: requests keep arriving at a fixed rate whatever the latency, as real traffic does. The full tables are in eval/reports/benchmarks.md.

Scenario	Without	With
Overload at 2× capacity, real model	0.05 req/s served, 99.9% time out	38.6 req/s served, p95 567 ms, 0 timeouts	admission control
Overload at 2× capacity, 2 replicas	4.7 req/s served, 72% time out	173 req/s served, p95 278 ms, 0 timeouts	admission control
Noisy neighbour: batch tenant flooding at 3×, interactive at 50%	interactive 23.8% served	interactive 98.7% served, p95 311 ms	priority admission
Replica killed (SIGKILL) at 60% load, 3 replicas	—	100% served, 0 errors, p95 202 ms	retries, ejection, discovery
Pod deleted under load on a real kind cluster (CI, every push)	—	100% of 340 requests served	readiness, preStop drain, DNS discovery
Scale-out, saturation throughput	1 replica: 85 req/s	4 replicas: 340 req/s, p95 337 ms
Fan-out, 128 texts in one request	1 replica: p50 1,096 ms	4 replicas: p50 412 ms	8 chunks in parallel
Batching, accelerator-like model	17.6 req/s	86.4 req/s (4.9×)	max batch 8
Batching, bge-small on a laptop CPU	36.8 req/s	39.2 req/s (1.07×)	max batch 16

The machine is an i7-1260P laptop, and every worker container is limited to 2 CPUs. Two things about the numbers:

• Where the synthetic backend is used. CPU-bound replicas on one laptop all compete for the same power-limited chip, so a second real-model replica adds no throughput. The scale-out, isolation, failover and fan-out rows therefore use a synthetic backend whose cost is waiting, not computing (50 ms per call + 5 ms per text), as a model on its own accelerator behaves. The HTTP, batching, queueing and failure paths are all real.
• Batching helps an accelerator far more than a CPU. On the CPU, padding variable-length texts eats most of the gain; on an accelerator the fixed per-call cost dominates.

The design rationale behind each number is in docs/DESIGN.md.

The dashboard during a mixed load. When one of three replicas is killed, healthy replicas drop to two and the batch tenant's shed rate rises. The interactive tenant stays at about 120 req/s with no errors (99.7% served).

Architecture

flowchart LR
  T1[search app<br/>interactive] & T2[support bot<br/>interactive] & T3[nightly index<br/>batch] --> GW
  subgraph GW[gateway · Go · N replicas]
    direction TB
    A[auth + quota<br/>per tenant, in texts] --> AC[admission control<br/>Little's law, priority share]
    AC --> FO[deadline + fan-out<br/>16-text chunks]
    FO --> LB[power of two choices<br/>retry · eject]
  end
  LB --> E[embed workers<br/>batcher → ONNX Runtime]
  LB --> R[rerank workers<br/>batcher → ONNX Runtime]
  DNS[(headless service DNS)] -. discovery .-> LB
  E & R & GW -. metrics .-> P[Prometheus → Grafana, KEDA]

Loading

Quick start

docker compose up -d --build            # gateway :8080 · Grafana :3000 · Prometheus :9090

curl -s localhost:8080/v1/rerank -H "Authorization: Bearer demo-key-support-bot" \
  -d '{"query": "why is the pump vibrating", "top_n": 2, "documents": ["Invoice 4471 is overdue",
       "Pump vibration rises when the bearing wears out", "The compressor seal leaked"]}'

docker compose up -d --scale worker-embed=3     # the gateway discovers new replicas through DNS

The demo keys (demo-key-search-app, demo-key-support-bot, demo-key-nightly-index, admin demo-admin-key) exist only as SHA-256 hashes in config/tenants.json. Replace them before exposing anything. To deploy to a kind cluster and run the end-to-end checks, use make e2e.

Documentation

docs/DESIGN.md	Request path, admission control and Little's law, batching, discovery, Kubernetes design, profiling findings, trade-offs
docs/OPERATIONS.md	Configuration, sizing the admission limit, API status codes, metrics and alerts, runbooks
docs/DEPLOYMENT.md	kind and production Kubernetes (KEDA, ingress, secrets), sizing, rollouts
docs/DEVELOPMENT.md	Layout, tests, running locally, benchmarks
docs/known-limitations.md	Static admission limit, large requests, per-gateway counting, and more
eval/reports/benchmarks.md	Every benchmark table, including the per-second failover timeline