PAIR node on OpenShift

Run an NVIDIA Personal AI Router (PAIR) node as a pod on OpenShift, join it to an existing PAIR cluster on your LAN, and give every workload inside the cluster an OpenAI-compatible endpoint that routes to the GPUs on your other machines.

http://pair.pair.svc.cluster.local:1234/v1     # OpenAI-compatible
http://pair.pair.svc.cluster.local:11434       # Ollama-compatible

Verified against PAIR v0.1.1 on a single-node OpenShift 4.22 cluster (OVN-Kubernetes), clustered with a Linux GPU node and a macOS node. A five-minute in-cluster soak test ran 71 requests with no failures, GPU-bound, with negligible LAN traffic.

What is in here

Path	Purpose
Containerfile	Services-only PAIR node image: UBI 9 minimal plus the upstream service-binaries-linux-x64.zip, pinned by sha256.
entrypoint.sh	Starts the two relays, then hands the terminal to nvpair-tui, which supervises the node.
relay.py	Re-originates in-cluster TCP connections from loopback so PAIR's loopback-only proxy accepts them.
manifests/	Namespace, ServiceAccount, SCC binding, StatefulSet with a PVC, Service.
examples/load-test.yaml	A Job that soaks the endpoint from inside the cluster and prints latency and tokens/s.
.github/workflows/pair-image.yml	Builds the image on GitHub Actions and pushes it to ghcr.
docs/findings.md	Everything non-obvious learned on the way, so you do not have to rediscover it.
docs/upstreaming.md	What it would take to land this in the NVIDIA repository.

Why it is built this way

mDNS never leaves a local link. A pod on the cluster's software-defined network is not on the LAN's L2 segment, so it can neither hear the other PAIR nodes nor be heard by them. hostNetwork: true puts the pod on the LAN through the node's own interface. dnsPolicy: Default keeps the node's resolver, so LAN names still resolve.

PAIR's proxy ports are loopback-only for plaintext, deliberately: on a node running PAIR they are cluster front doors, and peers are meant to arrive over mTLS. In-cluster workloads are neither, and issuing each one a client certificate would hand out a cluster identity per consumer. relay.py listens on a routable port in the same network namespace and opens a fresh connection from 127.0.0.1, which the proxy accepts. It copies bytes and parses nothing, so streaming completions pass straight through. The relay is the trust boundary and enforces it itself: NetworkPolicy does not apply to host-network pods, so the relay refuses any source outside --allow-cidr (loopback and 10.128.0.0/14, the default OVN-Kubernetes cluster network; change it if yours differs). Deliberately no Route.

Identity must persist. node.crt, node.key and trusted/ live under $XDG_CONFIG_HOME/Nvidia Corporation/Personal AI Router (the path upstream's own wipe-app-data.sh uses). The StatefulSet's PVC keeps them; without it the node is a stranger on every restart and the other nodes keep listing the dead member.

nvpair-tui needs a terminal. It is a bubbletea program and exits without a PTY, so the container sets tty: true, and stdin: true as well because the six-digit pairing code has to be typed into it.

The image runs as an arbitrary UID. OpenShift assigns a UID from the namespace's range and guarantees only gid 0, so everything PAIR writes is group-writable and nothing depends on a fixed uid. The hostnetwork-v2 SCC is the narrowest grant that works; see the comment in manifests/00-namespace-sa-rbac.yaml for why it is -v2 and not hostnetwork.

Build the image

The workflow builds and pushes ghcr.io/<your-account>/pair-openshift:v0.1.1 on every push to main. To build locally instead:

podman build --platform linux/amd64 -t pair-node:v0.1.1 .

Point manifests/10-statefulset.yaml at wherever you push it, ideally by digest.

Deploy

The first step grants the hostnetwork-v2 SCC. That is a privilege grant and is a cluster-administrator action.

kubectl apply -f manifests/00-namespace-sa-rbac.yaml

# only if your image is in a private registry
kubectl -n pair create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io --docker-username=<user> --docker-password=<token>
kubectl -n pair patch serviceaccount pair -p '{"imagePullSecrets":[{"name":"ghcr-pull"}]}'

kubectl apply -f manifests/10-statefulset.yaml -f manifests/20-service.yaml
kubectl -n pair rollout status statefulset/pair

Join the PAIR cluster

Pairing needs a person at one of the existing nodes, because the six-digit code is shown on the inviting node and expires quickly.

1. On an existing node, open the Nodes tab and press i to invite the OpenShift node. If discovery does not list it, add it manually by the OpenShift node's LAN address (see docs/findings.md for why the address PAIR advertises may be the wrong one).
2. Attach to the pod's terminal interface and enter the code on the Cluster tab:

   kubectl -n pair attach -it pair-0

   Detach with Ctrl-P Ctrl-Q. Ctrl-C kills the TUI, and the TUI is the node.
3. Confirm any node's Cluster tab lists the new member.

Use it from a workload

kubectl -n pair run pair-probe --rm -it --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal -- \
  curl -sS http://pair.pair.svc.cluster.local:1234/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"<model-id>","messages":[{"role":"user","content":"reply with the word ok"}],"max_tokens":8}'

Use exactly the model id the cluster advertises at /v1/models. A completion coming back means the request left OpenShift, crossed the LAN over mTLS, and was served by a GPU node.

Before pairing, the same request returns 503 model inventory unavailable. That is correct: the node is up, but it has no engine and no cluster to route to.

Soak test

sed -i 's/CHANGE-ME/<model-id>/' examples/load-test.yaml
kubectl apply -f examples/load-test.yaml
kubectl -n pair logs -f job/pair-inference-loadtest

License

Apache-2.0. PAIR itself is Apache-2.0 and is downloaded at image build time from NVIDIA's GitHub releases; this repository redistributes none of it.