feat(kubernetes): add cni-sidecar supervisor topology - #2606
Draft
russellb wants to merge 13 commits into
Draft
Conversation
Add a cni-sidecar supervisor topology that keeps the sidecar runtime model but installs pod-network bypass-prevention rules through a privileged OpenShell chained CNI plugin instead of a pod-local network init container. A new openshell-cni crate provides the chained plugin and node installer. The Kubernetes driver emits openshell.ai/* pod annotations consumed by the plugin during CNI ADD, omits the network init container in this topology, and reuses the sidecar network-only process supervision path. Helm gains a privileged CNI installer DaemonSet gated on cni.enabled and a supervisor.topology=cni-sidecar option. Reconstructed on current main: uses the renamed topology config field and drops the abandoned proxy-pod topology. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
On OpenShift the binary-aware network sidecar must run as UID 0 with SYS_PTRACE and DAC_READ_SEARCH to inspect cross-UID /proc, which the restricted-v2 SCC forbids. Add a gated minimal SecurityContextConstraints (restricted-v2 baseline plus exactly those two capabilities and the image volume type) with a ClusterRole and ClusterRoleBinding granting it to the sandbox ServiceAccount. Controlled by sandboxServiceAccount.openshift.binaryAwareSCC (default false); the cni-sidecar OpenShift overlay enables it. Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Remediate merge-blocking review findings on the cni-sidecar supervisor topology: - driver: mount the gateway client mTLS bundle directly onto the network sidecar (cni-sidecar omits network-init, so the bundle was never seeded and the sidecar crash-looped on policy fetch) - driver: annotate the effective sidecar proxy UID (0 in binary-aware mode) so the node CNI exempts the right identity instead of redirecting the sidecar's own egress - cni: fail closed when ip6tables is missing in the iptables fallback so a dual-stack sandbox cannot bypass policy over IPv6 - cni: emit a CNI-spec error object on stdout so the runtime surfaces plugin failures instead of an opaque crash - helm: re-patch the chained plugin on the reconcile tick when absent so a CNI config rewrite or restart cannot drop egress enforcement - helm: keep the component label out of the immutable gateway selector so existing releases can upgrade - helm: mirror supervisorImage repository/tag fallbacks for the CNI image - docs: document the sidecar privilege profile and cold-start window; trim RFC non-goals Signed-off-by: Russell Bryant <rbryant@redhat.com>
Address the second-round review of the cni-sidecar topology: - cni/driver: add a per-node readiness scheduling gate. The installer labels its node openshell.ai/cni-ready via a new `openshell-cni node-ready` subcommand (minimal cluster-scoped nodes get/patch RBAC), clearing it on shutdown or when a reconcile tick cannot restore the plugin; the driver sets a required nodeAffinity on that label for cni-sidecar sandbox pods, so a pod cannot schedule before per-node egress enforcement is active. - driver: run the untrusted workspace-init container as the sandbox UID under binary-aware policy instead of root, so it cannot inherit the CNI's UID-0 egress exemption and bypass policy before the sidecar enforces it. - helm: gate the conflist/chain-conf mv on jq success and non-empty output so a jq failure cannot truncate the host CNI config and break node networking. - helm: give the CNI DaemonSet a distinct app name (openshell-cni) and revert the gateway Service selector to base-only, so a helm upgrade no longer blanks gateway endpoints for pods that predate the component label. - cni: only require ip6tables in the iptables fallback when the pod has a routable IPv6 address (from prevResult), so IPv4-only nodes are not rejected. - build: add crates/openshell-cni/BUILD.bazel so Bazel and the aggregate rustfmt suite cover the crate. - docs: document the scheduling gate and corrected sidecar privilege profile across architecture, published docs, crate READMEs, values, AGENTS, and the debug-openshell-cluster skill. Signed-off-by: Russell Bryant <rbryant@redhat.com>
russellb
changed the title
Address the third review round of the cni-sidecar topology: - cni/helm: fence before repair and stop failing open on teardown. The reconcile loop clears the readiness label the instant enforcement is not verifiably present, then repairs and re-marks. Enforcement lives in the host CNI config and survives pod restarts, so preStop no longer strips it on an ordinary restart/rolling update: a new `openshell-cni daemonset-active` check removes it only when the owning DaemonSet is confirmed terminating, and fences the node first. Reconcile interval 300s -> 30s. - cni/helm: stamp an owner (<namespace>/<release>) on the chained plugin entry and refuse to overwrite an entry owned by a different release (fail closed). Only one OpenShell release per cluster is supported for cni-sidecar today; documented in the RFC. - helm: validate the multus chain conf content (type, owner, kubeconfig, non-empty namespaces) before marking the node ready, instead of accepting any existing file. - driver: harden the untrusted workspace-init container (allowPrivilegeEscalation=false, drop ALL capabilities, runAsNonRoot when non-root) so a setuid binary cannot regain the exempt UID 0; restrict the non-root init to the Sidecar/CniSidecar topologies so combined keeps root. - cni: determine IPv6 enforcement by probing /proc/net/if_inet6 in the pod netns (fail closed on indeterminate; link-local counts) instead of trusting prevResult, keeping IPv4-only nodes working without ip6tables. - build: stage openshell-cni in the prebuilt 'all' target. - rbac: add get on the installer's own DaemonSet (resourceNames-scoped). Signed-off-by: Russell Bryant <rbryant@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
cni-sidecar, a third Kubernetes supervisor topology for OpenShell sandboxpods. It keeps the split-supervisor model of the
sidecartopology but movespod-network rule installation out of the sandbox pod: a privileged, node-level
OpenShell CNI DaemonSet installs a chained CNI plugin that programs the
bypass-prevention rules during CNI
ADD, removing the per-pod privileged networkinit container. Includes OpenShift/Multus enablement (multus-chain install mode,
purpose-built SCCs) and a per-node readiness scheduling gate.
Related Issue
Related: #899 (restricted SCC support for
managed Kubernetes). Design is captured in the in-branch RFC
(
rfc/cni-sidecar-topology-DRAFT.md).Changes
cni-sidecarSupervisorTopologyin the Kubernetes driver;omits the in-pod network-init container and annotates sandbox pods for the
chained plugin.
openshell-cni): chained plugin that installs nftables/iptablesbypass-prevention rules in the pod netns during CNI
ADD; fail-closed on jqand IPv6-enforcement errors; structured CNI-spec error objects on stdout.
openshell-cni node-readysubcommand labels thenode
openshell.ai/cni-ready(minimal cluster-scopednodes get;patchRBAC),cleared on shutdown/repair-failure; the driver sets a required
nodeAffinityso sandbox pods cannot schedule before per-node enforcement is active.
workspace-initcontainer runs as the sandboxUID (non-exempt) under binary-aware policy, so it cannot inherit the CNI's UID-0
egress exemption.
the gateway Service selector stays base-only (no upgrade endpoint gap);
OpenShift
multus-chainmode and cni-sidecar values overlay; minimalbinary-aware sandbox SCC.
compute-runtimes.md), published docs(
docs/kubernetes/topology.mdx), crate READMEs, and thedebug-openshell-clusterskill.Testing
Platform coverage so far: on-cluster E2E has been performed on OpenShift
(Multus / OVN-Kubernetes) using
multus-chainmode only. The defaultconflistmode (vanilla Kubernetes / k3s) is covered by unit tests but has notyet been validated on a live cluster.
mise run pre-commitpassestests) — covers both
conflistandmultus-chainpathsmultus-chainonly: node labeled
cni-ready, sandbox pod scheduled 2/2 with the requirednodeAffinity,
workspace-initrunning as the sandbox UID, sidecarauthenticated to the gateway, and default-deny egress enforced.
conflistmode) — not yet doneChecklist