fix(arc): run the role controller's probe only on voters and eligible hosts

README.md

@@ -197,7 +197,7 @@ Its image, `github_runner_arc_node_recovery_image`, is pulled without a pull Sec
 
 `fleet-role-probe` and `fleet-role` (an optional DaemonSet and Deployment in the platform namespace, see `roles/github_runner_arc/templates/fleet-role.yaml.j2`, `scripts/fleet-role-probe.sh` and `scripts/fleet-role.sh`; `github_runner_cluster_role_controller`, off by default, which `github_runner_arc_role_controller_enabled` takes as its default) measure which hosts are fit to hold an etcd vote and report the swap of an unfit voter for a fit standby that the controller would make. They only report: the pods' ClusterRoles can `get` and `list` nodes and nothing else, and the only objects they write are their own two ConfigMaps. The controller refuses to start with any setting other than `FLEET_ROLE_DRY_RUN=true`, and runs unchanged when no host is marked eligible, when it reports the voters' fitness alone.
 
-The probe runs on every Linux node that has a kubelet (a server regraded to etcd-only has none, and its side is read from its peers), on the node's own network, but records only on a voter or a host marked eligible (a node that is neither has no entry, and loses one it had), so the ConfigMap, which holds at most a mebibyte, grows with the hosts that decide the control plane and not with the fleet. Every `github_runner_arc_role_controller_sample_seconds` (15) it appends one sample to its node's key of the `fleet-role-probe` ConfigMap, which keeps the last window of them so that a replaced pod loses nothing. A sample holds the TCP connect time to every voter's etcd peer port (a connect that does not finish within the election timeout is recorded as the election timeout, so loss shows as tail latency); on a voter, what its etcd metrics (the `127.0.0.1:2381` listener) gained since the last sample: the WAL fsync and backend commit histograms, the round trip to each peer, leader changes, and whether it has a leader, is a learner and knows as many members as there are voters (a learner or a member that has not started shows as an extra known member); on a node that is not a voter, an fdatasync reading from fio with the write size etcd's own disk check uses, once per window; the memory pressure and `MemAvailable` the regrade core reads, with the core's own functions; and whether the interface the default route leaves by is `wired`, `wifi` or `unknown` (an interface that is virtual, such as the k3s container's veth, hides the link beneath it). The link is read from the interface's own `/sys/class/net` entry and never by walking `/sys`, whose symlinks loop. `scripts/fleet-role-probe.sh check link|connect <address>|memory|etcd|fsync` takes one of these readings alone, with no cluster access, to compare with a reading taken by hand on the host.
+The probe runs on every Linux node that is an etcd voter (k3s's `node-role.kubernetes.io/etcd` label) or is marked eligible and has a kubelet (a server regraded to etcd-only has none, and its side is read from its peers), on the node's own network, through a node affinity on those two labels, so a node that is neither (a Unraid host, an image-flashed agent) has no pod and nothing is asked of its kernel. A pod stopped on a node that has lost both labels, or is gone, removes the node's entry as it ends, so the ConfigMap, which holds at most a mebibyte, grows with the hosts that decide the control plane and not with the fleet. Every `github_runner_arc_role_controller_sample_seconds` (15) it appends one sample to its node's key of the `fleet-role-probe` ConfigMap, which keeps the last window of them so that a replaced pod loses nothing. A sample holds the TCP connect time to every voter's etcd peer port (a connect that does not finish within the election timeout is recorded as the election timeout, so loss shows as tail latency); on a voter, what its etcd metrics (the `127.0.0.1:2381` listener) gained since the last sample: the WAL fsync and backend commit histograms, the round trip to each peer, leader changes, and whether it has a leader, is a learner and knows as many members as there are voters (a learner or a member that has not started shows as an extra known member); on a node that is not a voter, an fdatasync reading from fio with the write size etcd's own disk check uses, once per window; the memory pressure (`/proc/pressure/memory` of the pod's own procfs, which serves the whole node's, so nothing is mounted) and `MemAvailable` the regrade core reads, with the core's own functions, where a node on a kernel without pressure stall information gives `MemAvailable` and `MemTotal` and no pressure figure, which the controller counts as not over the memory limit; and whether the interface the default route leaves by is `wired`, `wifi` or `unknown` (an interface that is virtual, such as the k3s container's veth, hides the link beneath it). The link is read from the interface's own `/sys/class/net` entry and never by walking `/sys`, whose symlinks loop. `scripts/fleet-role-probe.sh check link|connect <address>|memory|etcd|fsync` takes one of these readings alone, with no cluster access, to compare with a reading taken by hand on the host.
 
 The controller reads the Nodes and the probes' samples every sample period and publishes `fleet-role-status`, which the heartbeat republishes in its gist beside the autoscaler's status. For every voter and every eligible host it gives a verdict, `fit`, `unfit` or `unknown`, with each reason and the numbers behind it. Every limit derives from etcd's guidance and the cluster's own timing, not chosen numbers:
 

roles/github_runner_arc/README.md

@@ -481,9 +481,9 @@ A pruned node returns only when its k3s agent process restarts. A running k3s ag
 
 ## Role controller
 
-With `github_runner_cluster_role_controller: true` (or `github_runner_arc_role_controller_enabled: true`, for a host that does not run the cluster role) the role installs `fleet-role-probe`, a DaemonSet on every Linux node that has a kubelet, and `fleet-role`, a Deployment, both in `github_runner_arc_platform_namespace` and both from `github_runner_arc_role_controller_image`. Together they report, for every etcd voter and every host marked eligible, whether it is fit to hold an etcd vote and why, and the swap of an unfit voter for a fit standby that the controller would make. They change nothing in the cluster: each has a ClusterRole that can `get` and `list` nodes and nothing else, and a Role that reaches only the two ConfigMaps it uses by name (`get` and `patch`, never `create`; the role creates both empty and keeps them). `false` removes the workloads, their RBAC and both ConfigMaps. The repository README's Role controller section explains each signal, how it is read, the thresholds and when a swap is held back.
+With `github_runner_cluster_role_controller: true` (or `github_runner_arc_role_controller_enabled: true`, for a host that does not run the cluster role) the role installs `fleet-role-probe`, a DaemonSet on every Linux node that is an etcd voter or is marked eligible (`github_runner_arc_role_controller_eligible_label`) and has a kubelet, and `fleet-role`, a Deployment, both in `github_runner_arc_platform_namespace` and both from `github_runner_arc_role_controller_image`. Together they report, for every etcd voter and every host marked eligible, whether it is fit to hold an etcd vote and why, and the swap of an unfit voter for a fit standby that the controller would make. They change nothing in the cluster: each has a ClusterRole that can `get` and `list` nodes and nothing else, and a Role that reaches only the two ConfigMaps it uses by name (`get` and `patch`, never `create`; the role creates both empty and keeps them). `false` removes the workloads, their RBAC and both ConfigMaps. The repository README's Role controller section explains each signal, how it is read, the thresholds and when a swap is held back.
 
-The probe runs with the node's network and as root with every capability dropped, because the directory it measures the disk in (`github_runner_arc_role_controller_data_dir`, on the filesystem k3s keeps its data on) is created by the kubelet and owned by root. It mounts the host's `/proc/pressure` read-only. The controller runs as an unprivileged user with a read-only root filesystem. The heartbeat, when it is installed, reads the controller's status ConfigMap too and republishes it in its gist as `fleet-role-status.json`.
+The probe runs with the node's network and as root with every capability dropped, because the directory it measures the disk in (`github_runner_arc_role_controller_data_dir`, on the filesystem k3s keeps its data on) is created by the kubelet and owned by root. It mounts nothing for memory pressure: `/proc/pressure/memory` of its own procfs is the node's, as `/proc/meminfo` is, and a `hostPath` mount of `/proc/pressure` would stop the pod starting on a kernel without pressure stall information, since the container runtime then tries to create the directory. The controller runs as an unprivileged user with a read-only root filesystem. The heartbeat, when it is installed, reads the controller's status ConfigMap too and republishes it in its gist as `fleet-role-status.json`.
 
 The check in `tasks/validate.yml` refuses a target (`github_runner_arc_role_controller_voters`) that is even, under 3 or without a place of headroom below `github_runner_arc_role_controller_max_servers`, a period, election timeout or maximum that is not a whole number of at least 1, and two labels that are not label keys or are the same. Nothing is checked while the controller is off.
 

roles/github_runner_arc/tasks/install_psi_reporter.yml

@@ -1,5 +1,5 @@
 ---
-# The per-node pressure reporter (scripts/psi-reporter.sh) and what the autoscaler needs to read it, installed when github_runner_arc_autoscaler_node_pressure_enabled is on and a profile is autoscaled, removed otherwise. One DaemonSet pod on each node runner pods may use, with the host's /proc/pressure mounted read-only; each patches its own node's key of one shared ConfigMap. Like the autoscaler's own state ConfigMap, that one is pre-created empty, so no ServiceAccount here is granted `create`, only get/patch scoped by resourceNames.
+# The per-node pressure reporter (scripts/psi-reporter.sh) and what the autoscaler needs to read it, installed when github_runner_arc_autoscaler_node_pressure_enabled is on and a profile is autoscaled, removed otherwise. One DaemonSet pod on each node runner pods may use, reading /proc/pressure of its own procfs, with nothing mounted; each patches its own node's key of one shared ConfigMap. Like the autoscaler's own state ConfigMap, that one is pre-created empty, so no ServiceAccount here is granted `create`, only get/patch scoped by resourceNames.
 
 - name: "Platform: decide whether the node pressure reporter is wanted"
   ansible.builtin.set_fact:
@@ -156,10 +156,6 @@
                         value: "{{ github_runner_arc_node_pressure_configmap }}"
                       - name: REPORT_SECONDS
                         value: "{{ github_runner_arc_psi_reporter_seconds | string }}"
-                    volumeMounts:
-                      - name: pressure
-                        mountPath: /host-pressure
-                        readOnly: true
                     resources:
                       requests:
                         cpu: "10m"
@@ -169,7 +165,3 @@
                         cpu: "50m"
                         memory: "32Mi"
                         ephemeral-storage: "{{ github_runner_arc_platform_ephemeral_storage_limit }}"
-                volumes:
-                  - name: pressure
-                    hostPath:
-                      path: /proc/pressure

roles/github_runner_arc/templates/fleet-role.yaml.j2

@@ -5,6 +5,8 @@
 {% set namespace = github_runner_arc_platform_namespace %}
 {% set probe_configmap = github_runner_arc_role_controller_probe_configmap %}
 {% set status_configmap = github_runner_arc_role_controller_status_configmap %}
+{# The label k3s puts, with the value true, on every node that is an etcd member: the controller (scripts/fleet-role.sh) and the probe (scripts/fleet-role-probe.sh) each name it, and tests/unit/test_fleet_role_arc.py holds the three together. #}
+{% set voter_label = 'node-role.kubernetes.io/etcd' %}
 {# A ClusterRole can only name nodes cluster-wide; the controller and the probe both list them to find the voters and the standbys, and neither can write one. #}
 {% set node_rules = [{'apiGroups': [''], 'resources': ['nodes'], 'verbs': ['get', 'list']}] %}
 {% for name in [controller, probe] %}
@@ -175,7 +177,15 @@ spec:
       hostNetwork: true
       dnsPolicy: ClusterFirstWithHostNet
       nodeSelector: {{ {'kubernetes.io/os': 'linux'} | to_json }}
-      # Every node with a kubelet, whatever it is tainted for; a server regraded to etcd-only has no kubelet, so no pod, and its side is read from its peers.
+      # Only where the controller needs a reading: a node that is an etcd voter (the label k3s gives every member) or that the owner marked eligible (the label the controller reads to find its candidates). The terms of a node affinity are alternatives, so a node matching either gets a pod and any other gets none. A node that stops matching loses its pod, which removes the node's entry from the probe ConfigMap as it ends (scripts/fleet-role-probe.sh).
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+{% for label in [voter_label, github_runner_arc_role_controller_eligible_label] %}
+              - matchExpressions: {{ [{'key': label, 'operator': 'In', 'values': ['true']}] | to_json }}
+{% endfor %}
+      # Whatever such a node is tainted for; a server regraded to etcd-only has no kubelet, so no pod, and its side is read from its peers.
       tolerations:
         - operator: Exists
       securityContext:
@@ -196,6 +206,8 @@ spec:
               value: {{ github_runner_arc_role_controller_election_timeout_ms | string | to_json }}
             - name: ELIGIBLE_LABEL
               value: {{ github_runner_arc_role_controller_eligible_label | to_json }}
+            - name: VOTER_LABEL
+              value: {{ (voter_label ~ '=true') | to_json }}
             - name: PROBE_CONFIGMAP
               value: {{ probe_configmap | to_json }}
             - name: REPORT_SECONDS
@@ -224,19 +236,13 @@ spec:
           volumeMounts:
             - name: tmp
               mountPath: /tmp
-            - name: pressure
-              mountPath: /host-pressure
-              readOnly: true
             - name: fsync
               mountPath: /fsync
       volumes:
         - name: tmp
           emptyDir:
             medium: Memory
             sizeLimit: 8Mi
-        - name: pressure
-          hostPath:
-            path: /proc/pressure
         - name: fsync
           hostPath:
             path: {{ github_runner_arc_role_controller_data_dir | to_json }}