From 68149c059ed573f1633df1a1814df298db7f529a Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Wed, 4 Nov 2026 09:10:00 +0000 Subject: [PATCH 01/12] refactor(cluster): copy a node's identity between containers and volumes with one script The node password seeding pipe moves into files/node-identity.sh, which copies a directory of a running or stopped node container into a named volume through a throwaway container that is never started, so the role actuator can move a Docker node's password and Tailscale state between its server and agent services the same way. With --stopped it refuses while either side could change the files. --- .../files/node-identity.sh | 53 +++++++++++++++++++ .../tasks/node_password.yml | 34 +++--------- tests/unit/test_node_password.py | 16 +++--- 3 files changed, 68 insertions(+), 35 deletions(-) create mode 100644 roles/github_runner_cluster/files/node-identity.sh diff --git a/roles/github_runner_cluster/files/node-identity.sh b/roles/github_runner_cluster/files/node-identity.sh new file mode 100644 index 00000000..eaad841e --- /dev/null +++ b/roles/github_runner_cluster/files/node-identity.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash +# node-identity.sh [--stopped] CONTAINER PATH VOLUME PROJECT VOLUME_KEY: copies what the directory PATH holds in the node container CONTAINER (running or stopped) into the named volume VOLUME, for the cluster role's seeding of a node's password volume (tasks/node_password.yml) and for the role actuator's switch of a Docker host between its server and agent services (files/role/fleet-role-actuator-docker.sh), which moves the node's password (/etc/rancher/node) and its Tailscale state (/var/lib/tailscale) from the volumes of one service to the other's, so the node keeps its name in the cluster and its device on the tailnet. +# +# The files go from one docker cp straight into another, so they pass through a pipe between the two commands and are on no disk of the host, in no command line and in no output. The throwaway container that receives them is created from CONTAINER's own image and never started: docker cp writes through a created container's mounts, so the copy lands in the volume with the ownership and mode the tar stream carries, on any engine (Linux, Docker Desktop, Colima). A volume this script creates carries the labels Compose puts on the volumes it creates (com.docker.compose.project=PROJECT, com.docker.compose.volume=VOLUME_KEY), so Compose adopts it without warning, and is removed again when the copy fails, so a later attempt starts from the same state. A volume that exists already keeps its other files: the copy replaces each file CONTAINER holds at PATH, which for a node password and a Tailscale state is the whole identity. +# +# --stopped refuses, with nothing changed, while CONTAINER runs or any running container mounts VOLUME, so that neither side of a switch can write its identity while it is copied. +# +# Exit status: 0 copied, 1 the copy failed (docker's own message on standard error), 2 refused or bad usage. +set -euo pipefail + +stopped=0 +if [ "${1:-}" = --stopped ]; then + stopped=1 + shift +fi +if [ $# -ne 5 ]; then + echo "usage: node-identity.sh [--stopped] CONTAINER PATH VOLUME PROJECT VOLUME_KEY" >&2 + exit 2 +fi +container=$1 path=$2 volume=$3 project=$4 volume_key=$5 + +if [ "$stopped" = 1 ]; then + if [ "$(docker ps --all --filter "name=^/${container}\$" --format '{{.State}}')" = running ]; then + echo "node-identity: ${container} is running, so its files at ${path} could change while they are copied" >&2 + exit 2 + fi + users=$(docker ps --filter "volume=${volume}" --format '{{.Names}}') + if [ -n "$users" ]; then + echo "node-identity: ${volume} is mounted by the running container ${users//$'\n'/, }, which could change it while it is written" >&2 + exit 2 + fi +fi + +copy="${volume}-identity-copy" +created=0 +cleanup() { + local rc=$? + docker rm --force "$copy" > /dev/null 2>&1 || true + if [ "$rc" -ne 0 ] && [ "$created" = 1 ]; then + docker volume rm "$volume" > /dev/null 2>&1 || true + fi + exit "$rc" +} +trap cleanup EXIT + +docker rm --force "$copy" > /dev/null 2>&1 || true +image=$(docker inspect --format '{{.Image}}' "$container") +if ! docker volume ls --quiet --filter "name=${volume}" | grep -Fx -- "$volume" > /dev/null; then + docker volume create --label "com.docker.compose.project=${project}" --label "com.docker.compose.volume=${volume_key}" "$volume" > /dev/null + created=1 +fi +docker create --name "$copy" --volume "${volume}:${path}" "$image" > /dev/null +docker cp --archive "${container}:${path}/." - | docker cp --archive - "${copy}:${path}" diff --git a/roles/github_runner_cluster/tasks/node_password.yml b/roles/github_runner_cluster/tasks/node_password.yml index 344576a2..77d4232f 100644 --- a/roles/github_runner_cluster/tasks/node_password.yml +++ b/roles/github_runner_cluster/tasks/node_password.yml @@ -1,7 +1,7 @@ --- # Seeds the named volume that keeps this node's password (/etc/rancher/node, see docker-compose.yml) from the container that runs now, so that a recreated container keeps the password the cluster holds a Secret for, instead of generating a new one that the cluster rejects. Nothing to do on a host with no container yet (the first container then writes its password into the volume as it is made) or whose volume exists already. Goes before anything that can recreate the container: the mesh step and the Compose tasks below it. # -# The password is copied from one docker cp straight into another, so it passes through a pipe between the two commands and is on no disk of this host, in no command line and in no output. The throwaway container that receives it is created from the node's own image and never started: docker cp writes through a created container's mounts, so the copy lands in the volume with the ownership and mode (root, 0600) the tar stream carries, on any engine (Linux, Docker Desktop, Colima). The volume carries the labels Compose puts on the volumes it creates, so Compose adopts it without warning. A failed copy removes both again, so the next run starts from the same state. Under --check nothing is created or copied; the run says it would. +# The copy is files/node-identity.sh's, which the role actuator also moves a node's identity between a Docker host's services with: from one docker cp straight into another, so the password is on no disk of this host, in no command line and in no output, into a throwaway container created from the node's own image and never started, so the volume gets the ownership and mode (root, 0600) the tar stream carries, on any engine. The volume carries the labels Compose puts on the volumes it creates, so Compose adopts it without warning. A failed copy removes both again, so the next run starts from the same state. Under --check nothing is created or copied; the run says it would. - name: Read whether this node's password volume exists and whether its container does # Read-only, and every task after it depends on the result, so it runs under --check too. docker ps rather than docker inspect, so that a container that does not exist reads as absent while a daemon that cannot be reached fails the run. @@ -39,33 +39,11 @@ - ansible_check_mode - name: Seed the password volume from the running container - # The password never leaves the two docker cp processes, but the task stays no_log, so that nothing it ever prints can reach the log; the failure below reports the exit status and docker's own message, which carry no file content. - ansible.builtin.shell: | - set -eu - set -o pipefail - seed="${container}-password-seed" - created=0 - cleanup() { - rc=$? - docker rm --force "$seed" >/dev/null 2>&1 || true - if [ "$rc" -ne 0 ] && [ "$created" = 1 ]; then - docker volume rm "$volume" >/dev/null 2>&1 || true - fi - exit "$rc" - } - trap cleanup EXIT - docker rm --force "$seed" >/dev/null 2>&1 || true - image=$(docker inspect --format '{{ '{{' }}.Image{{ '}}' }}' "$container") - docker volume create --label "com.docker.compose.project=${project}" --label "com.docker.compose.volume=${volume_key}" "$volume" >/dev/null - created=1 - docker create --name "$seed" --volume "${volume}:/etc/rancher/node" "$image" >/dev/null - docker cp --archive "${container}:/etc/rancher/node/." - | docker cp --archive - "${seed}:/etc/rancher/node" - args: - executable: /bin/bash - environment: >- - {{ github_runner_cluster_docker_cli_environment - | combine({'volume': github_runner_cluster_node_password_volume, 'volume_key': github_runner_cluster_node_password_volume_key, - 'project': github_runner_cluster_compose_project, 'container': github_runner_cluster_node_container}) }} + # files/node-identity.sh makes the copy (see the comment at the top of this file). The password never leaves its two docker cp processes, but the task stays no_log, so that nothing it ever prints can reach the log; the failure below reports the exit status and docker's own message, which carry no file content. + ansible.builtin.script: + cmd: "node-identity.sh {{ [github_runner_cluster_node_container, '/etc/rancher/node', github_runner_cluster_node_password_volume, github_runner_cluster_compose_project, github_runner_cluster_node_password_volume_key] | map('quote') | join(' ') }}" + executable: bash + environment: "{{ github_runner_cluster_docker_cli_environment }}" register: github_runner_cluster_password_seeded changed_when: github_runner_cluster_password_seeded.rc == 0 failed_when: false diff --git a/tests/unit/test_node_password.py b/tests/unit/test_node_password.py index 3d4c6ef4..dd5d4fb1 100644 --- a/tests/unit/test_node_password.py +++ b/tests/unit/test_node_password.py @@ -88,8 +88,8 @@ def test_the_throwaway_container_is_created_from_the_nodes_own_image_and_removed self.container("node1-k3s", password=self.SECRET, image="sha256:feed") self.assertEqual(self.seed().returncode, 0) creates = [call for call in self.calls() if call.startswith("- create ")] - self.assertEqual(creates, [f"- create --name node1-k3s-password-seed --volume {self.VOLUME}:/etc/rancher/node sha256:feed"]) - self.assertFalse((self.stub / "containers" / "node1-k3s-password-seed.state").exists()) + self.assertEqual(creates, [f"- create --name {self.VOLUME}-identity-copy --volume {self.VOLUME}:/etc/rancher/node sha256:feed"]) + self.assertFalse((self.stub / "containers" / f"{self.VOLUME}-identity-copy.state").exists()) def test_a_second_run_changes_nothing(self) -> None: self.container("node1-k3s", password=self.SECRET) @@ -163,7 +163,7 @@ def test_a_failed_copy_fails_the_run_keeps_no_volume_and_leaks_nothing(self) -> self.assertIn("Error response from daemon: Could not find the file /etc/rancher/node in container node1-k3s", result.stdout) self.assertIn("docker volume create", result.stdout) self.assertFalse((self.stub / "volumes" / self.VOLUME).exists(), "a half-seeded volume would stop every later run from seeding") - self.assertFalse((self.stub / "containers" / "node1-k3s-password-seed.state").exists()) + self.assertFalse((self.stub / "containers" / f"{self.VOLUME}-identity-copy.state").exists()) self.nothing_on_the_host_holds_the_secret(result) def test_the_run_after_a_failed_copy_seeds_again(self) -> None: @@ -176,7 +176,7 @@ def test_the_run_after_a_failed_copy_seeds_again(self) -> None: def test_a_throwaway_container_left_by_an_earlier_run_is_replaced(self) -> None: self.container("node1-k3s", password=self.SECRET) - self.write("containers", "node1-k3s-password-seed.state", text="created") + self.write("containers", f"{self.VOLUME}-identity-copy.state", text="created") result = self.seed() self.assertEqual(result.returncode, 0, result.stdout + result.stderr) self.assertEqual((self.stub / "volumes" / self.VOLUME / "password").read_text(), self.SECRET) @@ -207,9 +207,11 @@ def test_the_seeding_task_that_handles_the_password_never_prints_it(self) -> Non seed = next(task for task in role_tasks("node_password.yml") if task.get("name") == "Seed the password volume from the running container") self.assertIs(seed["no_log"], True) self.assertIn("ansible_check_mode", " ".join(seed["when"])) - script = seed["ansible.builtin.shell"] - self.assertNotIn(">", re.sub(r">/dev/null( 2>&1)?", "", script).replace("2>&1", ""), "the copy must not be redirected to a file") - self.assertIn("set -o pipefail", script) + self.assertTrue(seed["ansible.builtin.script"]["cmd"].startswith("node-identity.sh ")) + # The script the task runs: its only redirections throw output away, so the copy is never written to a file, and a failed docker cp on either side of the pipe fails it. + script = (ROLE / "files" / "node-identity.sh").read_text() + self.assertNotIn(">", re.sub(r"> /dev/null( 2>&1)?", "", script).replace(">&2", "").replace("2>&1", ""), "the copy must not be redirected to a file") + self.assertIn("set -euo pipefail", script) if __name__ == "__main__": From 2f032a36472bf85ca6b9104a2acc8c6cd84aa56e Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Wed, 4 Nov 2026 09:25:56 +0000 Subject: [PATCH 02/12] refactor(cluster): set up the role actuator's credential from task files any platform can include Choosing the server whose API applies the credential, and applying it, move out of the native watcher's tasks into role_actuator_api.yml and role_actuator_credential.yml, so a Docker host's watcher can be given the same credential the same way. --- .../tasks/role_actuator_api.yml | 14 +++ .../tasks/role_actuator_credential.yml | 79 ++++++++++++++++ .../tasks/role_actuator_watcher.yml | 93 +------------------ 3 files changed, 98 insertions(+), 88 deletions(-) create mode 100644 roles/github_runner_cluster/tasks/role_actuator_api.yml create mode 100644 roles/github_runner_cluster/tasks/role_actuator_credential.yml diff --git a/roles/github_runner_cluster/tasks/role_actuator_api.yml b/roles/github_runner_cluster/tasks/role_actuator_api.yml new file mode 100644 index 00000000..174267ae --- /dev/null +++ b/roles/github_runner_cluster/tasks/role_actuator_api.yml @@ -0,0 +1,14 @@ +--- +# Chooses the other server of the inventory through whose API the role sets up the role actuator's credential (native/api.yml, which reaches a Docker server through its container as it does a native one through its k3s), since this host's own may be the one a swap stops, and fails, before anything is changed, when the inventory names none. For tasks/role_actuator_watcher.yml and tasks/role_actuator_docker.yml, which run it first; read-only, so it runs under --check too. + +- name: Reach the cluster through another server, since this host's own may be the one a swap stops + ansible.builtin.include_tasks: native/api.yml + vars: + github_runner_cluster_native_api_self: false + +- name: Fail if no other server of the inventory can apply the watcher's credential + ansible.builtin.fail: + msg: >- + github_runner_cluster_role_controller is true, but the inventory names no server other than {{ inventory_hostname }} to apply the + role actuator's credential through. Nothing was changed. + when: github_runner_cluster_native_api_host | length == 0 diff --git a/roles/github_runner_cluster/tasks/role_actuator_credential.yml b/roles/github_runner_cluster/tasks/role_actuator_credential.yml new file mode 100644 index 00000000..ea0a7602 --- /dev/null +++ b/roles/github_runner_cluster/tasks/role_actuator_credential.yml @@ -0,0 +1,79 @@ +--- +# Gives the role actuator's watcher its credential, for tasks/role_actuator_watcher.yml (a native host) and tasks/role_actuator_docker.yml (a Docker host), each of which includes it only on a host whose watcher runs and outside --check: applies its ServiceAccount, its scoped rules and its token Secret (templates/role-actuator-rbac.yaml.j2) through the other server's API that role_actuator_api.yml, which the caller runs first, chose, creates the report ConfigMap the controller reads when there is none, writes a kubeconfig holding the token, mode 0600, by a task that logs nothing, and a file mapping every other server and eligible host's node name to the URL a promotion joins it at. A native host's files are root's; a Docker host's are the connecting user's, in its Compose project, which the role-actuator service's container mounts. Registers github_runner_cluster_role_actuator_kubeconfig_file and github_runner_cluster_role_actuator_join_urls for the caller's restart decision. + +- name: Apply the watcher's ServiceAccount, its rules and its token Secret + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + # docker exec passes its standard input to kubectl only with --interactive, as tasks/colima_size_turn.yml gives it; k3s kubectl reads its own. + argv: >- + {{ ((github_runner_cluster_native_api_argv[:2] + ['--interactive'] + github_runner_cluster_native_api_argv[2:]) + if github_runner_cluster_native_api_argv[0] == 'docker' else github_runner_cluster_native_api_argv) + + ['apply', '--filename', '-'] }} + stdin: "{{ lookup('ansible.builtin.template', 'role-actuator-rbac.yaml.j2') }}" + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_applied + changed_when: github_runner_cluster_role_actuator_applied.stdout_lines | reject('search', ' unchanged$') | list | length > 0 + +- name: Create the report ConfigMap the controller reads, when there is none + # Created rather than applied, since applying an empty one again would wipe the reports. + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + argv: >- + {{ github_runner_cluster_native_api_argv + ['create', 'configmap', github_runner_cluster_role_actuator_configmap, + '--namespace', github_runner_cluster_role_actuator_namespace] }} + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_configmap_created + changed_when: github_runner_cluster_role_actuator_configmap_created.rc == 0 + failed_when: + - github_runner_cluster_role_actuator_configmap_created.rc != 0 + - "'AlreadyExists' not in github_runner_cluster_role_actuator_configmap_created.stderr" + +- name: Read the watcher's token and the cluster's CA, once the token controller has filled them + # Both base64, as the Secret holds them; the token is a secret, so nothing of this task is logged. + become: "{{ github_runner_cluster_native_api_become }}" + ansible.builtin.command: + argv: >- + {{ github_runner_cluster_native_api_argv + ['get', 'secret', github_runner_cluster_role_actuator_name ~ '-token', + '--namespace', github_runner_cluster_role_actuator_namespace, '--request-timeout=' ~ github_runner_cluster_cluster_read_timeout ~ 's', + '--output', 'jsonpath={.data.token} {.data.ca\.crt}'] }} + environment: "{{ github_runner_cluster_native_api_environment }}" + delegate_to: "{{ github_runner_cluster_native_api_host }}" + register: github_runner_cluster_role_actuator_secret + until: + - github_runner_cluster_role_actuator_secret.rc == 0 + - github_runner_cluster_role_actuator_secret.stdout | trim | split(' ') | length == 2 + retries: "{{ ((github_runner_cluster_cluster_read_timeout | int) / github_runner_cluster_native_poll) | round(0, 'ceil') | int }}" + delay: "{{ github_runner_cluster_native_poll }}" + vars: + # Seconds between reads, the pace every native wait polls at (tasks/native/server_ready.yml). + github_runner_cluster_native_poll: 5 + changed_when: false + no_log: true + +- name: Write the watcher's kubeconfig + # Its server is the first API candidate; the watcher passes the one that answers (KUBE_SERVERS in its settings) on every call. + become: "{{ github_runner_cluster_node_mode == 'native' }}" + ansible.builtin.copy: + content: >- + {{ {'apiVersion': 'v1', 'kind': 'Config', 'current-context': 'fleet', + 'clusters': [{'name': 'fleet', 'cluster': {'server': 'https://' ~ (github_runner_cluster_api_candidates | first).value.address ~ ':' ~ github_runner_cluster_api_port, + 'certificate-authority-data': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[1]}}], + 'users': [{'name': github_runner_cluster_role_actuator_name, 'user': {'token': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[0] | b64decode}}], + 'contexts': [{'name': 'fleet', 'context': {'cluster': 'fleet', 'user': github_runner_cluster_role_actuator_name}}]} | to_nice_yaml }} + dest: "{{ github_runner_cluster_role_actuator_kubeconfig }}" + mode: "0600" + no_log: true + register: github_runner_cluster_role_actuator_kubeconfig_file + +- name: Write the URL a promotion joins each other server and eligible host at + become: "{{ github_runner_cluster_node_mode == 'native' }}" + ansible.builtin.copy: + content: | + {% for candidate in github_runner_cluster_api_candidates %} + {{ candidate.value.node_name }} https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }} + {% endfor %} + dest: "{{ github_runner_cluster_role_actuator_dir }}/join-urls" + mode: "0644" + register: github_runner_cluster_role_actuator_join_urls diff --git a/roles/github_runner_cluster/tasks/role_actuator_watcher.yml b/roles/github_runner_cluster/tasks/role_actuator_watcher.yml index bef32517..b6126d31 100644 --- a/roles/github_runner_cluster/tasks/role_actuator_watcher.yml +++ b/roles/github_runner_cluster/tasks/role_actuator_watcher.yml @@ -1,5 +1,5 @@ --- -# Runs the role actuator's watcher on a native host the role controller may switch, for tasks/role_actuator.yml, which includes this file only on such a host: an include skipped by its condition loads none of its tasks, whereas a task in a block skipped by its condition still has its delegate_to templated (ansible-core resolves delegate_to before when, _calculate_delegate_to in TaskExecutor._execute), and the delegated tasks here name github_runner_cluster_native_api_host, which only native/api.yml below sets. Installs the commands the watcher needs, applies its credential through another server's API, writes its kubeconfig, join URLs and unit, and starts or restarts it. Under --check nothing is changed and each change a real run would make is reported. +# Runs the role actuator's watcher on a native host the role controller may switch, for tasks/role_actuator.yml, which includes this file only on such a host: an include skipped by its condition loads none of its tasks, whereas a task in a block skipped by its condition still has its delegate_to templated (ansible-core resolves delegate_to before when, _calculate_delegate_to in TaskExecutor._execute), and the delegated tasks of role_actuator_credential.yml name github_runner_cluster_native_api_host, which only role_actuator_api.yml sets. Installs the commands the watcher needs, applies its credential through another server's API, writes its kubeconfig, join URLs and unit, and starts or restarts it. Under --check nothing is changed and each change a real run would make is reported. - name: Read which of the commands the watcher needs are installed # Read-only, so it runs under --check too. @@ -50,17 +50,8 @@ DEBIAN_FRONTEND: noninteractive changed_when: true -- name: Reach the cluster through another server, since this host's own may be the one a swap stops - ansible.builtin.include_tasks: native/api.yml - vars: - github_runner_cluster_native_api_self: false - -- name: Fail if no other server of the inventory can apply the watcher's credential - ansible.builtin.fail: - msg: >- - github_runner_cluster_role_controller is true, but the inventory names no server other than {{ inventory_hostname }} to apply the - role actuator's credential through. Nothing was changed. - when: github_runner_cluster_native_api_host | length == 0 +- name: Choose the server whose API sets up the watcher's credential + ansible.builtin.include_tasks: role_actuator_api.yml - name: Write the watcher's unit # Under --check too, as the regrade unit is (native/regrade_service.yml), so a check run reports a changed unit. @@ -91,82 +82,8 @@ - name: Set up the watcher's credential and run it when: not ansible_check_mode block: - - name: Apply the watcher's ServiceAccount, its rules and its token Secret - become: "{{ github_runner_cluster_native_api_become }}" - ansible.builtin.command: - # docker exec passes its standard input to kubectl only with --interactive, as tasks/colima_size_turn.yml gives it; k3s kubectl reads its own. - argv: >- - {{ ((github_runner_cluster_native_api_argv[:2] + ['--interactive'] + github_runner_cluster_native_api_argv[2:]) - if github_runner_cluster_native_api_argv[0] == 'docker' else github_runner_cluster_native_api_argv) - + ['apply', '--filename', '-'] }} - stdin: "{{ lookup('ansible.builtin.template', 'role-actuator-rbac.yaml.j2') }}" - environment: "{{ github_runner_cluster_native_api_environment }}" - delegate_to: "{{ github_runner_cluster_native_api_host }}" - register: github_runner_cluster_role_actuator_applied - changed_when: github_runner_cluster_role_actuator_applied.stdout_lines | reject('search', ' unchanged$') | list | length > 0 - - - name: Create the report ConfigMap the controller reads, when there is none - # Created rather than applied, since applying an empty one again would wipe the reports. - become: "{{ github_runner_cluster_native_api_become }}" - ansible.builtin.command: - argv: >- - {{ github_runner_cluster_native_api_argv + ['create', 'configmap', github_runner_cluster_role_actuator_configmap, - '--namespace', github_runner_cluster_role_actuator_namespace] }} - environment: "{{ github_runner_cluster_native_api_environment }}" - delegate_to: "{{ github_runner_cluster_native_api_host }}" - register: github_runner_cluster_role_actuator_configmap_created - changed_when: github_runner_cluster_role_actuator_configmap_created.rc == 0 - failed_when: - - github_runner_cluster_role_actuator_configmap_created.rc != 0 - - "'AlreadyExists' not in github_runner_cluster_role_actuator_configmap_created.stderr" - - - name: Read the watcher's token and the cluster's CA, once the token controller has filled them - # Both base64, as the Secret holds them; the token is a secret, so nothing of this task is logged. - become: "{{ github_runner_cluster_native_api_become }}" - ansible.builtin.command: - argv: >- - {{ github_runner_cluster_native_api_argv + ['get', 'secret', github_runner_cluster_role_actuator_name ~ '-token', - '--namespace', github_runner_cluster_role_actuator_namespace, '--request-timeout=' ~ github_runner_cluster_cluster_read_timeout ~ 's', - '--output', 'jsonpath={.data.token} {.data.ca\.crt}'] }} - environment: "{{ github_runner_cluster_native_api_environment }}" - delegate_to: "{{ github_runner_cluster_native_api_host }}" - register: github_runner_cluster_role_actuator_secret - until: - - github_runner_cluster_role_actuator_secret.rc == 0 - - github_runner_cluster_role_actuator_secret.stdout | trim | split(' ') | length == 2 - retries: "{{ ((github_runner_cluster_cluster_read_timeout | int) / github_runner_cluster_native_poll) | round(0, 'ceil') | int }}" - delay: "{{ github_runner_cluster_native_poll }}" - vars: - # Seconds between reads, the pace every native wait polls at (tasks/native/server_ready.yml). - github_runner_cluster_native_poll: 5 - changed_when: false - no_log: true - - - name: Write the watcher's kubeconfig - # Its server is the first API candidate; the watcher passes the one that answers (KUBE_SERVERS in its settings) on every call. - become: true - ansible.builtin.copy: - content: >- - {{ {'apiVersion': 'v1', 'kind': 'Config', 'current-context': 'fleet', - 'clusters': [{'name': 'fleet', 'cluster': {'server': 'https://' ~ (github_runner_cluster_api_candidates | first).value.address ~ ':' ~ github_runner_cluster_api_port, - 'certificate-authority-data': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[1]}}], - 'users': [{'name': github_runner_cluster_role_actuator_name, 'user': {'token': (github_runner_cluster_role_actuator_secret.stdout | trim | split(' '))[0] | b64decode}}], - 'contexts': [{'name': 'fleet', 'context': {'cluster': 'fleet', 'user': github_runner_cluster_role_actuator_name}}]} | to_nice_yaml }} - dest: "{{ github_runner_cluster_role_actuator_kubeconfig }}" - mode: "0600" - no_log: true - register: github_runner_cluster_role_actuator_kubeconfig_file - - - name: Write the URL a promotion joins each other server and eligible host at - become: true - ansible.builtin.copy: - content: | - {% for candidate in github_runner_cluster_api_candidates %} - {{ candidate.value.node_name }} https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }} - {% endfor %} - dest: "{{ github_runner_cluster_role_actuator_dir }}/join-urls" - mode: "0644" - register: github_runner_cluster_role_actuator_join_urls + - name: Give the watcher its credential and the URLs a promotion joins + ansible.builtin.include_tasks: role_actuator_credential.yml - name: Read whether the watcher is enabled and running ansible.builtin.command: From 84fcac864274e58c03cd8ab059d9b56b083d6086 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Wed, 4 Nov 2026 09:26:09 +0000 Subject: [PATCH 03/12] feat(cluster): switch a Docker host's role from the same actuator The role actuator gains a Docker platform, so the role controller's voter swap can move a Docker host between its k3s and k3s-agent services as it moves a native host between its units. On a server or eligible Docker host with the controller on, the role runs it as the role-actuator service of the host's Compose project, from its own Compose file, with the Docker socket, the project directory at its own path and the server's k3s-data volume. A switch writes the role directory the entrypoint reads, stops one service, copies the node password and the Tailscale state from its stopped container into the other service's volumes through node-identity.sh, and starts the other service, so the node keeps its name, password and tailnet address. A step-down restarts the server at etcd-only through regrading's Docker actuator. kubectl runs in whichever node container is up, since the actuator's own container is on no tailnet, and a server that does not come up brings the agent back at once, so the host never loses its way to the API. The regrade service now sees the role directory and idles while it names another role than the server. --- roles/github_runner_cluster/README.md | 15 +- roles/github_runner_cluster/defaults/main.yml | 6 +- .../files/docker-compose.fleet-role.yml | 5 +- .../files/regrade/fleet-regrade.sh | 17 + .../files/role/Dockerfile | 4 + .../files/role/fleet-role-actuator-docker.sh | 201 ++++++++++ .../files/role/fleet-role-actuator.sh | 6 +- roles/github_runner_cluster/tasks/main.yml | 6 + .../tasks/role_actuator.yml | 49 ++- .../tasks/role_actuator_docker.yml | 71 ++++ .../docker-compose.role-actuator.yml.j2 | 16 + .../templates/role-actuator.env.j2 | 22 +- tests/role_actuator/fake-docker.py | 259 ++++++++++++ tests/unit/test_node_entrypoint.py | 3 +- tests/unit/test_role_actuator.py | 376 +++++++++++++++++- 15 files changed, 1036 insertions(+), 20 deletions(-) create mode 100644 roles/github_runner_cluster/files/role/Dockerfile create mode 100644 roles/github_runner_cluster/files/role/fleet-role-actuator-docker.sh create mode 100644 roles/github_runner_cluster/tasks/role_actuator_docker.yml create mode 100644 roles/github_runner_cluster/templates/docker-compose.role-actuator.yml.j2 create mode 100755 tests/role_actuator/fake-docker.py diff --git a/roles/github_runner_cluster/README.md b/roles/github_runner_cluster/README.md index 7cf1e69d..939750e0 100644 --- a/roles/github_runner_cluster/README.md +++ b/roles/github_runner_cluster/README.md @@ -337,7 +337,7 @@ A server never bootstraps a second cluster from a former member's host: with no ### Role actuator -On a native host the controller may switch or take a snapshot on, a server or a host marked `github_runner_cluster_voter_eligible`, the role runs the role actuator's watcher while `github_runner_cluster_role_controller` is true: the `fleet-role-actuator` unit, which runs `files/role/fleet-role-actuator.sh watch` as root from `/usr/local/lib/github-runner-k3s/role` (`tasks/role_actuator.yml`). It switches the host between its server and agent units when the controller asks through the host's Node, and does nothing on the API but read: the controller cordons, waits for runner pods, drains, taints, labels the level, removes the etcd member, cleans the Node's labels and annotations and uncordons, because an agent host has no credential that can evict pods and its own k3s is down while it switches. Turning the controller off, or a host's eligibility, stops and removes the unit, its kubeconfig and its join URLs; the scripts stay on every native host, since the role's own promotion and demotion run them. +On a native host the controller may switch or take a snapshot on, a server or a host marked `github_runner_cluster_voter_eligible`, the role runs the role actuator's watcher (a Docker host's is below) while `github_runner_cluster_role_controller` is true: the `fleet-role-actuator` unit, which runs `files/role/fleet-role-actuator.sh watch` as root from `/usr/local/lib/github-runner-k3s/role` (`tasks/role_actuator.yml`). It switches the host between its server and agent units when the controller asks through the host's Node, and does nothing on the API but read: the controller cordons, waits for runner pods, drains, taints, labels the level, removes the etcd member, cleans the Node's labels and annotations and uncordons, because an agent host has no credential that can evict pods and its own k3s is down while it switches. Turning the controller off, or a host's eligibility, stops and removes the unit, its kubeconfig and its join URLs; the scripts stay on every native host, since the role's own promotion and demotion run them. **What the controller writes on the Node**, all under `github-runner.exadev.com/`: the label `desired-role` (`server` or `agent`), and the annotations `role-swap` (the swap's id, `--`), `role-step` (`promote`, `step-down` or `rollback`), `role-join-node` (the node of the voter a promotion joins) and `snapshot-request` (a swap whose snapshot a remaining voter is to take). The watcher reads its Node at the pace regrading waits for a restarted server at (`POLL_SECONDS` in `files/regrade/fleet-regrade-k3s.sh`) and takes each step once: a step reported `done`, `failed` or `refused` for the swap is not taken again (what follows a failure is the controller's to decide), and a step whose `desired-role` does not go with it is refused. Finishing a demotion is taken until it is done. @@ -356,6 +356,15 @@ A switch the role makes and one the actuator makes never run together: both hold **Its credential.** The role applies, through another server's API, the ServiceAccount `fleet-role-actuator` in the platform namespace with a ClusterRole that can `get`, `list` and `watch` Nodes, a Role that can `get` the regrade Lease by name, a Role that can `get` and `patch` the report ConfigMap by name, and a token Secret the token controller fills (`templates/role-actuator-rbac.yaml.j2`); it creates the ConfigMap when there is none, and writes a kubeconfig holding the token to `/etc/rancher/k3s/fleet-role-actuator.kubeconfig`, mode 0600, by a task that logs nothing. The watcher reaches the cluster through the first of the other servers and eligible hosts whose `/readyz` answers (`KUBE_SERVERS`, as regrading does), never through its own host's API. It needs `jq`, `ps` and `flock`, which the role installs on Debian and Ubuntu when the host lacks them and refuses the run for otherwise. +**On a Docker host** the watcher is the `role-actuator` service of the host's Compose project, on a server or a host marked eligible while the controller is on (`tasks/role_actuator_docker.yml`): an Alpine image with `bash`, `jq` and the Docker CLI with its Compose and Buildx plugins (`files/role/Dockerfile`), running the same `fleet-role-actuator.sh` with the Docker platform (`files/role/fleet-role-actuator-docker.sh`) from the project's `role` directory, where the role places the scripts, the settings, the kubeconfig and the join URLs. Its own Compose file, `docker-compose.role-actuator.yml`, which the role templates and names only in the Compose command for this service, mounts the Docker socket, the project directory at its own absolute path, so that the paths a Compose command in the container passes the daemon are the host's, and the server's `k3s-data` volume where the server's container mounts it. The steps are those above with containers for units: + +- `promote` writes `server` and the join URL into the role directory (`fleet-role.d/role` and `join-url`, which the entrypoint reads), stops the `k3s-agent` service, copies the node's identity from the stopped agent's volumes into the server's (`_k3s-agent-node` to `_k3s-node` and `_k3s-agent-tailscale-data` to `_k3s-tailscale-data`) and starts the `k3s` service. When the server's container does not come up it starts the agent's again, with the role file naming it, and reports the failure: a Docker host with neither node container up cannot reach the API to be rolled back. +- `step-down` restarts the server's container at `etcd-only` through regrading's Docker actuator (`write_level` and `restart_into` from `files/regrade/fleet-regrade-docker.sh`), whose restart stops every pod process in it, then writes `agent` to the role file and stops the `regrade` service's container. +- `finish-demote` and `rollback` stop the server's container, move `server/db` aside and forget the level in its data volume, remove the join URL, copy the identity back into the agent's volumes and start the `k3s-agent` service. +- `snapshot` runs `k3s etcd-snapshot save` in the server's container. + +The identity moves through `files/node-identity.sh`, the same pipe that seeds a node's password volume ([Node passwords](#node-passwords-and-recreating-servers)): one `docker cp` out of the stopped container straight into another, into a throwaway container created from the node's image and never started, so the node password and the Tailscale state are on no disk of the host, in no command line and in no output, and it refuses while either container runs. The node therefore keeps its name, its node password and its Tailscale device and address. The actuator's container is on no tailnet, so it runs kubectl in whichever node container is up with Tailscale running, with its kubeconfig on kubectl's standard input. The `regrade` service, which the role directory is mounted into too (`docker-compose.fleet-role.yml`), idles while the role file names another role than the server, as the k3s containers do, so a host stepped down or demoted for a swap is never stepped back up by it. Turning the controller off, or a host's eligibility, removes the service's container, its kubeconfig, join URLs and Compose file. + ## Networking Every node's k3s uses its mesh address for everything: its node IP, etcd's peer URLs, the API server and pod traffic. Which physical path the mesh takes between two nodes is Tailscale's choice, among the endpoints each node's Tailscale client can offer its peers. On Compose's default bridge network (`github_runner_cluster_network_mode: bridge`, the default) the client sits behind the Docker host's NAT and, inside a Docker VM, behind the VM's too, so it has no address its peers can reach and every pair of nodes is relayed, even on one LAN. @@ -487,7 +496,7 @@ The role controller holds the regrade Lease (`kube-system/fleet-regrade`, the Le - `github_runner_cluster_native_tailscale_install` (default `true`): whether the role installs the pinned Tailscale package from Tailscale's apt repository (Debian and Ubuntu). `false` leaves the host's client as it is and only checks its login. - `github_runner_cluster_native_tailscale_repository_url` (default `https://pkgs.tailscale.com/stable`): where that repository's signing key and list file are fetched from, when the host has none. - `github_runner_cluster_native_tailscale_leave_other_tailnet` (default `false`): whether the role logs a client out of another tailnet so that k3s can join the cluster's; `false` stops the run instead. -- `github_runner_cluster_role_actuator_dir`, `github_runner_cluster_role_actuator_service`, `github_runner_cluster_role_actuator_name`, `github_runner_cluster_role_actuator_configmap`, `github_runner_cluster_role_actuator_kubeconfig`, `github_runner_cluster_role_actuator_namespace` and `github_runner_cluster_role_actuator_commands` are derived: where the role actuator's scripts and settings go, its unit, its ServiceAccount, its report ConfigMap, its kubeconfig, the platform namespace it reports in and the commands its watcher needs with the package of each (see [Role actuator](#role-actuator)); `github_runner_cluster_api_candidates` is derived too, the other servers and eligible hosts whose API the regrade service and the role actuator reach the cluster through. +- `github_runner_cluster_role_actuator_dir`, `github_runner_cluster_role_actuator_service`, `github_runner_cluster_role_actuator_name`, `github_runner_cluster_role_actuator_configmap`, `github_runner_cluster_role_actuator_kubeconfig`, `github_runner_cluster_role_actuator_namespace` and `github_runner_cluster_role_actuator_commands` are derived: where the role actuator's scripts and settings go (on a native host `/usr/local/lib/github-runner-k3s/role`, on a Docker host the Compose project's `role` directory), its unit, its ServiceAccount, its report ConfigMap, its kubeconfig, the platform namespace it reports in and the commands its watcher needs with the package of each (see [Role actuator](#role-actuator)); `github_runner_cluster_api_candidates` is derived too, the other servers and eligible hosts whose API the regrade service and the role actuator reach the cluster through. - `github_runner_cluster_docker_role_file`, `github_runner_cluster_role_lease_holder` and `github_runner_cluster_role_lease_swap_id_annotation` are derived: the file that records an eligible Docker host's role, and the role controller's holder name on the regrade Lease and the annotation naming its swap (see [A Docker host's recorded role](#a-docker-hosts-recorded-role) and [Runs during a voter swap](#runs-during-a-voter-swap)). - `github_runner_cluster_native_agent_service`, `github_runner_cluster_native_server_service`, `github_runner_cluster_native_service`, `github_runner_cluster_native_join_url`, `github_runner_cluster_native_join_host`, `github_runner_cluster_native_role_file` and `github_runner_cluster_native_restart_marker` are derived: the units' names (`k3s-agent` and `k3s`), the unit of the host's role, the URL the node joins and the host in it, which the Tailscale wrapper waits for, the role file both units' conditions read, and the prefix of the per-unit file that says a restart is waiting; `github_runner_cluster_native_regrade_level_file`, `github_runner_cluster_native_regrade_started_file`, `github_runner_cluster_native_regrade_dir`, `github_runner_cluster_native_regrade_service`, `github_runner_cluster_native_regrade_commands`, `github_runner_cluster_native_agent_runtime_dir` and `github_runner_cluster_native_server_runtime_dir` are derived too: a native server's requested and started regrading levels, where its regrade scripts and settings go, the unit that runs them, the commands regrading needs with the package of each, and each unit's own runtime directory under `/run` (one per unit, since systemd removes a unit's runtime directory when its condition skips a start, which on a shared one would take the running unit's resolver file). `github_runner_cluster_native_drain_for` is derived as well: what `tasks/native/drain.yml` says was not done when it cannot empty the node, a role switch's by default, which a Colima VM resize passes as its own. `github_runner_cluster_native_bin_dir`, `github_runner_cluster_native_lib_dir`, `github_runner_cluster_native_config_dir`, `github_runner_cluster_native_unit_dir`, `github_runner_cluster_native_data_dir`, `github_runner_cluster_native_tailscale_cli`, `github_runner_cluster_native_apt_keyring`, `github_runner_cluster_native_apt_source`, `github_runner_cluster_native_run_dir` and `github_runner_cluster_native_psi_file` are where the native node's files live, which are not settings, and are variables only so the tests can point them at a stand-in. @@ -539,7 +548,7 @@ or read from 1Password by adding `exadev.github_runner.github_runner_secrets_one `tests/unit/test_cluster_wait_nodes.py` runs the wait for nodes to be Ready and its rescue against a stand-in for `kubectl`: a node outside the inventory that is not Ready is reported and never waited for, and an inventory node that stays not Ready still fails the run. The mesh test does not cover it, since its nodes publish no API port for the host's `kubectl` to reach. -`tests/unit/test_node_entrypoint.py` runs the entrypoint's guards in a shell with a stand-in for `sleep` ([The role directory and a former member](#the-role-directory-and-a-former-member)): the join URL's precedence, `--cluster-init` with a datastore, with one moved aside and with none, the idling of a service its role file does not name and of a server on a tombstone, a host with no role file starting as before, and the override that mounts the directory. `tests/unit/test_native_server.py` runs the native server unit's condition on a tombstone. `tests/unit/test_role_actuator.py` runs the [role actuator](#role-actuator) against stand-ins for `systemctl`, `kubectl` and `k3s` (`tests/role_actuator/`, over regrading's own in `tests/regrade/`) and the real `files/native/fleet-role-is.sh` as both units' condition: each step's success, failure, refusal (no Lease, a Lease for another swap, a host that is not provisioned, a role file that names no role, a desired role that does not go with the step) and report, a demotion finished without the Lease once a tombstone, k3s's removal annotation or a missing etcd label says the member is gone and not before, one moved-aside datastore kept, the join drop-ins written and removed, the switch lock, `status`, the watcher taking a swap through each step once, and, through the role, the watcher's unit, settings and kubeconfig on an eligible host, its credential scoped to reading Nodes and the Lease and patching its report, its token kept out of everything the role prints, nothing set up on a host the controller may not switch or under `--check`, and the regrade unit's condition. The `role_actuator` mesh scenario acts as the controller by hand against real k3s and etcd: the native standby's actuator promotes it (a learner, then a voter, with writes succeeding throughout), steps it down (etcd serving at etcd-only, no pod's shim left), and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together. The `rejoin` mesh scenario does the same with real k3s and etcd: it removes the bootstrap Docker server's member through another server and checks the restarted server idles on its tombstone, that with its datastore moved aside and nothing to join it refuses to start, that a `join-url` naming another server makes it join the existing cluster as a learner that is promoted, with the same etcd cluster ID before and after, and that a role file naming the agent leaves the server container running nothing but its idle wait. +`tests/unit/test_node_entrypoint.py` runs the entrypoint's guards in a shell with a stand-in for `sleep` ([The role directory and a former member](#the-role-directory-and-a-former-member)): the join URL's precedence, `--cluster-init` with a datastore, with one moved aside and with none, the idling of a service its role file does not name and of a server on a tombstone, a host with no role file starting as before, and the override that mounts the directory. `tests/unit/test_native_server.py` runs the native server unit's condition on a tombstone. `tests/unit/test_role_actuator.py` runs the [role actuator](#role-actuator) against stand-ins for `systemctl`, `kubectl` and `k3s` (`tests/role_actuator/`, over regrading's own in `tests/regrade/`) and the real `files/native/fleet-role-is.sh` as both units' condition: each step's success, failure, refusal (no Lease, a Lease for another swap, a host that is not provisioned, a role file that names no role, a desired role that does not go with the step) and report, a demotion finished without the Lease once a tombstone, k3s's removal annotation or a missing etcd label says the member is gone and not before, one moved-aside datastore kept, the join drop-ins written and removed, the switch lock, `status`, the watcher taking a swap through each step once, and, through the role, the watcher's unit, settings and kubeconfig on an eligible host, its credential scoped to reading Nodes and the Lease and patching its report, its token kept out of everything the role prints, nothing set up on a host the controller may not switch or under `--check`, and the regrade unit's condition. The `role_actuator` mesh scenario acts as the controller by hand against real k3s and etcd: the native standby's actuator promotes it (a learner, then a voter, with writes succeeding throughout), steps it down (etcd serving at etcd-only, no pod's shim left), and once its member is removed through k3s's annotation finishes it as an agent with the same node password and Tailscale address and its datastore moved aside, the two k3s units never running together. For a Docker host the same module runs the steps against a stand-in for `docker` (`tests/role_actuator/fake-docker.py`) that keeps containers and volumes as files and moves a real tar stream between its `docker cp` processes: a promotion and a finished demotion moving the identity between the services' volumes, never out of a running container or into a volume a running one mounts, the volumes it creates named and labelled for the project, the identity, the node's password and the actuator's token kept out of every call and output, a server that does not come up bringing the agent back, a step-down through regrading's Docker actuator, kubectl run in the node container that is up with its kubeconfig on standard input, the snapshot taken in the server's container, `files/node-identity.sh`'s refusals and clean-up, and the regrade service idling on a host whose role file names the agent. The scenario then demotes a Docker server through its role-actuator service in the same way, by the annotations set by hand, and promotes it again: the agent keeps the server's node password and Tailscale address as one Node with its datastore moved aside in `k3s-data`, and the promoted server joins the same cluster. The `rejoin` mesh scenario does the same with real k3s and etcd: it removes the bootstrap Docker server's member through another server and checks the restarted server idles on its tombstone, that with its datastore moved aside and nothing to join it refuses to start, that a `join-url` naming another server makes it join the existing cluster as a learner that is promoted, with the same etcd cluster ID before and after, and that a role file naming the agent leaves the server container running nothing but its idle wait. `tests/unit/test_docker_role.py` runs the Docker host's role decision, its record and the guard for a voter swap through the role against the stand-ins for `docker` and the Compose module ([A Docker host's recorded role](#a-docker-hosts-recorded-role), [Runs during a voter swap](#runs-during-a-voter-swap)): an eligible host keeps its recorded role over the inventory's, the role is recorded only when absent and only for an eligible host, the servers' turns and the regrade service follow it, a host whose containers or record differ from the role it is to run is refused, a run is refused while the controller holds the Lease and goes on while regrading holds it or it has run out, `--check` reports the same refusals, and the one `.env` gives each service the configuration hash it had before. `tests/unit/test_native_node.py` runs the same guard on a native host. diff --git a/roles/github_runner_cluster/defaults/main.yml b/roles/github_runner_cluster/defaults/main.yml index 29797ed3..263d0eaa 100644 --- a/roles/github_runner_cluster/defaults/main.yml +++ b/roles/github_runner_cluster/defaults/main.yml @@ -213,11 +213,13 @@ github_runner_cluster_api_candidates: >- + (github_runner_cluster_topology.hosts | dict2items | rejectattr('key', 'equalto', inventory_hostname) | rejectattr('value.role', 'equalto', 'server') | selectattr('value.eligible') | list) }} # Derived, never set directly: the role actuator (files/role/fleet-role-actuator.sh), which switches a native host between its server and agent units for the role controller's voter swap, as the host's Node asks, and holds the host-local part of the role's own promotion and demotion. Its scripts and settings go in github_runner_cluster_role_actuator_dir on every native host; its watcher, the github_runner_cluster_role_actuator_service unit, runs on every server and every eligible host while github_runner_cluster_role_controller is true (tasks/role_actuator.yml), with a credential of its own (the ServiceAccount github_runner_cluster_role_actuator_name, which can read Nodes and the regrade Lease and patch its report, the ConfigMap github_runner_cluster_role_actuator_configmap in github_runner_cluster_role_actuator_namespace) in a kubeconfig only root reads. -github_runner_cluster_role_actuator_dir: "{{ github_runner_cluster_native_lib_dir }}/role" +# On a Docker host (node mode compose) the scripts, settings and kubeconfig go in the Compose project's role directory instead, and the watcher is the project's role-actuator service (tasks/role_actuator_docker.yml), whose container mounts the project directory. +github_runner_cluster_role_actuator_dir: "{{ github_runner_cluster_native_lib_dir ~ '/role' if github_runner_cluster_node_mode == 'native' else github_runner_cluster_dir ~ '/role' }}" github_runner_cluster_role_actuator_service: fleet-role-actuator github_runner_cluster_role_actuator_name: fleet-role-actuator github_runner_cluster_role_actuator_configmap: fleet-role-actuators -github_runner_cluster_role_actuator_kubeconfig: "{{ github_runner_cluster_native_config_dir }}/fleet-role-actuator.kubeconfig" +github_runner_cluster_role_actuator_kubeconfig: >- + {{ github_runner_cluster_native_config_dir if github_runner_cluster_node_mode == 'native' else github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.kubeconfig # The ARC role's platform namespace, where the role controller runs and reads the actuators' reports: its github_runner_arc_platform_namespace where the inventory sets that, and that variable's default otherwise (tests/unit/test_role_actuator.py holds the two together). github_runner_cluster_role_actuator_namespace: "{{ github_runner_arc_platform_namespace | default('github-runner-platform') }}" # Not a setting: the commands the actuator's watcher runs that a minimal host may lack, each with the Debian package that installs it. jq reads the cluster's answers, ps finds the pods left at etcd-only (as the regrade service does) and flock holds the host's switch lock. diff --git a/roles/github_runner_cluster/files/docker-compose.fleet-role.yml b/roles/github_runner_cluster/files/docker-compose.fleet-role.yml index f603de85..2bc4b1ff 100644 --- a/roles/github_runner_cluster/files/docker-compose.fleet-role.yml +++ b/roles/github_runner_cluster/files/docker-compose.fleet-role.yml @@ -1,4 +1,4 @@ -# Merged after docker-compose.yml by the github_runner_cluster role when this host takes part in the role controller (github_runner_cluster_publishes_node_labels in defaults/main.yml: the controller is on, or this host is marked a voter candidate or pinned): mounts the host's role directory, fleet-role.d in the Compose project directory, read-only into both services at /etc/github-runner, where k3s/node-entrypoint.sh reads the role the host runs and the server a promoted server joins. A directory rather than its files, because a file replaced by a rename has a new inode, which a bind mount of the file itself does not follow. Its own file, as docker-compose.node-labels.yml is, so that a host the controller never moves keeps the Compose configuration it has and its container is not recreated by an upgrade. +# Merged after docker-compose.yml by the github_runner_cluster role when this host takes part in the role controller (github_runner_cluster_publishes_node_labels in defaults/main.yml: the controller is on, or this host is marked a voter candidate or pinned): mounts the host's role directory, fleet-role.d in the Compose project directory, read-only into both node services at /etc/github-runner, where k3s/node-entrypoint.sh reads the role the host runs and the server a promoted server joins, and into the regrade service, whose watcher idles while the role file names another role than the server (regrade/fleet-regrade.sh). A directory rather than its files, because a file replaced by a rename has a new inode, which a bind mount of the file itself does not follow. Its own file, as docker-compose.node-labels.yml is, so that a host the controller never moves keeps the Compose configuration it has and its container is not recreated by an upgrade. services: k3s: volumes: @@ -6,3 +6,6 @@ services: k3s-agent: volumes: - ./fleet-role.d:/etc/github-runner:ro + regrade: + volumes: + - ./fleet-role.d:/etc/github-runner:ro diff --git a/roles/github_runner_cluster/files/regrade/fleet-regrade.sh b/roles/github_runner_cluster/files/regrade/fleet-regrade.sh index 943a3ca1..b1245cd6 100755 --- a/roles/github_runner_cluster/files/regrade/fleet-regrade.sh +++ b/roles/github_runner_cluster/files/regrade/fleet-regrade.sh @@ -17,6 +17,23 @@ esac # shellcheck source=/dev/null . "$lib/fleet-regrade-$FLEET_REGRADE_ACTUATOR.sh" +# A Docker host's role directory, mounted from the Compose project by docker-compose.fleet-role.yml, records the role the host runs (files/k3s/node-entrypoint.sh reads the same file). On a host it names another role than the server, a voter swap has stepped this server down or demoted it, and a watcher that ran would step it back up: the service idles instead, as the entrypoint does, since restart: unless-stopped would start one that exited again. A native server's unit has the same guard as its ExecCondition (templates/native/fleet-regrade.service.j2). A host with no role file runs as before. +FLEET_ROLE_FILE=${FLEET_ROLE_FILE:-/etc/github-runner/role} +if [ "$FLEET_REGRADE_ACTUATOR" = docker ] && [ -e "$FLEET_ROLE_FILE" ] && [ "$(cat "$FLEET_ROLE_FILE")" != server ]; then + message="$FLEET_ROLE_FILE names '$(cat "$FLEET_ROLE_FILE")', not server, so there is no server here to regrade" + case ${1:-} in + watch) + log "${message}; idling until the service is stopped" + trap 'exit 0' TERM INT + while :; do + sleep infinity & + wait $! + done + ;; + *) die "$message" ;; + esac +fi + # Memory pressure is what the core decides on. A kernel without it (built without CONFIG_PSI, or booted with psi=0) would fail on every sample, so the service stops at once and says why. The role refuses to turn regrading on where it can read that the kernel has none. case ${1:-} in watch | status) diff --git a/roles/github_runner_cluster/files/role/Dockerfile b/roles/github_runner_cluster/files/role/Dockerfile new file mode 100644 index 00000000..683cc749 --- /dev/null +++ b/roles/github_runner_cluster/files/role/Dockerfile @@ -0,0 +1,4 @@ +# The role-actuator service of a Docker host's Compose project (templates/docker-compose.role-actuator.yml.j2): bash, jq, and the Docker CLI with its Compose and Buildx plugins, all from Alpine's apk, Buildx because Compose builds a service's image with it, which the first start of a service the host has never run needs. flock and date are BusyBox's. No kubectl: the actuator runs the kubectl of the node container that is up (node_kubectl in fleet-role-actuator-docker.sh), since only the node containers are on the tailnet the API servers are reached over. The scripts it runs are not copied in: the service mounts the project directory, which holds them, so a host picks up a collection upgrade when the service is created again, without an image Compose would not otherwise rebuild. +FROM alpine:3.22 + +RUN apk add --no-cache bash jq docker-cli docker-cli-compose docker-cli-buildx diff --git a/roles/github_runner_cluster/files/role/fleet-role-actuator-docker.sh b/roles/github_runner_cluster/files/role/fleet-role-actuator-docker.sh new file mode 100644 index 00000000..8998efde --- /dev/null +++ b/roles/github_runner_cluster/files/role/fleet-role-actuator-docker.sh @@ -0,0 +1,201 @@ +# The Docker platform of fleet-role-actuator, for a host whose k3s runs under the cluster role's Compose project (github_runner_cluster_node_mode: docker): the server's k3s service in the container SERVER_CONTAINER (profile server) or the agent's k3s-agent service in AGENT_CONTAINER (profile agent), each idling unless the role file in the project's role directory (ROLE_DIR, fleet-role.d, which docker-compose.fleet-role.yml mounts into both) names its role (files/k3s/node-entrypoint.sh). It runs as the role-actuator service of the same project (templates/docker-compose.role-actuator.yml.j2), in a container with the Docker socket, the project directory at its own absolute path (so that the paths a Compose command passes the daemon are the host's) and the server's k3s data volume at K3S_DATA_DIR, where the server's container mounts it too. fleet-role-actuator.sh sources fleet-regrade-core.sh, fleet-regrade-k3s.sh and fleet-regrade-docker.sh ahead of this file, which supply kubectl_, the Lease reader, write_level, restart_into and current_level, and then calls only the functions of the platform contract, which fleet-role-actuator-systemd.sh's header sets out and this file supplies for containers: a "unit" there is a service's container here. +# +# THE IDENTITY. The two services keep their node password (/etc/rancher/node) and their Tailscale state (/var/lib/tailscale) in volumes of their own, so a switch copies both from the stopped container of the role the host leaves into the volumes of the role it takes, through files/node-identity.sh, before that role's container starts: the node keeps its name and node password in the cluster and its device and address on the tailnet. A side whose container was never created has nothing to copy, and its volumes still hold the identity the host started with. +# +# THE API. This container is on no tailnet, and the node containers are: kubectl runs in whichever node container is running with Tailscale up (node_kubectl, the KUBECTL the settings name), given the actuator's own kubeconfig on its standard input, so the token is on no disk of that container and in no command line. While neither runs, partway through a switch, nothing reaches the API, and the watch reports the step once one does. +# +# It needs docker with its Compose plugin, flock and date, besides what fleet-regrade-docker.sh needs, and the settings named below, which the role writes to the actuator's environment file (templates/role-actuator.env.j2). + +: "${PROJECT_DIR:?PROJECT_DIR must be set}" "${COMPOSE_PROJECT:?COMPOSE_PROJECT must be set}" "${ROLE_DIR:?ROLE_DIR must be set}" "${STATE_DIR:?STATE_DIR must be set}" +: "${SERVER_CONTAINER:?SERVER_CONTAINER must be set}" "${AGENT_CONTAINER:?AGENT_CONTAINER must be set}" "${REGRADE_CONTAINER:?REGRADE_CONTAINER must be set}" + +SERVER_DIR=$K3S_DATA_DIR/server +ROLE_FILE=$ROLE_DIR/role +JOIN_FILE=$ROLE_DIR/join-url +SWITCH_LOCK=$STATE_DIR/fleet-role.lock +LAST_START_FILE=$STATE_DIR/fleet-role-started-at +# lib is fleet-role-actuator.sh's own directory, where the role places node-identity.sh beside it. +# shellcheck disable=SC2154 # Set by fleet-role-actuator.sh, which sources this file. +IDENTITY=$lib/node-identity.sh +# The Tailscale daemon's socket in a node container, as files/k3s/node-entrypoint.sh starts it: present only while that container's k3s runs, never in one that idles. +TAILSCALE_SOCKET=/var/run/tailscale/tailscaled.sock + +# The Compose service, profile and identity volume keys of each role, as files/docker-compose.yml names them. +service_of() { if [ "$1" = server ]; then echo k3s; else echo k3s-agent; fi; } +container_of() { if [ "$1" = server ]; then echo "$SERVER_CONTAINER"; else echo "$AGENT_CONTAINER"; fi; } +volume_of() { + local prefix + if [ "$1" = server ]; then prefix=k3s; else prefix=k3s-agent; fi + echo "${prefix}-$2" +} + +# Runs docker compose on the host's project with the Compose files the role runs it with (COMPOSE_FILES, names relative to the project directory; none means docker-compose.yml alone, as Compose reads it by itself) and the profile $1. +compose() { + local profile=$1 file files=() + shift + # shellcheck disable=SC2086 # COMPOSE_FILES is a list of file names, split on purpose. + for file in ${COMPOSE_FILES:-}; do files+=(--file "$file"); done + (cd "$PROJECT_DIR" && docker compose --project-directory "$PROJECT_DIR" --project-name "$COMPOSE_PROJECT" "${files[@]}" --profile "$profile" "$@") +} + +container_state() { + docker ps --all --filter "name=^/$1\$" --format '{{.State}}' +} + +# The kubectl of a node container whose k3s runs, given this actuator's kubeconfig ($2, after --kubeconfig) on its standard input; kubectl_ calls it with every other argument after that. +node_kubectl() { + local kubeconfig container + [ "${1:-}" = --kubeconfig ] || { echo "node_kubectl: called without --kubeconfig" >&2; return 2; } + kubeconfig=$2 + shift 2 + for container in "$SERVER_CONTAINER" "$AGENT_CONTAINER"; do + [ "$(container_state "$container")" = running ] || continue + docker exec "$container" test -S "$TAILSCALE_SOCKET" 2> /dev/null || continue + docker exec -i "$container" kubectl --kubeconfig /dev/stdin "$@" < "$kubeconfig" + return + done + echo "no node container of this host runs k3s, so the cluster's API cannot be reached from it" >&2 + return 1 +} + +read_role() { + cat "$ROLE_FILE" 2> /dev/null || true +} + +write_role() { + printf '%s' "$1" > "$ROLE_FILE.new" && mv "$ROLE_FILE.new" "$ROLE_FILE" +} + +# The role's .env holds both services' variables on a host it may switch, the server's TLS names among them (templates/env.j2), so a server can start from it. +provisioned() { + grep -q '^K3S_TLS_SAN_LIST=.' "$PROJECT_DIR/.env" 2> /dev/null +} + +server_active() { + [ "$(container_state "$SERVER_CONTAINER")" = running ] +} + +agent_active() { + [ "$(container_state "$AGENT_CONTAINER")" = running ] +} + +datastore_present() { + [ -d "$SERVER_DIR/db" ] +} + +datastore_tombstoned() { + [ -e "$SERVER_DIR/db/etcd/tombstone" ] +} + +datastore_member() { + cat "$SERVER_DIR/db/etcd/name" 2> /dev/null || true +} + +datastore_moved_aside() { + [ -d "$SERVER_DIR/db.demoted" ] +} + +host_now() { + date -u +%Y-%m-%dT%H:%M:%SZ +} + +# The entrypoint reads join-url at every start of the server's container and joins the server it names, whatever K3S_JOIN_SERVER_URL in .env says, and waits for that server's host on the mesh. +join_set() { + printf '%s\n' "$1" > "$JOIN_FILE.new" && mv "$JOIN_FILE.new" "$JOIN_FILE" +} + +join_current() { + local url + url=$(cat "$JOIN_FILE" 2> /dev/null || true) + if [ -n "$url" ]; then echo "$url"; else echo "the join configuration the role wrote"; fi +} + +join_clear() { + rm -f "$JOIN_FILE" +} + +# Copies the node's identity from the role $1's stopped container into the volumes of the role $2, when the role $1 has a container. +copy_identity() { + local from=$1 to=$2 container kind + container=$(container_of "$from") + [ -n "$(container_state "$container")" ] || return 0 + for kind in node:/etc/rancher/node tailscale-data:/var/lib/tailscale; do + bash "$IDENTITY" --stopped "$container" "${kind#*:}" "${COMPOSE_PROJECT}_$(volume_of "$to" "${kind%%:*}")" "$COMPOSE_PROJECT" "$(volume_of "$to" "${kind%%:*}")" || return 1 + done +} + +# A server whose container did not come up has not joined anything, so the agent is brought back at once, with the role file naming it again and the join URL gone: unlike a native host, whose own Tailscale client keeps the API in reach, a Docker host with neither node container up cannot read the Lease that a rollback asks for, and would be left with nothing running. +start_server_unit() { + if server_active; then return 0; fi + compose agent stop "$(service_of agent)" || return 1 + if copy_identity agent server && record_start && compose server up --detach --no-deps "$(service_of server)"; then + return 0 + fi + log "the server's container did not come up; starting the agent's again" + compose server stop "$(service_of server)" || true + if ! { write_role agent && join_clear && compose agent up --detach --no-deps "$(service_of agent)"; }; then + log "the agent's container did not come up either" + fi + return 1 +} + +stop_server_unit() { + compose server stop "$(service_of server)" +} + +start_agent_unit() { + if agent_active; then return 0; fi + copy_identity server agent || return 1 + record_start || return 1 + compose agent up --detach --no-deps "$(service_of agent)" +} + +record_start() { + STARTED_AT=$(host_now) + printf '%s\n' "$STARTED_AT" > "$LAST_START_FILE" +} + +last_start() { + cat "$LAST_START_FILE" 2> /dev/null || true +} + +# The datastore is on the server's k3s data volume, mounted here at the path the server's container mounts it, and is moved only while that container is stopped. One copy only, as on a native host. +move_datastore_aside() { + datastore_present || return 0 + rm -rf "$SERVER_DIR/db.demoted" && mv "$SERVER_DIR/db" "$SERVER_DIR/db.demoted" +} + +forget_level() { + rm -f "$LEVEL_FILE" +} + +# Restarting the container stops everything in it, its pods' processes included (fleet-regrade-docker.sh's apply_level), so etcd-only needs only the level and the restart. +enter_etcd_only() { + write_level "$LEVEL_FILE" etcd-only || return 1 + restart_into etcd-only +} + +# docker stop rather than Compose's stop, since the service runs only beside a server and its profile is not in every run of the role; a stopped container is not started again by its restart: unless-stopped. +stop_regrade() { + [ "$(container_state "$REGRADE_CONTAINER")" = running ] || return 0 + docker stop "$REGRADE_CONTAINER" > /dev/null +} + +# As on a native host, through the server's own supervisor, inside its container, where k3s finds its token in its data directory. +take_snapshot() { + local output + output=$(docker exec "$SERVER_CONTAINER" k3s etcd-snapshot save --etcd-server "https://127.0.0.1:${API_PORT:?API_PORT must be set}" --name "$1" 2>&1) || { + log "$output" >&2 + return 1 + } + sed -n 's/.*Snapshot \([^ ]*\) saved\..*/\1/p' <<< "$output" +} + +lock_switch() { + exec 9> "$SWITCH_LOCK" || return 1 + flock -w "$OPERATION_TIMEOUT_SECONDS" 9 +} + +unlock_switch() { + exec 9>&- +} diff --git a/roles/github_runner_cluster/files/role/fleet-role-actuator.sh b/roles/github_runner_cluster/files/role/fleet-role-actuator.sh index d52295a9..3ac1c848 100644 --- a/roles/github_runner_cluster/files/role/fleet-role-actuator.sh +++ b/roles/github_runner_cluster/files/role/fleet-role-actuator.sh @@ -11,7 +11,7 @@ # # EXIT STATUS. 0: done. 1: failed; the host is left with at most one role's unit running and a role file naming that role. 2: refused, with nothing changed: the Lease is not held by ROLE_LEASE_HOLDER for this swap (or cannot be read), the host is not provisioned to be a server, the role file names something that is not a role, the step is not one this actuator takes, or the step's own precondition does not hold (each says which). 3: not yet (finish-demote before the member is gone). # -# The platform's file, fleet-role-actuator-$FLEET_ROLE_ACTUATOR.sh, says how each part is done on the host (its header lists the functions), after fleet-regrade-core.sh (kubectl_, select_api_server, lease_state, now_epoch, log), fleet-regrade-k3s.sh (write_level, restart_into, current_level, POLL_SECONDS) and the regrading actuator of the same platform, all from this script's directory. The settings are in fleet-role-actuator.env beside it, which the role templates (templates/role-actuator.env.j2). It needs bash, jq and kubectl besides what the platform needs. +# The platform's file, fleet-role-actuator-$FLEET_ROLE_ACTUATOR.sh (systemd for a native host, docker for a host under the role's Compose project), says how each part is done on the host (fleet-role-actuator-systemd.sh's header lists the functions), after fleet-regrade-core.sh (kubectl_, select_api_server, lease_state, now_epoch, log), fleet-regrade-k3s.sh (write_level, restart_into, current_level, POLL_SECONDS) and the regrading actuator of the same platform, all from this script's directory. The settings are in fleet-role-actuator.env beside it, which the role templates (templates/role-actuator.env.j2). It needs bash, jq and kubectl besides what the platform needs. set -euo pipefail lib=$(dirname "$(readlink -f "$0")") @@ -26,9 +26,9 @@ EXIT_NOT_YET=3 # shellcheck source=/dev/null . "$lib/fleet-regrade-core.sh" case ${FLEET_ROLE_ACTUATOR:-} in - systemd) ;; + systemd | docker) ;; *) - echo "FLEET_ROLE_ACTUATOR is '${FLEET_ROLE_ACTUATOR:-}' in $FLEET_REGRADE_ENV; it must be systemd." >&2 + echo "FLEET_ROLE_ACTUATOR is '${FLEET_ROLE_ACTUATOR:-}' in $FLEET_REGRADE_ENV; it must be systemd or docker." >&2 exit "$EXIT_REFUSED" ;; esac diff --git a/roles/github_runner_cluster/tasks/main.yml b/roles/github_runner_cluster/tasks/main.yml index 4bdfc6a6..8dade61f 100644 --- a/roles/github_runner_cluster/tasks/main.yml +++ b/roles/github_runner_cluster/tasks/main.yml @@ -177,6 +177,9 @@ - name: Run the regrade service beside this server, or remove it ansible.builtin.include_tasks: regrade_service.yml + - name: Run the role actuator's service, which switches this host for the role controller, or remove it + ansible.builtin.include_tasks: role_actuator.yml + - name: Check the runtime and this node's container are up ansible.builtin.include_tasks: end_checks.yml when: ansible_facts['system'] == 'Darwin' @@ -202,6 +205,9 @@ ansible.builtin.include_tasks: end_checks.yml when: ansible_facts['system'] == 'Darwin' + - name: Run the role actuator's service on an eligible agent, which the role controller may promote, or remove it + ansible.builtin.include_tasks: role_actuator.yml + - name: End the role here, since an agent host has no API of its own to wait on ansible.builtin.meta: end_role diff --git a/roles/github_runner_cluster/tasks/role_actuator.yml b/roles/github_runner_cluster/tasks/role_actuator.yml index 50485f83..3109892c 100644 --- a/roles/github_runner_cluster/tasks/role_actuator.yml +++ b/roles/github_runner_cluster/tasks/role_actuator.yml @@ -1,17 +1,23 @@ --- -# Runs the role actuator's watcher (files/role/fleet-role-actuator.sh watch, as the github_runner_cluster_role_actuator_service unit) on a native host the role controller may switch or take a snapshot on, a server or a host marked eligible, while github_runner_cluster_role_controller is true, and removes it otherwise; native/config.yml has placed its scripts and settings on every native host. The watcher gets a credential of its own: the role applies its ServiceAccount, its scoped rules and its token Secret (templates/role-actuator-rbac.yaml.j2) through another server's API, creates the report ConfigMap the controller reads when there is none, and writes a kubeconfig holding the token, mode 0600, by a task that logs nothing. A file maps every other server and eligible host's node name to the URL a promotion joins it at. Run after native/service.yml has run the host's unit. While github_runner_cluster_native_start is false the unit is left as it is found, running or not, as the k3s units are, and only its files are written. Under --check nothing is changed and each change a real run would make is reported. +# Runs the role actuator's watcher (files/role/fleet-role-actuator.sh watch) on a host the role controller may switch or take a snapshot on, a server or a host marked eligible, while github_runner_cluster_role_controller is true, and removes it otherwise: on a native host as the github_runner_cluster_role_actuator_service unit (role_actuator_watcher.yml; native/config.yml has placed its scripts and settings on every native host), on a Docker host as the role-actuator service of its Compose project (role_actuator_docker.yml). The watcher gets a credential of its own (role_actuator_credential.yml). Run after the host's node has been started: on a native host after native/service.yml, where while github_runner_cluster_native_start is false the unit is left as it is found, running or not, as the k3s units are, and only its files are written. Under --check nothing is changed and each change a real run would make is reported. - name: Work out whether the role actuator's watcher runs on this host ansible.builtin.set_fact: - github_runner_cluster_role_actuator_on: "{{ github_runner_cluster_role_controller | bool and github_runner_cluster_native_stage_server | bool }}" + github_runner_cluster_role_actuator_on: >- + {{ github_runner_cluster_role_controller | bool + and (github_runner_cluster_native_stage_server | bool if github_runner_cluster_node_mode == 'native' + else (github_runner_cluster_docker_role == 'server' or github_runner_cluster_topology.hosts[inventory_hostname].eligible)) }} - name: Run the role actuator's watcher - ansible.builtin.include_tasks: role_actuator_watcher.yml + ansible.builtin.include_tasks: + file: "{{ 'role_actuator_watcher.yml' if github_runner_cluster_node_mode == 'native' else 'role_actuator_docker.yml' }}" when: github_runner_cluster_role_actuator_on | bool -- name: Remove the role actuator's watcher, its credential and its join URLs from the host +- name: Remove the role actuator's watcher, its credential and its join URLs from a native host # The scripts and settings stay: the role's own promotion and demotion run them. What the role applied to the cluster stays too, shared by every host's watcher, and grants nothing to anyone without its token. - when: not github_runner_cluster_role_actuator_on | bool + when: + - not github_runner_cluster_role_actuator_on | bool + - github_runner_cluster_node_mode == 'native' block: - name: Read whether the watcher is installed ansible.builtin.stat: @@ -42,3 +48,36 @@ ansible.builtin.command: argv: [systemctl, daemon-reload] changed_when: true + +- name: Remove the role actuator's service, its credential and its join URLs from a Docker host + # What the role applied to the cluster stays, as on a native host. The kubeconfig says whether the role ever set the watcher up here, so a host that never ran it is not asked for its container. + when: + - not github_runner_cluster_role_actuator_on | bool + - github_runner_cluster_node_mode != 'native' + block: + - name: Read whether the watcher was set up + ansible.builtin.stat: + path: "{{ github_runner_cluster_role_actuator_kubeconfig }}" + register: github_runner_cluster_role_actuator_installed + + - name: Remove the watcher + when: github_runner_cluster_role_actuator_installed.stat.exists + block: + - name: Remove the role actuator's container + ansible.builtin.command: + argv: [docker, rm, --force, "{{ github_runner_cluster_compose_project }}-role-actuator-1"] + environment: "{{ github_runner_cluster_docker_cli_environment }}" + register: github_runner_cluster_role_actuator_removed + changed_when: github_runner_cluster_role_actuator_removed.rc == 0 + failed_when: + - github_runner_cluster_role_actuator_removed.rc != 0 + - "'No such container' not in github_runner_cluster_role_actuator_removed.stderr" + + - name: Remove the watcher's kubeconfig, join URLs and Compose file + ansible.builtin.file: + path: "{{ item }}" + state: absent + loop: + - "{{ github_runner_cluster_role_actuator_kubeconfig }}" + - "{{ github_runner_cluster_role_actuator_dir }}/join-urls" + - "{{ github_runner_cluster_dir }}/docker-compose.role-actuator.yml" diff --git a/roles/github_runner_cluster/tasks/role_actuator_docker.yml b/roles/github_runner_cluster/tasks/role_actuator_docker.yml new file mode 100644 index 00000000..a17bd6c4 --- /dev/null +++ b/roles/github_runner_cluster/tasks/role_actuator_docker.yml @@ -0,0 +1,71 @@ +--- +# Runs the role actuator's watcher on a Docker host the role controller may switch, for tasks/role_actuator.yml, which includes this file only on such a host (see role_actuator_watcher.yml for why an include): the role-actuator service of the host's Compose project, built from files/role/Dockerfile and running files/role/fleet-role-actuator.sh watch with the docker platform. Places the scripts beside its settings in the project's role directory (github_runner_cluster_role_actuator_dir), with regrading's core and Docker actuator, which it runs on, and files/node-identity.sh, which it moves the node's identity between the services with; gives it its credential; and starts the service, or creates it again when something it reads at its start changed. Run after the host's node has been started. Under --check nothing is changed and each change a real run would make is reported. + +- name: Choose the server whose API sets up the watcher's credential + ansible.builtin.include_tasks: role_actuator_api.yml + +- name: Report the watcher a real run would set up + ansible.builtin.debug: + msg: >- + place the role actuator in {{ github_runner_cluster_role_actuator_dir }}, apply its credential ({{ github_runner_cluster_role_actuator_name }} in + {{ github_runner_cluster_role_actuator_namespace }}) and run the role-actuator service of {{ github_runner_cluster_compose_project }} + changed_when: true + when: ansible_check_mode + +- name: Set up the watcher and run it + when: not ansible_check_mode + block: + - name: Create the role actuator's directory + ansible.builtin.file: + path: "{{ github_runner_cluster_role_actuator_dir }}" + state: directory + mode: "0755" + + - name: Place the role actuator's image build context and scripts + # Copied on every run, so a collection upgrade reaches the host; the service runs them from the mounted project directory, so a change takes effect when the service is created again, below. + ansible.builtin.copy: + src: "{{ item.src }}" + dest: "{{ github_runner_cluster_role_actuator_dir }}/{{ item.src | basename }}" + mode: "{{ item.mode }}" + loop: + - {src: role/Dockerfile, mode: "0644"} + - {src: role/fleet-role-actuator.sh, mode: "0755"} + - {src: role/fleet-role-actuator-docker.sh, mode: "0644"} + - {src: regrade/fleet-regrade-core.sh, mode: "0644"} + - {src: regrade/fleet-regrade-k3s.sh, mode: "0644"} + - {src: regrade/fleet-regrade-docker.sh, mode: "0644"} + - {src: node-identity.sh, mode: "0755"} + loop_control: + label: "{{ item.src }}" + register: github_runner_cluster_role_actuator_scripts + + - name: Template the role actuator's settings + ansible.builtin.template: + src: role-actuator.env.j2 + dest: "{{ github_runner_cluster_role_actuator_dir }}/fleet-role-actuator.env" + mode: "0644" + register: github_runner_cluster_role_actuator_settings + + - name: Template the role actuator's Compose file + ansible.builtin.template: + src: docker-compose.role-actuator.yml.j2 + dest: "{{ github_runner_cluster_dir }}/docker-compose.role-actuator.yml" + mode: "0644" + + - name: Give the watcher its credential and the URLs a promotion joins + ansible.builtin.include_tasks: role_actuator_credential.yml + + - name: Start the role actuator's service + # recreate: always when something the watcher reads once, at its start, changed, since Compose sees no change in a mounted directory's contents; a service nothing changed for is left running. Its own Compose file goes last, after every file the k3s services are run with. + community.docker.docker_compose_v2: + project_src: "{{ github_runner_cluster_dir }}" + project_name: "{{ github_runner_cluster_compose_project }}" + profiles: ["role-actuator"] + services: ["role-actuator"] + cli_context: "{{ github_runner_cluster_docker_cli_context or omit }}" + files: "{{ github_runner_cluster_compose_file_list + ['docker-compose.role-actuator.yml'] }}" + recreate: >- + {{ 'always' if (github_runner_cluster_role_actuator_scripts is changed or github_runner_cluster_role_actuator_settings is changed + or github_runner_cluster_role_actuator_kubeconfig_file is changed or github_runner_cluster_role_actuator_join_urls is changed) + else 'auto' }} + state: present diff --git a/roles/github_runner_cluster/templates/docker-compose.role-actuator.yml.j2 b/roles/github_runner_cluster/templates/docker-compose.role-actuator.yml.j2 new file mode 100644 index 00000000..0c47a2f0 --- /dev/null +++ b/roles/github_runner_cluster/templates/docker-compose.role-actuator.yml.j2 @@ -0,0 +1,16 @@ +# Generated by the exadev.github_runner collection's github_runner_cluster role on a Docker host the role controller may switch (github_runner_cluster_role_controller: true, on a server or a host marked github_runner_cluster_voter_eligible). Do not edit by hand. Merged after docker-compose.yml only by the role's own Compose command for this service (tasks/role_actuator_docker.yml), never by the k3s services' commands, so a host the controller never moves keeps the Compose configuration it has. A template rather than a file of the collection, because the project directory is mounted at its own absolute path, which only the role knows: the paths a Compose command run in this container passes the daemon (the k3s build context, the entrypoint and role directory bind mounts) are then the host's. +services: + # The role actuator (files/role/fleet-role-actuator.sh with files/role/fleet-role-actuator-docker.sh), which switches this host between its k3s and k3s-agent services when the role controller asks through the host's Node; see the role README's Role actuator. It needs the Docker socket to stop, start and copy between the node containers, the same permission cost as the regrade and docker-prune services, and the server's k3s data volume, at the path the server mounts it, for the datastore it moves aside and the regrading level it forgets. + role-actuator: + build: + context: ./role + restart: unless-stopped + profiles: ["role-actuator"] + entrypoint: [{{ (github_runner_cluster_role_actuator_dir ~ '/fleet-role-actuator.sh') | to_json }}] + command: ["watch"] + environment: + - {{ ('FLEET_ROLE_ACTUATOR_ENV=' ~ github_runner_cluster_role_actuator_dir ~ '/fleet-role-actuator.env') | to_json }} + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - {{ (github_runner_cluster_dir ~ ':' ~ github_runner_cluster_dir) | to_json }} + - k3s-data:/var/lib/rancher/k3s diff --git a/roles/github_runner_cluster/templates/role-actuator.env.j2 b/roles/github_runner_cluster/templates/role-actuator.env.j2 index 97826309..dba1e3f6 100644 --- a/roles/github_runner_cluster/templates/role-actuator.env.j2 +++ b/roles/github_runner_cluster/templates/role-actuator.env.j2 @@ -1,6 +1,6 @@ -# Generated by the exadev.github_runner collection's github_runner_cluster role on every run. Do not edit by hand: change this host's ansible/host_vars file and rerun ansible-playbook. Read by the role actuator (files/role/fleet-role-actuator.sh), by its watcher unit and by the role's own promotion and demotion of this host; see the role README's Role actuator. It holds no secret: the actuator's credential is in its kubeconfig, which only root reads. +# Generated by the exadev.github_runner collection's github_runner_cluster role on every run. Do not edit by hand: change this host's ansible/host_vars file and rerun ansible-playbook. Read by the role actuator (files/role/fleet-role-actuator.sh), by its watcher (a native host's unit, a Docker host's role-actuator service) and, on a native host, by the role's own promotion and demotion; see the role README's Role actuator. It holds no secret: the actuator's credential is in its kubeconfig, which only root, or on a Docker host the connecting user and the service's container, reads. +{% if github_runner_cluster_node_mode == 'native' %} FLEET_ROLE_ACTUATOR=systemd -NODE_NAME={{ github_runner_cluster_effective_node_name }} ROLE_FILE={{ github_runner_cluster_native_role_file }} K3S_SERVICE={{ github_runner_cluster_native_server_service }} K3S_AGENT_SERVICE={{ github_runner_cluster_native_agent_service }} @@ -10,6 +10,21 @@ CONFIG_DIR={{ github_runner_cluster_native_config_dir }} UNIT_DIR={{ github_runner_cluster_native_unit_dir }} LEVEL_FILE={{ github_runner_cluster_native_regrade_level_file }} STARTED_FILE={{ github_runner_cluster_native_regrade_started_file }} +{% else %} +{# A Docker host's: the paths are those of the role-actuator service's container, which mounts the project directory at its own path and the server's k3s data volume where the server's container mounts it (templates/docker-compose.role-actuator.yml.j2), so LEVEL_FILE and STARTED_FILE are fleet-regrade-docker.sh's own. #} +FLEET_ROLE_ACTUATOR=docker +PROJECT_DIR={{ github_runner_cluster_dir }} +COMPOSE_PROJECT={{ github_runner_cluster_compose_project }} +COMPOSE_FILES="{{ github_runner_cluster_compose_file_list | join(' ') }}" +ROLE_DIR={{ github_runner_cluster_dir }}/fleet-role.d +STATE_DIR={{ github_runner_cluster_role_actuator_dir }} +SERVER_CONTAINER={{ github_runner_cluster_container_name }} +AGENT_CONTAINER={{ github_runner_cluster_agent_container_name }} +NODE_CONTAINER={{ github_runner_cluster_container_name }} +REGRADE_CONTAINER={{ github_runner_cluster_compose_project }}-regrade-1 +K3S_DATA_DIR=/var/lib/rancher/k3s +{% endif %} +NODE_NAME={{ github_runner_cluster_effective_node_name }} {# A step's waits (a restart into etcd-only, the switch lock) are bounded by regrading's, which they are made of. #} OPERATION_TIMEOUT_SECONDS={{ github_runner_cluster_regrade_operation_timeout_seconds }} API_PORT={{ github_runner_cluster_api_port }} @@ -20,7 +35,8 @@ SWAP_ID_ANNOTATION={{ github_runner_cluster_role_lease_swap_id_annotation }} ACTUATOR_NAMESPACE={{ github_runner_cluster_role_actuator_namespace }} ACTUATOR_CONFIGMAP={{ github_runner_cluster_role_actuator_configmap }} JOIN_URLS_FILE={{ github_runner_cluster_role_actuator_dir }}/join-urls -KUBECTL="{{ github_runner_cluster_native_bin_dir }}/k3s kubectl" +{# On a Docker host, the kubectl of whichever node container is up, since only those are on the tailnet (node_kubectl in files/role/fleet-role-actuator-docker.sh). #} +KUBECTL={{ ('"' ~ github_runner_cluster_native_bin_dir ~ '/k3s kubectl"') if github_runner_cluster_node_mode == 'native' else 'node_kubectl' }} KUBECONFIG_PATH={{ github_runner_cluster_role_actuator_kubeconfig }} {# The API servers the actuator reaches the cluster through: never this host's own, which a server stepped down to etcd-only does not have and an agent never has. #} KUBE_SERVERS="{% for candidate in github_runner_cluster_api_candidates %}{{ ' ' if not loop.first else '' }}https://{{ candidate.value.address }}:{{ github_runner_cluster_api_port }}{% endfor %}" diff --git a/tests/role_actuator/fake-docker.py b/tests/role_actuator/fake-docker.py new file mode 100755 index 00000000..fa61e14f --- /dev/null +++ b/tests/role_actuator/fake-docker.py @@ -0,0 +1,259 @@ +#!/usr/bin/env python3 +"""A stand-in for docker for fleet-role-actuator's Docker platform tests, keeping one Docker host in the directory FAKE_DOCKER. + +containers// holds a container: state (running, exited or created), image, mounts (one " " per line), started-at, and tailscale while its k3s runs with Tailscale up. volumes// holds a volume's files and volumes/.labels its labels. log/ holds "