Repository navigation
feat(cluster): switch a Docker host's role from the same actuator - #217
Merged
Merged
Conversation
…mes with one script The node password seeding pipe moves into files/node-identity.sh, which copies a directory of a running or stopped node container into a named volume through a throwaway container that is never started, so the role actuator can move a Docker node's password and Tailscale state between its server and agent services the same way. With --stopped it refuses while either side could change the files.
…les any platform can include Choosing the server whose API applies the credential, and applying it, move out of the native watcher's tasks into role_actuator_api.yml and role_actuator_credential.yml, so a Docker host's watcher can be given the same credential the same way.
The role actuator gains a Docker platform, so the role controller's voter swap can move a Docker host between its k3s and k3s-agent services as it moves a native host between its units. On a server or eligible Docker host with the controller on, the role runs it as the role-actuator service of the host's Compose project, from its own Compose file, with the Docker socket, the project directory at its own path and the server's k3s-data volume. A switch writes the role directory the entrypoint reads, stops one service, copies the node password and the Tailscale state from its stopped container into the other service's volumes through node-identity.sh, and starts the other service, so the node keeps its name, password and tailnet address. A step-down restarts the server at etcd-only through regrading's Docker actuator. kubectl runs in whichever node container is up, since the actuator's own container is on no tailnet, and a server that does not come up brings the agent back at once, so the host never loses its way to the API. The regrade service now sees the role directory and idles while it names another role than the server.
…ote it again The role_actuator scenario's Docker part demotes the third Docker server by the Node's annotations set by hand, checks its agent keeps the server's node password and Tailscale address as one Node with the datastore moved aside in k3s-data, then promotes it again into the same etcd cluster. The scenario's node helpers now take the node to read.
…ver's API serves On a first run the Docker servers start together, and the server the credential is applied through may have no API and no kubeconfig yet, so kubectl fell back to localhost and failed. The role now reads that server's /readyz until it serves, for as long as it waits for a server to be Ready, before applying.
… in use Its service and its settings named the whole Compose file list, which every Compose command of the role is to take from github_runner_cluster_compose_files_in_use instead.
… another host created it Every host whose watcher runs creates the ConfigMap in the same play, and kubectl create configmap words the server's refusal as `configmaps "<name>" already exists` without the AlreadyExists reason the task looked for, so every host but the first failed. The refusal is now told by that text naming this ConfigMap; any other failure still fails the run. The kubectl stand-in answers in kubectl's own words.
Every host whose watcher runs applies the same objects in one play. A client-side apply reads each object and then creates it, so two hosts that both find it absent both create it and one is refused; a server-side apply creates or updates it in one request.
The cluster role now makes the namespace for the role actuators' credential while the controller is on, so the scenario's create of it was refused.
The actuator's switch lock waits with `flock -w`, which BusyBox's flock, the one Alpine has by default, does not take. Every step on a Docker host therefore failed on its lock before it changed anything, and reported "another role switch on this host held its lock" at once.
…cenario fails A Docker host's role-actuator container and the fleet-role-actuators ConfigMap were absent from the failure dump, so a step that failed or was refused looked the same as one never taken.
…on again when the API lost it A finish-demote whose report the API did not take left nothing of the server behind (the server stopped, its datastore moved aside), so the watch's guard, which looked for something of the server, never took the finish again, and the controller waited for a report that was not coming. On a Docker host the node container that carries kubectl is the agent just started, with no route to the API yet, so that report was lost on every demotion. The finish is now taken while the role file names the agent, it is not reported done, and either something of the server is left or the Node still carries the swap's step-down. It is idempotent and member_gone still decides whether the member is gone. The finish's report replaces the step-down's, so a step-down counts as taken once the finish of the same swap is reported in any phase. Without that the watch ran the step-down again over the finish's report, and the two reports replaced each other on every pass.
Mearman
marked this pull request as ready for review
November 4, 2026 14:01
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
🎉 This PR is included in version 2.38.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #183
Gives the role actuator a Docker platform (
files/role/fleet-role-actuator-docker.sh), so the role controller's voter swap can move a Docker host between itsk3sandk3s-agentservices the way it moves a native host between its units. On a server or eligible Docker host with the controller on, the role runs it as therole-actuatorservice of the host's Compose project. The service mounts the Docker socket, the project directory at its own absolute path, and the server'sk3s-datavolume where the server's container mounts it.promotewritesfleet-role.d/roleandjoin-urland stopsk3s-agent. It then copies the node password and Tailscale state from the stopped agent's volumes into the server's (<project>_k3s-agent-nodeto<project>_k3s-node,<project>_k3s-agent-tailscale-datato<project>_k3s-tailscale-data) and startsk3s.step-downrestarts the server atetcd-onlythroughfleet-regrade-docker.sh(write_level,restart_into), then writes the role file and stops the regrade container.finish-demoteandrollbackstop the server, then ink3s-datamoveserver/dbaside and drop the level file. They then remove the join URL, copy the identity back and startk3s-agent.snapshotrunsk3s etcd-snapshot savein the server's container.The identity copy is
files/node-identity.sh, which is the node password seeding pipe fromtasks/node_password.ymlgeneralised:docker cpout of a stopped container straight into a throwaway container that is never started.node_password.ymlnow runs that script too. The regrade wrapper idles while the role file names another role than the server.Tests
New unit tests in
tests/unit/test_role_actuator.py, against a new Docker stand-in (tests/role_actuator/fake-docker.py). The stand-in keeps containers and volumes as files and moves a real tar stream between itsdocker cpprocesses. The tests cover:etcd-onlywith regrade stopped;The
role_actuatormesh scenario gains a Docker part. It demotes the third Docker server by hand-set annotations and checks:db.demotedis ink3s-data;It then promotes the server again and checks it becomes a voter in the same etcd cluster ID.
I checked that each new test discriminates. In a throwaway clone I made each break below, saw the named test fail, then saw it pass on this branch:
test_a_promotion_moves_the_identity_to_the_server_and_starts_it_in_place_of_the_agenttest_a_demotion_is_finished_by_moving_the_identity_back_and_starting_the_agent--stoppedcheck removed from node-identitytest_a_copy_out_of_a_running_container_is_refused_with_nothing_changedtest_the_identity_volumes_a_promotion_creates_are_named_and_labelled_for_the_projecttest_a_failed_copy_removes_the_volume_it_created_and_its_throwaway_containertest_kubectl_runs_in_the_node_container_with_tailscale_up_and_takes_the_kubeconfig_on_its_standard_inputtest_a_promotion_whose_server_does_not_come_up_brings_the_agent_back_with_its_identitytest_a_demotion_is_finished_by_moving_the_identity_back_and_starting_the_agenttest_a_step_down_restarts_the_server_at_etcd_only_then_names_the_agent_and_stops_regradingtest_the_watcher_idles_on_a_host_whose_role_file_names_the_agenttest_the_service_mounts_the_socket_the_project_at_its_own_path_and_the_servers_data_where_the_settings_saytest_the_role_places_every_script_the_docker_actuator_runstest_the_credential_is_applied_once_the_other_servers_api_servesAlreadyExistsreason, whichkubectl create configmapnever printstest_a_report_configmap_another_host_created_first_is_taken_as_it_istest_any_other_failure_to_create_the_report_configmap_fails_the_run--server-side)test_an_eligible_host_runs_the_watcher_with_its_own_credential_and_settingsflocknot installed in the image (BusyBox's takes no-w)test_the_image_installs_the_flock_the_switch_lock_waits_withtest_a_finish_whose_report_the_api_did_not_take_is_reported_again_though_nothing_of_the_server_is_leftandtest_a_report_the_api_refused_to_take_is_made_again_once_it_takes_it_and_the_entry_ends_at_the_finishtest_a_finish_reported_in_every_phase_is_not_replaced_by_the_step_down_it_followsDeviations from the plan
docker-compose.role-actuator.yml), not indocker-compose.yml. Only the role's Compose command for this service names that file. Two reasons: the project directory has to be mounted at its own absolute path, which only the role knows, and a variable indocker-compose.ymlwould be interpolated for every service on every host.--kubeconfig /dev/stdin), so it never sits on that container's disk and never appears on a command line. The image instead gainsdocker-cli-buildx, which Compose needs to build the other service's image the first time a host runs it.k3s-datavolume, so the datastore and level file are handled as files rather than through a throwaway container.<volume>-identity-copy. The node password tests were updated to match..envholds the server's TLS names.env.j2writes both services' variables.docker-compose.fleet-role.ymlalso mounts the role directory intoregrade. As a result, a host that already uses that override recreates its regrade container once.role_actuator_api.ymlandrole_actuator_credential.yml, shared by both platforms.node_is_voter,member_removedandnode_ready_sincetake a node and default to the native host.AlreadyExists, whichkubectl create configmapdoes not print (it sayserror: failed to create configmap: configmaps "<name>" already exists). The native kubectl stand-in now answers in kubectl's own words./readyzserves. On a first run the Docker servers start together, so the server may not have an API or a kubeconfig yet.fleet-role-actuator. Every host whose watcher runs applies the same objects in one play. A client-side apply reads each object before creating it, so two hosts that both see it missing would both create it, and one would be refused withAlreadyExists. A server-side apply creates or updates each object in one request. As a result, therole_controllermesh scenario now applies the platform namespace, since the role has already made it.flock. BusyBox's, the one Alpine has by default, takes no-w, so every step on a Docker host failed on its switch lock before it changed anything.step-down, even when nothing of the server is left; it then only reports, andmember_gonestill decides. A step-down counts as taken once the finish of the same swap is reported in any phase, since the finish's report replaces the step-down's. The promote and step-down reports were already retried by thesettledguard, since both steps re-run idempotently; a rollback gets no finish.fleet-role-actuatorsdata.