Skip to content

Step-down to etcd-only times out on a voter slowed with netem #227

Open

Description

@Mearman

In the role_controller_swap mesh scenario (PR #218), the outgoing voter is slowed with 600 ms of netem each way on its mesh interface. Its role actuator's step-down then failed with the server did not come back at etcd-only within 300s (run 41937263962, job 126948762038).

The likely cause is the scenario's own setup. It put a tmpfs over server/db in the k3s service alone, and a tmpfs mount is emptied at every restart of the container. So every restart into etcd-only came back with no datastore and joined etcd again as a new learner, slowly under the netem delay. The server container's restart count of 12 and the outgoing Node's removed-node-name being reset in all three runs fit that. PR #218's head ba45aa6 (commit ded0fe6) moves etcd onto a tmpfs volume for the whole k3s data directory, which the role actuator mounts too and which survives the server's restarts.

Deferred until a run on that head shows whether the step-down now reaches etcd-only on its first try.

Revisit trigger: a role_controller_swap run on PR #218's head ba45aa6 or later logs the step-down failing again.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions