feat(arc): leave a role swap's nodes and regraded servers alone in node-recovery

README.md

@@ -189,6 +189,8 @@ The watcher only acts on a node whose kubelet has gone silent (`Unknown`), never
 
 The threshold is also where a stopped node's runner pods are given up, earlier than the five-minute default eviction would. Their containers stopped with the node, so their jobs are lost either way unless the node returns within that time; lower the threshold for faster recovery, or raise it to wait longer for a node that is only asleep.
 
+It also leaves alone the nodes a voter swap or a regrading step is moving through, since those stay silent on purpose: while the Lease `kube-system/fleet-regrade` is unexpired it neither taints nor prunes a node its `github-runner.exadev.com/swap-nodes` annotation lists or its holder names, and it never taints a node carrying the regrade level label. See the role README's Node recovery section.
+
 Pruning is opt-in (`github_runner_arc_node_recovery_prune_after_seconds`, empty by default). When it is set, the watcher also deletes the Node object of a node that has been `Unknown` for that long and carries the `github-runner.exadev.io/removable=true` label (`github_runner_arc_node_recovery_prune_label`), so a device that is never coming back stops being listed. It never deletes an unlabelled node, a node that is a control-plane or etcd member (even if labelled), its own node or a node without a `Ready` condition, and it deletes nothing unless at least three `Ready` nodes carrying the etcd role label would remain, because k3s removes a server's etcd member when its Node is deleted. It sends the delete with the node's `uid` and `resourceVersion` as preconditions so a node that has come back since it was listed is left alone. Turning it on gives the watcher a cluster-wide `delete` on nodes. Its pod is then confined to control-plane nodes, since a token that can delete nodes should not be readable from a worker. A pruned node comes back only when its k3s agent process restarts: a running kubelet does not register itself again once its Node object is gone, so pruning is safe only for nodes that restart their agent when their Node vanishes or that reboot, which the node has to arrange. See the role README's Node recovery section for the details.
 
 Its image, `github_runner_arc_node_recovery_image`, is pulled without a pull Secret, so the published package must be public.

roles/github_runner_arc/README.md

@@ -55,7 +55,7 @@ Helm 4 and server-side apply. The role reads `helm version --short` and, on Helm
 - `github_runner_arc_image_pull_registry`, `github_runner_arc_image_pull_secret_name`, `github_runner_arc_verify_image_pull`: the pull Secret's registry and name, and whether to prove the credential before writing it.
 - `github_runner_arc_heartbeat_gist_id`: install the fleet-health platform from this host, refreshing this gist. `github_runner_arc_heartbeat_bootstrap_gist`, `github_runner_arc_heartbeat_gist_description` and `github_runner_arc_heartbeat_gist_consumer` control the bootstrap described above.
 - `github_runner_arc_autoscaler_usable_budget_gi`, `github_runner_arc_autoscaler_floor`: required when a profile sets `autoscale: true` and the platform is installed. The floor must equal the autoscaled profile's `maxRunners`, which every Helm upgrade reverts to. The autoscaler works its ceiling out from the nodes on every poll (see the project README's Autoscaler section); `github_runner_arc_autoscaler_max_ceiling`, optional, is an upper bound on it and must be no lower than the floor. The other `github_runner_arc_autoscaler_*` and `github_runner_arc_heartbeat_*` settings have defaults; see `defaults/main.yml`.
-- `github_runner_arc_node_recovery_enabled` (default `true`), `github_runner_arc_node_recovery_after_seconds` (default 120), `github_runner_arc_node_recovery_poll_seconds` (default 15), `github_runner_arc_node_recovery_prune_after_seconds` (default empty, which leaves pruning off), `github_runner_arc_node_recovery_prune_label` (default `github-runner.exadev.io/removable`), `github_runner_arc_node_recovery_image`: the node recovery watcher (see [Node recovery](#node-recovery)).
+- `github_runner_arc_node_recovery_enabled` (default `true`), `github_runner_arc_node_recovery_after_seconds` (default 120), `github_runner_arc_node_recovery_poll_seconds` (default 15), `github_runner_arc_node_recovery_prune_after_seconds` (default empty, which leaves pruning off), `github_runner_arc_node_recovery_prune_label` (default `github-runner.exadev.io/removable`), `github_runner_arc_node_recovery_level_label` (default the cluster role's `github_runner_cluster_regrade_level_label`, else `github-runner.exadev.com/regrade-level`), `github_runner_arc_node_recovery_image`: the node recovery watcher (see [Node recovery](#node-recovery)).
 - `github_runner_arc_role_controller_enabled` (default the cluster role's `github_runner_cluster_role_controller`, itself `false`), `github_runner_arc_role_controller_image`, `github_runner_arc_role_controller_sample_seconds` (default 15), `github_runner_arc_role_controller_voters` (default 3), `github_runner_arc_role_controller_max_servers`, `github_runner_arc_role_controller_election_timeout_ms`, `github_runner_arc_role_controller_pressure_threshold`, `github_runner_arc_role_controller_available_floor_percent`, `github_runner_arc_role_controller_eligible_label`, `github_runner_arc_role_controller_pinned_label`, `github_runner_arc_role_controller_level_label`, `github_runner_arc_role_controller_link_label`, `github_runner_arc_role_controller_probe_configmap`, `github_runner_arc_role_controller_status_configmap`, `github_runner_arc_role_controller_data_dir`: the role controller, which reports which hosts are fit to hold an etcd vote (see [Role controller](#role-controller)). Those that mirror the cluster role (`voters`, `max_servers`, `election_timeout_ms`, `pressure_threshold`, `available_floor_percent`, `eligible_label`, `pinned_label`, `level_label`, `link_label`) take the cluster role's own variable when it is defined.
 - `github_runner_arc_runner_reaper_enabled` (default `false`), `github_runner_arc_runner_reaper_max_job_seconds` (default 432000, five days), `github_runner_arc_runner_reaper_completed_grace_seconds` (default 300), `github_runner_arc_runner_reaper_schedule` (default every 5 minutes), `github_runner_arc_runner_reaper_dry_run` (default `false`), `github_runner_arc_runner_reaper_image` (default the renewal's image): the runner reaper (see [Runner reaper](#runner-reaper)).
 - `github_runner_arc_runner_active_deadline_seconds` (default unset): `activeDeadlineSeconds` on every runner pod (see [Runner reaper](#runner-reaper)).
@@ -473,6 +473,8 @@ github_runner_arc_orgs:
 
 Every host that installs anything also installs a small watcher, the `node-recovery` Deployment in `github_runner_arc_platform_namespace`, so that a node that stops does not leave its pods `Terminating` for ever. ARC waits for a listener's old pod to go before starting a new one, so a listener on a stopped node otherwise stalls its scale set until the pod is force-deleted by hand. Once a node's `Ready` condition has been `Unknown` for `github_runner_arc_node_recovery_after_seconds`, the watcher gives it the `node.kubernetes.io/out-of-service=nodeshutdown:NoExecute` taint of Kubernetes' non-graceful node shutdown, which makes the control plane evict the node's pods and force-delete the terminating ones; it removes the taint when the node reports `Ready` again. It never acts on a node reporting `NotReady`, never on its own node, and never removes a taint it did not apply. Its ClusterRole allows `get`, `list` and `patch` on nodes and nothing else, unless pruning is switched on (below). `github_runner_arc_node_recovery_enabled: false` removes the watcher, its RBAC and any taint it left behind. Its image is pulled without a pull Secret, so it must be public. The repository README's Node recovery section explains the default threshold.
 
+The watcher also leaves alone the nodes a voter swap or a regrading step is moving through, because those are meant to be silent for a while: the host stops one k3s unit and starts the other, or a server runs at etcd-only with no kubelet, and an out-of-service taint would evict pods that the controller is draining on purpose. It reads the Lease `kube-system/fleet-regrade`, which serialises both, through a Role that allows `get` on that one name and nothing else in the namespace. While the Lease is unexpired (its renew time plus its duration has not passed, the rule `lease_state` applies in the cluster role's regrade core) it neither taints nor prunes a node that the Lease's `github-runner.exadev.com/swap-nodes` annotation lists (comma separated) or that its holder names, which is a node name when regrading holds the Lease. A node carrying the regrade level label (`github_runner_arc_node_recovery_level_label`, `github-runner.exadev.com/regrade-level` unless the cluster role is configured otherwise) is never tainted, whether or not any Lease is held. A taint the watcher applied earlier is still removed once such a node reports `Ready`. A Lease that cannot be read fails the pass before it changes anything, since tainting on a guess could cut a swap short.
+
 Pruning is off by default. Setting `github_runner_arc_node_recovery_prune_after_seconds` to a whole number of seconds, greater than `github_runner_arc_node_recovery_after_seconds`, makes the watcher delete the Node object of a node whose `Ready` condition has been `Unknown` that long and which carries the label `github_runner_arc_node_recovery_prune_label` with the value `true`. The variable has no default value, since how long a node may be away before it is given up is the fleet's own decision. Use it for nodes that will not come back for a long time, such as agents on small devices that are unplugged, so that they stop being listed. The label is the node's own to set (for a k3s agent, `--node-label` on its command line, which it applies again every time it registers); a node without it is never deleted, and neither is a node with a `node-role.kubernetes.io/control-plane` or `node-role.kubernetes.io/etcd` label even if it also carries the removable one, because deleting a k3s server's Node removes its etcd member; the pass fails saying so until the label is removed. The watcher's own node and a node with no `Ready` condition are never deleted either. The pass also deletes nothing unless at least three `Ready` nodes carrying the `node-role.kubernetes.io/etcd` label would remain, three being the voters a cluster needs to tolerate losing one (a non-voting agent does not count, nor does a member that is not `Ready`); below that it keeps the node, fails the pass and says why, so a cluster with no etcd members, such as a single-server SQLite one, is never pruned. The floor is a constant in the script, not a setting. It exists because k3s removes a server's etcd member when that server's Node is deleted (`pkg/etcd/member_controller.go` in k3s: `Removing etcd member from cluster due to node delete`, done only for a Node carrying the etcd role label), so a prune must never be the thing that thins the voters.
 
 Turning pruning on adds `delete` on nodes to the watcher's ClusterRole. RBAC cannot select nodes by label, so that is a cluster-wide delete on nodes; the script is what narrows it. Because the script's checks are client-side, anyone who can read the pod's ServiceAccount token could call the API directly, so with pruning on the Deployment is confined to control-plane nodes (a required node affinity on `node-role.kubernetes.io/control-plane`, and a toleration for that taint) and not left to land on a lower-trust worker. That needs a control-plane node that can run the pod; with pruning off the pod stays unpinned. The delete is a request to the API carrying the node's `uid` and `resourceVersion` as preconditions (`kubectl delete` cannot send them, so the watcher image also contains `curl`), so a node that came back between being listed and being deleted is left alone.

roles/github_runner_arc/defaults/main.yml

@@ -186,6 +186,8 @@ github_runner_arc_node_recovery_poll_seconds: 15
 github_runner_arc_node_recovery_prune_after_seconds: ""
 # The label a node must carry, with the value true, to be pruned. Set it on the nodes that may go (for a k3s agent, --node-label on its command line, which it applies again each time it registers), never on a server.
 github_runner_arc_node_recovery_prune_label: github-runner.exadev.io/removable
+# The label regrading puts on a server it has stepped down, with the level as its value, which the watcher never taints: the cluster role's github_runner_cluster_regrade_level_label when that is defined, else github-runner.exadev.com/regrade-level (held together with the role controller's by the same expression).
+github_runner_arc_node_recovery_level_label: "{{ github_runner_cluster_regrade_level_label | default('github-runner.exadev.com/regrade-level') }}"
 # Image the watcher runs. Pulled without a pull Secret, so it must be public.
 github_runner_arc_node_recovery_image: "ghcr.io/exadev/github-runner-node-recovery:2.31.0"
 

roles/github_runner_arc/meta/argument_specs.yml

@@ -371,6 +371,9 @@ argument_specs:
         type: str
         default: github-runner.exadev.io/removable
         description: The label key a node must carry, with the value true, to be pruned. A Kubernetes qualified name.
+      github_runner_arc_node_recovery_level_label:
+        type: str
+        description: The label regrading puts on a server it has stepped down, which the node recovery watcher never taints, from github_runner_cluster_regrade_level_label when defined, else github-runner.exadev.com/regrade-level.
       github_runner_arc_node_recovery_image:
         type: str
         description: Image the node recovery watcher runs, pulled without a pull Secret.

roles/github_runner_arc/templates/node-recovery.yaml.j2

@@ -1,7 +1,8 @@
-{# The node recovery watcher (see tasks/install_node_recovery.yml and scripts/node-recovery.sh): a ServiceAccount, a ClusterRole allowing get, list and patch on nodes and, only while github_runner_arc_node_recovery_prune_after_seconds is set, delete, its binding, and the Deployment. The namespace is ensured separately, since the platform's other Deployments share it. #}
+{# The node recovery watcher (see tasks/install_node_recovery.yml and scripts/node-recovery.sh): a ServiceAccount, a ClusterRole allowing get, list and patch on nodes and, only while github_runner_arc_node_recovery_prune_after_seconds is set, delete, its binding, a Role allowing get on the one Lease that serialises voter swaps and regrading with its binding, and the Deployment. The namespace is ensured separately, since the platform's other Deployments share it. #}
 {% set name = 'node-recovery' %}
 {% set prune = github_runner_arc_node_recovery_prune_after_seconds | string | length > 0 %}
 {% set cluster_name = 'github-runner-node-recovery' %}
+{% set lease_role = 'github-runner-node-recovery-lease' %}
 {% set labels = {'app.kubernetes.io/name': name, 'app.kubernetes.io/part-of': 'github-runner'} %}
 apiVersion: v1
 kind: ServiceAccount
@@ -35,6 +36,34 @@ roleRef:
   kind: ClusterRole
   name: {{ cluster_name | to_json }}
 ---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: {{ lease_role | to_json }}
+  namespace: {{ github_runner_arc_node_recovery_lease_namespace | to_json }}
+  labels: {{ labels | to_json }}
+rules:
+  # Reading the Lease is how the watcher learns which nodes a swap or a regrade step is moving through. Named, so it can read no other Lease.
+  - apiGroups: ["coordination.k8s.io"]
+    resources: ["leases"]
+    resourceNames: [{{ github_runner_arc_node_recovery_lease_name | to_json }}]
+    verbs: ["get"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: {{ lease_role | to_json }}
+  namespace: {{ github_runner_arc_node_recovery_lease_namespace | to_json }}
+  labels: {{ labels | to_json }}
+subjects:
+  - kind: ServiceAccount
+    name: {{ name | to_json }}
+    namespace: {{ github_runner_arc_platform_namespace | to_json }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: {{ lease_role | to_json }}
+---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -92,6 +121,14 @@ spec:
               value: {{ github_runner_arc_node_recovery_poll_seconds | string | to_json }}
             - name: NODE_RECOVERY_ANNOTATION
               value: {{ github_runner_arc_node_recovery_annotation | to_json }}
+            - name: NODE_RECOVERY_LEASE_NAMESPACE
+              value: {{ github_runner_arc_node_recovery_lease_namespace | to_json }}
+            - name: NODE_RECOVERY_LEASE_NAME
+              value: {{ github_runner_arc_node_recovery_lease_name | to_json }}
+            - name: NODE_RECOVERY_SWAP_NODES_ANNOTATION
+              value: {{ github_runner_arc_node_recovery_swap_nodes_annotation | to_json }}
+            - name: NODE_RECOVERY_LEVEL_LABEL
+              value: {{ github_runner_arc_node_recovery_level_label | to_json }}
 {% if prune %}
             - name: NODE_RECOVERY_PRUNE_AFTER_SECONDS
               value: {{ github_runner_arc_node_recovery_prune_after_seconds | string | to_json }}

roles/github_runner_arc/vars/main.yml

@@ -13,6 +13,11 @@ github_runner_arc_pull_secret_field_manager: github-runner-pull-secret-renewer
 # Annotation the node recovery watcher writes alongside the out-of-service taint it applies, recording when it applied it. The watcher removes only a taint carrying this annotation, so an operator's own out-of-service taint is left alone.
 github_runner_arc_node_recovery_annotation: github-runner.exadev/out-of-service-applied-at
 
+# The Lease that serialises voter swaps and regrading steps (the cluster role's fleet-regrade-core.sh, LEASE_NAMESPACE and LEASE_NAME), and the annotation on it that lists the nodes a swap is moving through, which the role controller writes: the node recovery watcher reads them to leave those nodes alone, and its Role grants get on that one name.
+github_runner_arc_node_recovery_lease_namespace: kube-system
+github_runner_arc_node_recovery_lease_name: fleet-regrade
+github_runner_arc_node_recovery_swap_nodes_annotation: github-runner.exadev.com/swap-nodes
+
 # Whether the runner pods and the platform's Deployments reference a pull Secret at all. With github_runner_arc_image_pull_secret_source none every image they run is public, so the role writes, checks and references none, and removes one an earlier run wrote.
 github_runner_arc_pull_secret_in_use: "{{ github_runner_arc_image_pull_secret_source != 'none' }}"