Repository navigation
Conversation
Reads each node's probe entry and the Nodes, and reports per host whether it is fit, unfit or unknown with the numbers behind it, and the swap of an unfit voter for a fit standby that it would make. It changes nothing in the cluster: its only write is its own status ConfigMap. Every limit derives from the election timeout, the tail percentile or a named fact about etcd or k3s. The unit tests hold each to its boundary, and the memory limit to the regrade core's own is_over.
…lth for the controller A per-node probe appends one sample per period to its key of a shared ConfigMap and keeps the last window of them: the TCP connect time to every voter's peer port, what a voter's etcd metrics gained since the last sample, an fdatasync reading on a node that is not a voter, the memory signals the regrade core reads, and whether the default route leaves by wired, Wi-Fi or an interface whose link cannot be seen. The tests read the link from a sysfs tree whose links loop as the kernel's do and from the host's own sysfs where there is one, the connect times from real listeners, etcd's metrics from pages a real three-member cluster served (and a live local cluster when an etcd binary is available), and the disk reading from the real fio.
One image holds the controller, which its entrypoint runs on an interval, and the probe, which the DaemonSet starts by name. It carries fio, the disk check etcd's own documentation uses, and the regrade core whose functions the probe reads memory with. The unit job installs fio so that the probe's fdatasync reading is tested with the real tool.
…n by hand `fleet-role-probe.sh check link|connect|memory|etcd|fsync` prints one reading and exits without touching the cluster, so that every probe can be run on a real host and its output set beside a reading taken there by hand.
A fleet-role Deployment and a fleet-role-probe DaemonSet in the platform namespace, each with a ServiceAccount that can get and list nodes and reach only its own ConfigMaps by name, and neither able to change a node. The controller's settings follow the cluster role's own where that role defines them, with fallbacks the tests hold to its defaults, and the target is refused unless it is odd, at least three and a place below the maximum number of servers.
…toscaler's The heartbeat reads the controller's status ConfigMap, and is allowed that one by name, only when the role controller is installed, and publishes it to the gist while the fleet is vouched for, on the same terms as the autoscaler's status.
…he controller The role works out the cluster's election timeout once and gives the same figure to the controller and the probe, so neither script holds a default of its own that could differ from the other.
Mearman
changed the title
A probe on any other node recorded a window of samples nobody reads, and the shared ConfigMap, which holds at most a mebibyte, would have filled with the size of the fleet. Such a node now records nothing and loses an entry it had, so the ConfigMap grows with the hosts that decide the control plane.
…to act A voter that is paused or unreachable stops reporting while its last sample still looks healthy, so the gate on etcd's state read stale gauges. It now says the probe has not reported, as the reason nothing would change.
Linux limits one argument to a process to 128 KiB, and the entries of a few voters together pass it, so the controller failed with an argument list too long on a runner while it passed on macOS. The tests now run with a jq that refuses such an argument, and hold both scripts to it with inputs the size of a full ConfigMap.
…runs The role controller's probe runs from its own image, whose Dockerfile contents are held by the role controller's tests instead.
…er_role_controller The opt-in is the cluster role's variable, off by default, which the ARC role takes as the default of its own switch, so one inventory setting turns on the probe and the controller for the group. The tests hold that switch and the names the status is published under.
…on pull requests A build-only job builds fleet-role/Dockerfile as the release does and runs what the pods run from the image as built. The unit job installs the etcd v3.6.15 release, checked against the SHA-256 its SHA256SUMS asset publishes before it is extracted, for the probe's live test.
…the voter it replaces A slow link makes both its ends unfit, so ranking unfit voters alone could send a healthy peer out in place of the one on the poor path, and a standby that measures the slow voter slow was never fit. The voter on the most slow links is now the one to replace, nothing is swapped when it is pinned, and a candidate is judged against the voters that would remain.
…o it is up etcd keeps one gauge per peer that counts its active streams, up to two, so a follower with both streams to the leader and one to the other follower read as connected to one peer and the gate held every swap back. The fixture is a page a follower of a real three-server k3s cluster served.
…refuses one A probe on a poor path made four sequential calls a sample and went stale, and its process exited on the first refusal and came back as a restart. It now reads its own entry once, reads the nodes once a sample, and a patch the API will not take is tried again next period with the first-sample restart mark kept until a sample is recorded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #182
The role controller of #179, in dry-run: it collects the fitness signals and reports what it would do, and changes nothing in the cluster.
fleet-role-probe(a DaemonSet on every Linux node with a kubelet, host network) appends one sample every period to its key of thefleet-role-probeConfigMap: TCP connect time to each voter's peer port, what a voter's etcd metrics gained since the last sample (WAL fsync, backend commit, per-peer round trip, leader changes, leader/learner/known-member gauges), a fio fdatasync reading once per window on a non-voter (a node that is neither a voter nor marked eligible records nothing, so the ConfigMap grows with the hosts that decide the control plane), the regrade core's own memory readings, and whether the default route leaves by wired, Wi-Fi or an interface whose link can't be seen.fleet-role(Deployment) reads those and the Nodes and publishesfleet-role-status, which the heartbeat republishes beside the autoscaler's. Per host it gives fit, unfit or unknown with every reason and its numbers, and the swap it would make or why it would not.Same form as
node-recoveryandpsi-reporter: bash, jq and kubectl in an Alpine image built by the release workflow, installed by the ARC role (github_runner_arc_role_controller_enabled, off by default), ClusterRoles that can only get and list nodes, ConfigMap access by name.Thresholds are derived as the epic states them (a tenth and a twentieth of the election timeout, 10 ms and 25 ms disk, memory against the regrade limit and the storage quota, a window of 100 samples = the fewest that give a p99 its own sample, cooldown = k3s's learner stall time plus a window). The tests hold each to its boundary, the memory limit to the regrade core's
is_over, and the probes to the real thing: a sysfs tree whose links loop like the kernel's (and the host's own/syswhere there is one), real listeners and curl, pages from a real etcd 3.6.15 three-member cluster (and a live local one whenetcdis on the PATH), the real fio.Not in this PR: the inventory variables that publish the eligible and pinned labels (they touch the cluster role files #180 is changing, so the controller reads two Node labels and the target from the ARC role until then), the mesh scenario with
tc netem, and the readings compared with hand readings on real hosts, which are below.Real-host checks (read-only)
Each reading alone, from inside the probe pod on the node, with nothing written to the cluster (
fsyncwrites and unlinks a small file in the probe's own directory). Compare each with the hand reading beside it. Pod for a node:kubectl -n github-runner-platform get pod -l app.kubernetes.io/name=fleet-role-probe --field-selector spec.nodeName=NODE -o name.Then, with the role controller on for at least one window,
kubectl -n github-runner-platform get cm fleet-role-status -o jsonpath='{.data.status\.json}' | jq .against the hosts, andkubectl -n github-runner-platform logs deploy/fleet-role.