From 5fe4b9982c2749da341dc320f27c8d1ddb59dcc7 Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 3 Nov 2026 18:04:29 +0000 Subject: [PATCH 1/2] feat(role): attribute each unfit verdict to the node at fault for it A slow link makes both of its ends unfit, so on a fleet where one voter is on a poor path every voter reads unfit and the status said nothing about which. Each host now carries attributed_to, and the status an attribution map of the hosts each node accounts for: a host's own disk, memory, readiness and restarts are its own fault, and its round trip is the fault of the end of the slow link whose links to the other voters are slower too, or of both ends when nothing tells them apart. The log names the attribution beside each unfit verdict. The voter to replace was the unfit voter on the most slow links, which ties when every link is over the limit and then fell to the worst-ranked voter, the one that measured the slowest tail, not the one at fault. It is now the unfit voter the most unfit verdicts are attributed to. --- README.md | 2 +- scripts/fleet-role.jq | 48 ++++++++++++++++++++++++----- scripts/fleet-role.sh | 2 +- tests/unit/test_fleet_role.py | 58 +++++++++++++++++++++++++++++++++-- 4 files changed, 98 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index b8671bfa..9b646734 100644 --- a/README.md +++ b/README.md @@ -208,7 +208,7 @@ The controller reads the Nodes and the probes' samples every sample period and p - A window is 100 samples, the fewest that give the 99th percentile a sample of its own, so one outlier never decides; a host with fewer is `unknown`, as is one whose probe has not reported for three sample periods. Readings from before the last change of the voter set are not counted. - Hosts that pass are ranked wired before Wi-Fi before unknown, then by round trip, then by disk. The link ranks hosts and is never a gate; the status gives each host's link and where it was read (`link` and `link_source`), and the log names it beside the verdict. -A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter on the most slow links (a link is slow when either end measures it above the limit), the worst-ranked among those on as many. A swap is reported only when that voter is eligible and not pinned (when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped) and a fit eligible standby ranks above it. A candidate is judged against the voters that would remain, since a slow link to the very voter it would replace says nothing against it. It is held back, with the reason in the status and the log, while the voter count is not `github_runner_arc_role_controller_voters`, any voter is below `full`, is not Ready, has no leader, is a learner or knows a member that is not a voter, is not connected to every other voter, fewer hosts are eligible than the target, or the cooldown after a change of the voter set has not passed (k3s's five minutes to remove a stalled learner plus one window). The target must be odd, at least 3, and at most `github_runner_cluster_max_servers` minus one, which leaves a swap one place of headroom; the role refuses anything else. A host is eligible, and a voter pinned, by a Node label with the value `true` (`github_runner_arc_role_controller_eligible_label` and `_pinned_label`), which the cluster role publishes from `github_runner_cluster_voter_eligible` and `github_runner_cluster_voter_pinned` per host (and the target from `github_runner_cluster_voters`, refused unless odd, at least 3 and below `github_runner_cluster_max_servers`; see the cluster role's README); a voter not marked eligible is never swapped out, since nothing has prepared it to come back. +A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so every unfit verdict is attributed to the node at fault for it (`attributed_to` on each host, and `attribution`, the hosts each node accounts for). A host's own disk, memory, readiness or restarts are its own fault. Its round trip is the link to its worst peer, and the end at fault for a slow link is the one whose links to the other voters are slower too, compared by the best of them, since a node on a poor path is slow to every peer and a node on a good one is fast to its others; when the two ends' other links are as fast as each other's, or one end has none, the link is attributed to both. The voter to replace is the unfit voter the most hosts' unfit verdicts are attributed to, the worst-ranked among those with as many, and the log names the attribution beside each unfit verdict. A swap is reported only when that voter is eligible and not pinned (when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped) and a fit eligible standby ranks above it. A candidate is judged against the voters that would remain, since a slow link to the very voter it would replace says nothing against it. It is held back, with the reason in the status and the log, while the voter count is not `github_runner_arc_role_controller_voters`, any voter is below `full`, is not Ready, has no leader, is a learner or knows a member that is not a voter, is not connected to every other voter, fewer hosts are eligible than the target, or the cooldown after a change of the voter set has not passed (k3s's five minutes to remove a stalled learner plus one window). The target must be odd, at least 3, and at most `github_runner_cluster_max_servers` minus one, which leaves a swap one place of headroom; the role refuses anything else. A host is eligible, and a voter pinned, by a Node label with the value `true` (`github_runner_arc_role_controller_eligible_label` and `_pinned_label`), which the cluster role publishes from `github_runner_cluster_voter_eligible` and `github_runner_cluster_voter_pinned` per host (and the target from `github_runner_cluster_voters`, refused unless odd, at least 3 and below `github_runner_cluster_max_servers`; see the cluster role's README); a voter not marked eligible is never swapped out, since nothing has prepared it to come back. The controller and the probe run one image, `github_runner_arc_role_controller_image`, pulled without a pull Secret, so the published package must be public. diff --git a/scripts/fleet-role.jq b/scripts/fleet-role.jq index bce988d8..6f3a3d0c 100644 --- a/scripts/fleet-role.jq +++ b/scripts/fleet-role.jq @@ -123,13 +123,18 @@ $config as $c etcd: ($entry.samples[-1].etcd // null) }; - # What makes a measured host unfit, as sentences carrying their numbers; empty when nothing does. A voter is judged by the unfit round trip limit and the disk it serves etcd from, a candidate by the stricter fit limit and a disk probe. - def faults($m): + # What makes a measured host unfit, as sentences carrying their numbers; empty when nothing does. A voter is judged by the unfit round trip limit and the disk it serves etcd from, a candidate by the stricter fit limit and a disk probe. The round trip is a link's, and another node may be at fault for it; every other fault is the host's own. + def rtt_faults($m): $m.node as $node | [ (if $m.rtt_ms != null and $m.rtt_ms > (if $node.voter then $unfit_rtt_ms else $fit_rtt_ms end) then "round trip p99 to \($m.rtt_peer) is \($m.rtt_ms | round2) ms, over \(if $node.voter then $unfit_rtt_ms else $fit_rtt_ms end) ms (a \(if $node.voter then "tenth" else "twentieth" end) of the \($c.election_timeout_ms) ms election timeout)" - else empty end), + else empty end) + ]; + + def own_faults($m): + $m.node as $node + | [ (if $node.voter then (if $m.wal_p99_ms != null and $m.wal_p99_ms >= wal_fsync_limit_ms then "WAL fsync p99 is \($m.wal_p99_ms | round2) ms, not under \(wal_fsync_limit_ms) ms" else empty end), (if $m.commit_p99_ms != null and $m.commit_p99_ms >= backend_commit_limit_ms then "backend commit p99 is \($m.commit_p99_ms | round2) ms, not under \(backend_commit_limit_ms) ms" else empty end) @@ -142,6 +147,8 @@ $config as $c (if $m.restarts > 0 then "k3s restarted \($m.restarts) time(s) in the window" else empty end) ]; + def faults($m): rtt_faults($m) + own_faults($m); + # What is missing for a verdict, as sentences; empty when the readings are all there. def gaps($m): $m.node as $node @@ -187,7 +194,31 @@ $config as $c m: $m}; # Hosts reported: every voter, and every eligible host that is not one. - ([$all[] | select(.voter or .eligible)] | sort_by(.name) | map(host(.))) as $hosts + ([$all[] | select(.voter or .eligible)] | sort_by(.name) | map(host(.))) as $measured + +# Every link a probe measured, between two hosts, as the worse of the two ends' tails (a standby measures its links to the voters alone, so those have one end's). +| ([$measured[] | .name as $a | .m.pairs[] | select(.ms != null) | {ends: ([$a, .peer] | sort), ms}] + | group_by(.ends) | map({ends: .[0].ends, ms: (map(.ms) | max)})) as $links + +# The best of a host's links to the voters other than $excluded, in milliseconds: how fast its path is where the link in question plays no part. Links to the voters alone, since every host measures those (a voter's to a standby is the standby's one-sided reading). Null when it has no other. +| def elsewhere($host; $excluded): + [$links[] | select((.ends | index($host)) != null and (.ends | index($excluded)) == null) + | select(.ends | map(select(. != $host)) | .[0] as $other | $voter_names | index($other) != null) | .ms] | min; + + # The end at fault for a slow link between $host and $peer: the one whose other links are slower too, since a node on a poor path is slow to every peer and a node on a good one is fast to its others. Both ends when their other links are as fast as each other's, or one has none to compare. + def at_fault($host; $peer): + elsewhere($host; $peer) as $own + | elsewhere($peer; $host) as $theirs + | if $own == null or $theirs == null or $own == $theirs then [$host, $peer] | sort + elif $theirs > $own then [$peer] + else [$host] end; + + # The nodes an unfit verdict is attributed to: for its round trip, the end at fault for the slow link to its worst peer; for any other fault (its own disk, memory, readiness or restarts), the host itself. Empty for a verdict that is not unfit. + def attributed($h): + if $h.verdict != "unfit" then [] + else (if (rtt_faults($h.m) | length) > 0 then at_fault($h.name; $h.m.rtt_peer) else [] end) + (if (own_faults($h.m) | length) > 0 then [$h.name] else [] end) | unique end; + + ($measured | map(. + {attributed_to: attributed(.)})) as $hosts | ([$hosts[] | select(.eligible)] | length) as $eligible_count | ($all | map(select(.voter))) as $voter_nodes @@ -207,9 +238,11 @@ $config as $c (if $cooldown_until != null and $now < $cooldown_until then "the membership changed at \($changed_at), and the cooldown lasts until \($cooldown_until)" else empty end) ]) as $blocks -# The swap. A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter on the most slow links (a link is slow when either end measures it above the unfit limit), the worst-ranked among those on as many. It goes out only if it is eligible and not pinned; when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped. The best fit eligible standby that ranks above it comes in, judged against the voters that would remain. -| ([$hosts[] | select(.role == "voter") | .name as $a | .m.pairs[] | select(.ms != null and .ms > $unfit_rtt_ms) | [$a, .peer] | sort] | unique) as $slow_links -| ([$hosts[] | select(.role == "voter" and .verdict == "unfit") | . + {slow_links: (.name as $n | [$slow_links[] | select(index($n) != null)] | length)}] | sort_by(.rank) | reverse | sort_by(-.slow_links) | first) as $culprit +# Which hosts' unfit verdicts each node accounts for, by node. +| ([$hosts[] | .name as $h | .attributed_to[] | {node: ., host: $h}] | group_by(.node) | map({key: .[0].node, value: map(.host)}) | from_entries) as $attribution + +# The swap. A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter the most hosts' unfit verdicts are attributed to, the worst-ranked among those with as many. It goes out only if it is eligible and not pinned; when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped. The best fit eligible standby that ranks above it comes in, judged against the voters that would remain. +| ([$hosts[] | select(.role == "voter" and .verdict == "unfit") | . + {accounts_for: ($attribution[.name] // [] | length)}] | sort_by(.rank) | reverse | sort_by(-.accounts_for) | first) as $culprit | (if $culprit != null and $culprit.eligible and ($culprit.pinned | not) then $culprit else null end) as $out | ([$hosts[] | select(.role == "standby" and .eligible) | (if $out == null then .m else (.m | without($out.name)) end) as $m @@ -238,6 +271,7 @@ $config as $c quota_bytes: $quota_bytes}, membership: {voters: $voter_names, changed_at: $changed_at, cooldown_until: $cooldown_until}, hosts: ($hosts | map({key: .name, value: del(.name, .rank, .m)}) | from_entries), + attribution: $attribution, ranking: ($hosts | map(select(.verdict == "fit")) | sort_by(.rank) | map(.name)), blocks: $blocks, decision: $decision diff --git a/scripts/fleet-role.sh b/scripts/fleet-role.sh index f89477d4..6f8b64de 100755 --- a/scripts/fleet-role.sh +++ b/scripts/fleet-role.sh @@ -80,7 +80,7 @@ kubectl patch configmap "$status_configmap" -n "$namespace" --type merge --patch # What each host's verdict rests on, then the decision, in the log. while IFS= read -r line; do log "$line"; done < <(jq -r ' .hosts | to_entries[] - | "\(.key) (\(.value.role)\(if .value.pinned then ", pinned" else "" end)\(if .value.eligible then "" else ", not eligible" end), link \(.value.link)): \(.value.verdict), \(.value.reasons | if length == 0 then "all readings within their limits" else join("; ") end)"' <<< "$result") + | "\(.key) (\(.value.role)\(if .value.pinned then ", pinned" else "" end)\(if .value.eligible then "" else ", not eligible" end), link \(.value.link)): \(.value.verdict), \(.value.reasons | if length == 0 then "all readings within their limits" else join("; ") end)\(if (.value.attributed_to | length) > 0 then " (attributed to \(.value.attributed_to | join(" and ")))" else "" end)"' <<< "$result") while IFS= read -r line; do log "$line"; done < <(jq -r ' if .decision.action == "swap" then "DRY RUN: would swap out \(.decision.swap.out) for \(.decision.swap.in): \(.decision.because | join("; "))" else "DRY RUN: would not act: \(.decision.because | join("; "))\(if .decision.wanted == null and (.blocks | length) > 0 then " (also: \(.blocks | join("; ")))" else "" end)" end' <<< "$result") diff --git a/tests/unit/test_fleet_role.py b/tests/unit/test_fleet_role.py index 9c46a1d2..7da78b35 100644 --- a/tests/unit/test_fleet_role.py +++ b/tests/unit/test_fleet_role.py @@ -384,7 +384,7 @@ def test_a_voter_not_marked_eligible_is_never_swapped_out(self) -> None: def slow_voter_c(self, **changes: dict[str, Any]) -> list[Host]: """Voter c has a slow link to every other host, as a voter on a poor path does: both ends of each of its links measure it slow, so a and b are unfit too, by the rule that a voter is unfit when its round trip to any other voter is.""" hosts = healthy( - a={"tcp": {"b": 5.0, "c": self.SLOW + 300}}, # the worst single reading is a's, but c is on the most slow links + a={"tcp": {"b": 5.0, "c": self.SLOW + 300}}, # the worst single reading is a's, but c is the end slow to every other voter b={"tcp": {"a": 5.0, "c": self.SLOW}}, c={"tcp": {"a": self.SLOW, "b": self.SLOW}}, d={"tcp": {"a": 5.0, "b": 5.0, "c": self.SLOW}}, @@ -394,10 +394,11 @@ def slow_voter_c(self, **changes: dict[str, Any]) -> list[Host]: setattr(host, key, value) return hosts - def test_a_slow_link_makes_both_its_ends_unfit_and_the_voter_on_the_most_slow_links_is_replaced(self) -> None: + def test_a_slow_link_makes_both_its_ends_unfit_and_the_voter_at_fault_is_replaced(self) -> None: status = self.pass_(self.slow_voter_c()) self.assertEqual(self.verdicts(status), {"a": "unfit", "b": "unfit", "c": "unfit", "d": "unfit"}) - self.assertEqual(status["decision"]["swap"], {"out": "c", "in": "d"}, "c is on two slow links, a and b on one each") + self.assertEqual(status["decision"]["swap"], {"out": "c", "in": "d"}, "c is slow to a and b, which reach each other fast") + self.assertEqual(status["attribution"], {"c": ["a", "b", "c", "d"]}) def test_a_candidate_is_judged_against_the_voters_that_would_remain(self) -> None: # d measures c slow, as everyone does, and is unfit against the full set; without c it is fit, which is what the swap needs. @@ -414,6 +415,57 @@ def test_nothing_is_swapped_when_the_voter_at_fault_is_pinned_even_though_its_pe self.assertEqual(status["decision"]["action"], "none") self.assertEqual(status["decision"]["because"], ["voter c is the most at fault and is pinned or not marked eligible, so it is not replaced"]) + def test_when_every_link_is_slow_the_voter_whose_other_links_are_slow_too_is_at_fault(self) -> None: + # As on a live fleet: a is on a poor path, slow to everyone, and b to c is only just over the limit. Every voter is unfit and every link is slow, so counting slow links ties, and b, which measures the worst single tail, ranks worst; the slowness follows a. + just_over = UNFIT_RTT_MS + 10 + hosts = healthy( + a={"tcp": {"b": self.SLOW, "c": self.SLOW}}, + b={"tcp": {"a": self.SLOW + 500, "c": just_over}}, + c={"tcp": {"a": self.SLOW, "b": just_over}}, + d={"tcp": {"a": self.SLOW, "b": 5.0, "c": 5.0}}, + ) + status = self.pass_(hosts) + self.assertEqual(self.verdicts(status), {"a": "unfit", "b": "unfit", "c": "unfit", "d": "unfit"}) + self.assertEqual({name: host["attributed_to"] for name, host in status["hosts"].items()}, {"a": ["a"], "b": ["a"], "c": ["a"], "d": ["a"]}) + self.assertEqual(status["attribution"], {"a": ["a", "b", "c", "d"]}) + self.assertEqual(status["decision"]["swap"], {"out": "a", "in": "d"}) + self.assertIn("b (voter, link wired): unfit, round trip p99 to a", self.log) + self.assertIn("(attributed to a)", self.log) + + def test_a_hosts_own_fault_is_attributed_to_it_and_a_link_fault_to_the_end_at_fault(self) -> None: + status = self.pass_(self.slow_voter_c(a={"wal": WAL_LIMIT_MS + 5})) + self.assertEqual(status["hosts"]["a"]["attributed_to"], ["a", "c"], "a's disk is its own, its slow link is c's") + self.assertEqual(status["hosts"]["b"]["attributed_to"], ["c"]) + self.assertEqual(status["attribution"], {"a": ["a"], "c": ["a", "b", "c", "d"]}) + + def test_a_standby_slow_to_every_voter_is_at_fault_itself(self) -> None: + status = self.pass_(healthy(d={"tcp": FIT_RTT_MS + 20})) + self.assertEqual((status["hosts"]["d"]["verdict"], status["hosts"]["d"]["attributed_to"]), ("unfit", ["d"])) + self.assertEqual(status["attribution"], {"d": ["d"]}) + + def test_a_standby_slow_to_a_voter_it_alone_reaches_slowly_shares_the_fault_with_it(self) -> None: + # d is fast to b and c, and a is fast to b and c: nothing says which end of the one slow link is at fault. + status = self.pass_(healthy(d={"tcp": {"a": FIT_RTT_MS + 20, "b": 5.0, "c": 5.0}})) + self.assertEqual((status["hosts"]["d"]["verdict"], status["hosts"]["d"]["attributed_to"]), ("unfit", ["a", "d"])) + self.assertEqual(status["hosts"]["a"]["attributed_to"], [], "a's own verdict is fit") + + def test_a_slow_link_between_voters_whose_other_links_are_fast_is_attributed_to_both(self) -> None: + # Only the link between a and b is slow: a is fast to c, and so is b, so nothing says which end is at fault. + status = self.pass_(healthy(a={"tcp": {"b": UNFIT_RTT_MS + 50, "c": 5.0}})) + self.assertEqual(status["hosts"]["a"]["attributed_to"], ["a", "b"]) + self.assertEqual(status["hosts"]["b"]["attributed_to"], [], "b's own verdict is fit") + + def test_a_voter_slow_to_every_other_voter_is_at_fault_though_a_standby_reaches_it_fast(self) -> None: + # a measures both its voter links slow, and b and c reach each other fast; d's one-sided reading of a does not clear it. + status = self.pass_(healthy(a={"tcp": UNFIT_RTT_MS + 50})) + self.assertEqual(status["hosts"]["a"]["attributed_to"], ["a"]) + self.assertEqual(status["attribution"], {"a": ["a"]}) + + def test_a_verdict_that_is_not_unfit_is_attributed_to_no_one(self) -> None: + status = self.pass_(healthy(d={"samples": WINDOW_SAMPLES - 1})) + self.assertEqual({name: host["attributed_to"] for name, host in status["hosts"].items()}, {"a": [], "b": [], "c": [], "d": []}) + self.assertEqual(status["attribution"], {}) + def test_an_unfit_standby_is_not_a_candidate(self) -> None: status = self.pass_(healthy(a={"tcp": UNFIT_RTT_MS + 50}, d={"fsync_ms": 50.0})) self.assertEqual(status["decision"]["action"], "none") From aab6825116617d3707ad202c68e3da14e7e68fee Mon Sep 17 00:00:00 2001 From: Joseph Mearman Date: Tue, 3 Nov 2026 18:37:10 +0000 Subject: [PATCH 2/2] test(role): check the slow voter's peers' verdicts are attributed to it The role_controller scenario makes one voter slow with tc netem; every unfit verdict it causes must now name that voter in attributed_to, and the attribution map must list it as accounting for its own. --- tests/mesh/run.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/mesh/run.sh b/tests/mesh/run.sh index 94fcafff..2e3ed9a6 100755 --- a/tests/mesh/run.sh +++ b/tests/mesh/run.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Mesh integration test for the github_runner_cluster role: stands up three k3s nodes in Docker on one machine, each in its own copy of the Compose project, and has the role join them through Headscale. Scenarios: hosted headscale_hosted: the role runs Headscale under Compose beside the first node. in_cluster headscale_in_cluster: the role runs Headscale inside the cluster, on the bootstrap server. existing headscale_existing: the policy check fails, with the entry to add, against a server whose policy lacks autoApprovers, and passes once it has one. Every scenario uses automatic server selection (three hosts, no overrides, so three servers). native_server three Docker servers and a native host (a privileged systemd container, tests/mesh/native/Dockerfile, managed over Ansible's docker connection) that joins as an agent, is then made a server by the inventory and promoted in place: it must join etcd as a learner, be promoted to a voter with its mesh address as its peer URL, and keep the cluster writable throughout, and a rerun must not restart it. role_controller four hosts (three servers and a standby agent, with etcd's data on memory so the machine's disk does not decide): the role controller's probe and controller run in the cluster from the image built from this checkout, with the manifests the ARC role renders; every host is reported fit from a full window of real samples, with the link the cluster role labelled its node with, a voter made slow with tc netem on its mesh interface is reported unfit with the swap named, a paused voter makes the controller report that it would not act, and the controller changes nothing (not in the default set; needs jq, and takes a window of 100 samples before it can judge). compose_faulty the role refuses a Docker Compose release that recreates the containers it has just built, before writing anything (run it under such a release; it is not in the default set). host_network each node runs in a Docker daemon of its own (a docker:dind container standing in for a VM with an address on the LAN, on the shared test network) with github_runner_cluster_network_mode host, joined through a Headscale server the test runs (headscale_existing); the nodes' Tailscale clients must then reach each other directly at those daemons' addresses, and the role must not publish any port; then the role must refuse to move two running nodes to the bridge network in one run, and move one there and back through its guard, the node keeping its mesh address and etcd membership. Linux only, since the control node reaches each daemon at its address on the test network. The cluster scenarios assert that etcd has three members, that every node is Ready, and that pods on different nodes reach each other over the mesh. +# Mesh integration test for the github_runner_cluster role: stands up three k3s nodes in Docker on one machine, each in its own copy of the Compose project, and has the role join them through Headscale. Scenarios: hosted headscale_hosted: the role runs Headscale under Compose beside the first node. in_cluster headscale_in_cluster: the role runs Headscale inside the cluster, on the bootstrap server. existing headscale_existing: the policy check fails, with the entry to add, against a server whose policy lacks autoApprovers, and passes once it has one. Every scenario uses automatic server selection (three hosts, no overrides, so three servers). native_server three Docker servers and a native host (a privileged systemd container, tests/mesh/native/Dockerfile, managed over Ansible's docker connection) that joins as an agent, is then made a server by the inventory and promoted in place: it must join etcd as a learner, be promoted to a voter with its mesh address as its peer URL, and keep the cluster writable throughout, and a rerun must not restart it. role_controller four hosts (three servers and a standby agent, with etcd's data on memory so the machine's disk does not decide): the role controller's probe and controller run in the cluster from the image built from this checkout, with the manifests the ARC role renders; every host is reported fit from a full window of real samples, with the link the cluster role labelled its node with, a voter made slow with tc netem on its mesh interface is reported unfit with the swap named and every unfit verdict attributed to it, a paused voter makes the controller report that it would not act, and the controller changes nothing (not in the default set; needs jq, and takes a window of 100 samples before it can judge). compose_faulty the role refuses a Docker Compose release that recreates the containers it has just built, before writing anything (run it under such a release; it is not in the default set). host_network each node runs in a Docker daemon of its own (a docker:dind container standing in for a VM with an address on the LAN, on the shared test network) with github_runner_cluster_network_mode host, joined through a Headscale server the test runs (headscale_existing); the nodes' Tailscale clients must then reach each other directly at those daemons' addresses, and the role must not publish any port; then the role must refuse to move two running nodes to the bridge network in one run, and move one there and back through its guard, the node keeping its mesh address and etcd membership. Linux only, since the control node reaches each daemon at its address on the test network. The cluster scenarios assert that etcd has three members, that every node is Ready, and that pods on different nodes reach each other over the mesh. # # With GRTEST_PIN_CONTEXT=1, the cluster scenarios run the role, and the in_cluster recovery playbook, with github_runner_cluster_docker_context pinned while the current Docker context they see points at nothing (tests/lib/docker_context.sh), so any docker call that ignored the pin fails the run; afterwards the current context must be unchanged. context_checks the role refuses a pinned context that does not exist, and a pinned context alongside DOCKER_HOST, before writing anything (not in the default set). # @@ -1256,6 +1256,8 @@ MANIFESTS role_status | jq . role_status | jq -r '.decision.because[]' role_status | jq -e ".hosts[\"$(node_name "$slow")\"].reasons | any(test(\"round trip p99\"))" >/dev/null || fail "the reason does not name the round trip" + role_status | jq -e --arg slow "$(node_name "$slow")" '([.hosts[] | select(.verdict == "unfit") | .attributed_to | index($slow) != null] | all) and (.attribution[$slow] | index($slow) != null)' >/dev/null \ + || fail "the unfit verdicts are not attributed to $(node_name "$slow"): $(role_status | jq -c '.attribution')" kubectl_on 1 --namespace "$platform_namespace" logs deployment/fleet-role --tail 20 | grep -F "DRY RUN: would swap out $(node_name "$slow") for $(node_name 4)" >/dev/null || fail "the controller did not log the swap it would make" log "A voter is paused: the controller reports that it would not act"