feat(role): attribute each unfit verdict to the node at fault for it

README.md

@@ -208,7 +208,7 @@ The controller reads the Nodes and the probes' samples every sample period and p
 - A window is 100 samples, the fewest that give the 99th percentile a sample of its own, so one outlier never decides; a host with fewer is `unknown`, as is one whose probe has not reported for three sample periods. Readings from before the last change of the voter set are not counted.
 - Hosts that pass are ranked wired before Wi-Fi before unknown, then by round trip, then by disk. The link ranks hosts and is never a gate; the status gives each host's link and where it was read (`link` and `link_source`), and the log names it beside the verdict.
 
-A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter on the most slow links (a link is slow when either end measures it above the limit), the worst-ranked among those on as many. A swap is reported only when that voter is eligible and not pinned (when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped) and a fit eligible standby ranks above it. A candidate is judged against the voters that would remain, since a slow link to the very voter it would replace says nothing against it. It is held back, with the reason in the status and the log, while the voter count is not `github_runner_arc_role_controller_voters`, any voter is below `full`, is not Ready, has no leader, is a learner or knows a member that is not a voter, is not connected to every other voter, fewer hosts are eligible than the target, or the cooldown after a change of the voter set has not passed (k3s's five minutes to remove a stalled learner plus one window). The target must be odd, at least 3, and at most `github_runner_cluster_max_servers` minus one, which leaves a swap one place of headroom; the role refuses anything else. A host is eligible, and a voter pinned, by a Node label with the value `true` (`github_runner_arc_role_controller_eligible_label` and `_pinned_label`), which the cluster role publishes from `github_runner_cluster_voter_eligible` and `github_runner_cluster_voter_pinned` per host (and the target from `github_runner_cluster_voters`, refused unless odd, at least 3 and below `github_runner_cluster_max_servers`; see the cluster role's README); a voter not marked eligible is never swapped out, since nothing has prepared it to come back.
+A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so every unfit verdict is attributed to the node at fault for it (`attributed_to` on each host, and `attribution`, the hosts each node accounts for). A host's own disk, memory, readiness or restarts are its own fault. Its round trip is the link to its worst peer, and the end at fault for a slow link is the one whose links to the other voters are slower too, compared by the best of them, since a node on a poor path is slow to every peer and a node on a good one is fast to its others; when the two ends' other links are as fast as each other's, or one end has none, the link is attributed to both. The voter to replace is the unfit voter the most hosts' unfit verdicts are attributed to, the worst-ranked among those with as many, and the log names the attribution beside each unfit verdict. A swap is reported only when that voter is eligible and not pinned (when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped) and a fit eligible standby ranks above it. A candidate is judged against the voters that would remain, since a slow link to the very voter it would replace says nothing against it. It is held back, with the reason in the status and the log, while the voter count is not `github_runner_arc_role_controller_voters`, any voter is below `full`, is not Ready, has no leader, is a learner or knows a member that is not a voter, is not connected to every other voter, fewer hosts are eligible than the target, or the cooldown after a change of the voter set has not passed (k3s's five minutes to remove a stalled learner plus one window). The target must be odd, at least 3, and at most `github_runner_cluster_max_servers` minus one, which leaves a swap one place of headroom; the role refuses anything else. A host is eligible, and a voter pinned, by a Node label with the value `true` (`github_runner_arc_role_controller_eligible_label` and `_pinned_label`), which the cluster role publishes from `github_runner_cluster_voter_eligible` and `github_runner_cluster_voter_pinned` per host (and the target from `github_runner_cluster_voters`, refused unless odd, at least 3 and below `github_runner_cluster_max_servers`; see the cluster role's README); a voter not marked eligible is never swapped out, since nothing has prepared it to come back.
 
 The controller and the probe run one image, `github_runner_arc_role_controller_image`, pulled without a pull Secret, so the published package must be public.
 

scripts/fleet-role.jq

@@ -123,13 +123,18 @@ $config as $c
         etcd: ($entry.samples[-1].etcd // null)
       };
 
-  # What makes a measured host unfit, as sentences carrying their numbers; empty when nothing does. A voter is judged by the unfit round trip limit and the disk it serves etcd from, a candidate by the stricter fit limit and a disk probe.
-  def faults($m):
+  # What makes a measured host unfit, as sentences carrying their numbers; empty when nothing does. A voter is judged by the unfit round trip limit and the disk it serves etcd from, a candidate by the stricter fit limit and a disk probe. The round trip is a link's, and another node may be at fault for it; every other fault is the host's own.
+  def rtt_faults($m):
     $m.node as $node
     | [
         (if $m.rtt_ms != null and $m.rtt_ms > (if $node.voter then $unfit_rtt_ms else $fit_rtt_ms end)
          then "round trip p99 to \($m.rtt_peer) is \($m.rtt_ms | round2) ms, over \(if $node.voter then $unfit_rtt_ms else $fit_rtt_ms end) ms (a \(if $node.voter then "tenth" else "twentieth" end) of the \($c.election_timeout_ms) ms election timeout)"
-         else empty end),
+         else empty end)
+      ];
+
+  def own_faults($m):
+    $m.node as $node
+    | [
         (if $node.voter then
            (if $m.wal_p99_ms != null and $m.wal_p99_ms >= wal_fsync_limit_ms then "WAL fsync p99 is \($m.wal_p99_ms | round2) ms, not under \(wal_fsync_limit_ms) ms" else empty end),
            (if $m.commit_p99_ms != null and $m.commit_p99_ms >= backend_commit_limit_ms then "backend commit p99 is \($m.commit_p99_ms | round2) ms, not under \(backend_commit_limit_ms) ms" else empty end)
@@ -142,6 +147,8 @@ $config as $c
         (if $m.restarts > 0 then "k3s restarted \($m.restarts) time(s) in the window" else empty end)
       ];
 
+  def faults($m): rtt_faults($m) + own_faults($m);
+
   # What is missing for a verdict, as sentences; empty when the readings are all there.
   def gaps($m):
     $m.node as $node
@@ -187,7 +194,31 @@ $config as $c
        m: $m};
 
   # Hosts reported: every voter, and every eligible host that is not one.
-  ([$all[] | select(.voter or .eligible)] | sort_by(.name) | map(host(.))) as $hosts
+  ([$all[] | select(.voter or .eligible)] | sort_by(.name) | map(host(.))) as $measured
+
+# Every link a probe measured, between two hosts, as the worse of the two ends' tails (a standby measures its links to the voters alone, so those have one end's).
+| ([$measured[] | .name as $a | .m.pairs[] | select(.ms != null) | {ends: ([$a, .peer] | sort), ms}]
+   | group_by(.ends) | map({ends: .[0].ends, ms: (map(.ms) | max)})) as $links
+
+# The best of a host's links to the voters other than $excluded, in milliseconds: how fast its path is where the link in question plays no part. Links to the voters alone, since every host measures those (a voter's to a standby is the standby's one-sided reading). Null when it has no other.
+| def elsewhere($host; $excluded):
+    [$links[] | select((.ends | index($host)) != null and (.ends | index($excluded)) == null)
+     | select(.ends | map(select(. != $host)) | .[0] as $other | $voter_names | index($other) != null) | .ms] | min;
+
+  # The end at fault for a slow link between $host and $peer: the one whose other links are slower too, since a node on a poor path is slow to every peer and a node on a good one is fast to its others. Both ends when their other links are as fast as each other's, or one has none to compare.
+  def at_fault($host; $peer):
+    elsewhere($host; $peer) as $own
+    | elsewhere($peer; $host) as $theirs
+    | if $own == null or $theirs == null or $own == $theirs then [$host, $peer] | sort
+      elif $theirs > $own then [$peer]
+      else [$host] end;
+
+  # The nodes an unfit verdict is attributed to: for its round trip, the end at fault for the slow link to its worst peer; for any other fault (its own disk, memory, readiness or restarts), the host itself. Empty for a verdict that is not unfit.
+  def attributed($h):
+    if $h.verdict != "unfit" then []
+    else (if (rtt_faults($h.m) | length) > 0 then at_fault($h.name; $h.m.rtt_peer) else [] end) + (if (own_faults($h.m) | length) > 0 then [$h.name] else [] end) | unique end;
+
+  ($measured | map(. + {attributed_to: attributed(.)})) as $hosts
 | ([$hosts[] | select(.eligible)] | length) as $eligible_count
 | ($all | map(select(.voter))) as $voter_nodes
 
@@ -207,9 +238,11 @@ $config as $c
     (if $cooldown_until != null and $now < $cooldown_until then "the membership changed at \($changed_at), and the cooldown lasts until \($cooldown_until)" else empty end)
   ]) as $blocks
 
-# The swap. A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter on the most slow links (a link is slow when either end measures it above the unfit limit), the worst-ranked among those on as many. It goes out only if it is eligible and not pinned; when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped. The best fit eligible standby that ranks above it comes in, judged against the voters that would remain.
-| ([$hosts[] | select(.role == "voter") | .name as $a | .m.pairs[] | select(.ms != null and .ms > $unfit_rtt_ms) | [$a, .peer] | sort] | unique) as $slow_links
-| ([$hosts[] | select(.role == "voter" and .verdict == "unfit") | . + {slow_links: (.name as $n | [$slow_links[] | select(index($n) != null)] | length)}] | sort_by(.rank) | reverse | sort_by(-.slow_links) | first) as $culprit
+# Which hosts' unfit verdicts each node accounts for, by node.
+| ([$hosts[] | .name as $h | .attributed_to[] | {node: ., host: $h}] | group_by(.node) | map({key: .[0].node, value: map(.host)}) | from_entries) as $attribution
+
+# The swap. A slow link makes both of its ends unfit, since a voter is unfit when its round trip to any other voter is, so the voter to replace is the one at fault: the unfit voter the most hosts' unfit verdicts are attributed to, the worst-ranked among those with as many. It goes out only if it is eligible and not pinned; when it is not, replacing a voter that shares a slow link with it would fix nothing, so nothing is swapped. The best fit eligible standby that ranks above it comes in, judged against the voters that would remain.
+| ([$hosts[] | select(.role == "voter" and .verdict == "unfit") | . + {accounts_for: ($attribution[.name] // [] | length)}] | sort_by(.rank) | reverse | sort_by(-.accounts_for) | first) as $culprit
 | (if $culprit != null and $culprit.eligible and ($culprit.pinned | not) then $culprit else null end) as $out
 | ([$hosts[] | select(.role == "standby" and .eligible)
     | (if $out == null then .m else (.m | without($out.name)) end) as $m
@@ -238,6 +271,7 @@ $config as $c
              quota_bytes: $quota_bytes},
     membership: {voters: $voter_names, changed_at: $changed_at, cooldown_until: $cooldown_until},
     hosts: ($hosts | map({key: .name, value: del(.name, .rank, .m)}) | from_entries),
+    attribution: $attribution,
     ranking: ($hosts | map(select(.verdict == "fit")) | sort_by(.rank) | map(.name)),
     blocks: $blocks,
     decision: $decision

scripts/fleet-role.sh

@@ -80,7 +80,7 @@ kubectl patch configmap "$status_configmap" -n "$namespace" --type merge --patch
 # What each host's verdict rests on, then the decision, in the log.
 while IFS= read -r line; do log "$line"; done < <(jq -r '
   .hosts | to_entries[]
-  | "\(.key) (\(.value.role)\(if .value.pinned then ", pinned" else "" end)\(if .value.eligible then "" else ", not eligible" end), link \(.value.link)): \(.value.verdict), \(.value.reasons | if length == 0 then "all readings within their limits" else join("; ") end)"' <<< "$result")
+  | "\(.key) (\(.value.role)\(if .value.pinned then ", pinned" else "" end)\(if .value.eligible then "" else ", not eligible" end), link \(.value.link)): \(.value.verdict), \(.value.reasons | if length == 0 then "all readings within their limits" else join("; ") end)\(if (.value.attributed_to | length) > 0 then " (attributed to \(.value.attributed_to | join(" and ")))" else "" end)"' <<< "$result")
 while IFS= read -r line; do log "$line"; done < <(jq -r '
   if .decision.action == "swap" then "DRY RUN: would swap out \(.decision.swap.out) for \(.decision.swap.in): \(.decision.because | join("; "))"
   else "DRY RUN: would not act: \(.decision.because | join("; "))\(if .decision.wanted == null and (.blocks | length) > 0 then " (also: \(.blocks | join("; ")))" else "" end)" end' <<< "$result")