Repository navigation
feat(role): attribute each unfit verdict to the node at fault for it - #206
Merged
Merged
Conversation
Mearman
force-pushed
the
feat/role-verdict-attribution
branch
from
November 3, 2026 18:38
823887d to
b220a67
Compare
A slow link makes both of its ends unfit, so on a fleet where one voter is on a poor path every voter reads unfit and the status said nothing about which. Each host now carries attributed_to, and the status an attribution map of the hosts each node accounts for: a host's own disk, memory, readiness and restarts are its own fault, and its round trip is the fault of the end of the slow link whose links to the other voters are slower too, or of both ends when nothing tells them apart. The log names the attribution beside each unfit verdict. The voter to replace was the unfit voter on the most slow links, which ties when every link is over the limit and then fell to the worst-ranked voter, the one that measured the slowest tail, not the one at fault. It is now the unfit voter the most unfit verdicts are attributed to.
The role_controller scenario makes one voter slow with tc netem; every unfit verdict it causes must now name that voter in attributed_to, and the attribution map must list it as accounting for its own.
Mearman
force-pushed
the
feat/role-verdict-attribution
branch
from
November 3, 2026 18:40
b220a67 to
aab6825
Compare
Mearman
marked this pull request as ready for review
November 3, 2026 19:11
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.30.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On the live fleet every voter reads unfit and the controller names the wrong one. The status read this turn has m1, m3 and mini all over the round trip limit, with every standby's worst peer m1; the pair tails show why. m1's links to m3 and mini are 2170 to 2800 ms, while m3 to mini is about 390 ms, also over the 300 ms limit. Every link is then slow, so counting each voter's slow links ties all three, and the tie went to the worst-ranked voter, m3, the one that measured the largest single tail (to m1). The status said "voter m3 is unfit" and nothing about m1.
Each unfit verdict is now attributed to the node at fault for it (
attributed_toon each host, and a top-levelattributionlisting the hosts each node accounts for). A host's own disk, memory, readiness and restarts are its own fault. Its round trip is the fault of the end of the slow link whose links to the other voters are slower too, compared by the best of them: a node on a poor path is slow to every peer, a node on a good one is fast to its others. When nothing tells the two ends apart, or one end has no other link, the link is attributed to both. Only links to voters are compared, since a voter's link to a standby is the standby's one-sided reading. The voter to replace is now the unfit voter the most unfit verdicts are attributed to, the worst-ranked among those with as many, and the log adds "(attributed to …)" to each unfit verdict.Run against the live fleet's nodes and probe entries read this turn, the controller now attributes every round trip fault to m1 (
"m1": ["asus", "i5", "m1", "m3", "mini"]), keeps m3's and the standbys' own disk and restart faults as their own, and reports "voter m1 is unfit and no eligible standby is fit and ranks above it" in place of m3.Tests: the live shape (every link slow, the worst single tail measured by a voter that is not at fault) with every verdict attributed to the slow voter and that voter swapped out; a host's own fault beside a link fault; a standby slow to every voter against one slow to one voter; a link with nothing to tell its ends apart; and no attribution for a verdict that is not unfit. Each fails with
at_faultmade to return both ends. Therole_controllermesh scenario now checks that every unfit verdict names the voter slowed with netem.Refs #183
Refs #184