Repository navigation
fix(ci): deal unit-test shards by the work each module does - #207
Merged
Merged
Conversation
The shards were dealt by a module's size in bytes, which says little about its time: a module's playbook runs decide it, so one shard held several times the work of another and the heaviest could reach its timeout. scripts/ci/shard_unit_tests.py counts each module's work from its tests (the processes they start through helpers, base classes and set-up methods, with a playbook's tasks added for a playbook run) and deals the modules heaviest first to the shard with the least work so far. The job takes its modules from it.
Mearman
marked this pull request as ready for review
November 3, 2026 19:12
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins. |
|
🎉 This PR is included in version 2.30.1 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The unit-test job dealt its modules to the five shards by file size, sorted largest first and taken modulo the shard count. Size says little about a module's time, which is set by the
ansible-playbookprocesses its tests start, so the shards ended up very unequal: on recent runs one shard did many times the work of another, and the heaviest hit its timeout on one of them.scripts/ci/shard_unit_tests.py SHARD SHARDSnow prints the modules for a shard, and the job reads them from it. It imports the test modules and counts each one's work from its tests, in units of one Ansible task:subprocess.runand its siblings) counts, through the helpers, base-class methods,super()calls,setUp,tearDownand (once per class)setUpClassit reaches; a launch site counts once however often a loop runs it, a conditional counts its dearer branch, a skipped test counts nothing;TASKS_PER_PROCESS_STARTtasks, a ratio I fitted against per-module durations in the logs of two recent successful runs (work against mean duration correlates at about 0.9 over 57 modules, file size at about 0.5). The dealing only needs a playbook start to be non-negligible beside a task, so the constant is not tuned further.The modules are then dealt heaviest first to the shard with the least work so far (ties to the lower index, equal modules in name order). A failed dealing now fails the step instead of leaving a shard with nothing to run.
Work per shard on origin/main's 58 modules, before and after:
Heaviest over lightest goes from 6.6 to 1.00 (heaviest over mean from 1.83 to 1.00). Playbook launch sites reached per shard ranged from 54 to 312 before and range from 104 to 278 after: a launch's cost differs with the playbook it runs, which is what the task counts account for. Replaying the dealing on the per-module durations from the same two runs, the heaviest shard was 13.5 times the lightest before and is 2.1 times after (about 1.7 and 1.3 times the mean); those durations are noisy and the after figure is a replay, not a CI result.
The timeout stays at 30 minutes: the heaviest shard's work is now 0.55 of the old heaviest's, so the guard sits further above it than before.
tests/unit/test_shard_unit_tests.pycovers the counting on synthetic modules (each launcher, helpers, base classes,super(), set-up methods, skips, recursion, loops, conditionals, playbook task counts for the three include forms, a module that cannot be imported) and the dealing on a skewed table and on this tree: every module in exactly one shard for 1 to 8 shards, the same assignment whatever the listing order, the heaviest shard within Graham's bound of the mean, and every module dealt to the lightest shard so far. Dealing by position modulo the shard count, or in name order, fails the last of these (and the first also fails the heavy-module case); a work measure that stops counting launches fails the counting tests.