Repository navigation
Run scenarios in parallel #390
Description
Scenarios run strictly one after another, even though almost all of them are independent and the CI runners have several cores. In a local experiment with 213 entity-command scenarios, a serial run took 491 seconds. With 8 workers it took 164 seconds when split by feature file, where one large file set the floor. Split by scenario, it took 140 seconds (3.5×).
Steps:
- Give each worker its own state: temporary directory,
HOME, WP-CLI cache and test database.composer prepare-testsneeds to grant access to the additional databases. - Keep scenarios that rely on fixed shared resources, such as port 8080 or the scaffold test database, from running at the same time.
- Split the work by scenario rather than by file, balanced by the per-scenario timings from Measure Behat run times reliably #384.
- Merge the results into a single report and keep
behat-rerunworking.
Measuring: take the baseline from #384, then open a draft PR, run the same CI jobs against it with the parallel runner enabled and record both results here before we decide. CI runners have fewer cores than a development machine, so the gain there will be smaller than locally.
Activity
schlessera commentedon Oct 30, 2026
Measurement
Measured on GitHub Actions with the full test matrix of
core-commandandcache-command, run onmeasure/*branches of wp-cli/core-command#350 and wp-cli/cache-command#137. All runs use wp-cli-tests at the same base commit (9298472). The parallel run uses #405 withWP_CLI_TEST_PARALLEL=4, matching the 4 cores of the Linux runners.The metric is the Behat wall time of each job, from the start of the Behat step to its last summary, including any rerun of failed scenarios. "Total" is the sum over the Linux jobs present in all runs, "per job" the median of each job's change against the mean of the two baseline runs, which run identical code and show the noise level.
core-commandtotalcore-commandper jobcache-commandtotalcache-commandper jobThe comparison covers 48 Linux jobs for
cache-commandand 29 forcore-command, the ones whose logs could be read for all runs. Both ran the same scenarios as the baselines (1,138 and about 2,550 across these jobs).The run time of the Behat step roughly halves, and more than halves for
core-command, whose scenarios are longer and balance better over the workers.Two things came up along the way:
features/core-download.feature:524incore-commandfails whenever it runs with an empty WP-CLI cache, also on its own in a serial run without any of these changes. A serial run of the whole suite only passes because the scenario before it puts the same download in the cache. In a parallel run, it fails in whichever worker it does not follow that scenario, as in the PHP 7.2 job above. That needs a fix incore-command(orwp core download --no-extract) before parallel runs can be enabled there.composer behat-rerunafter a parallel run reran every scenario serially whenWP_CLI_TEST_PARALLELwas not set for the rerun as well. Reruns now always cover exactly the failed scenarios. The other failed job is the Windows one, where Chocolatey timed out installing Ghostscript before any scenario ran.schlessera commentedon Oct 31, 2026
Measurement after wp-cli/core-command#355
The
core-download.feature:524failure is fixed in wp-cli/core-command#355, which is merged. Thecore-commandmeasurements were repeated on top of it, with fresh baselines and the same wp-cli-tests commit:core-commandtotalcore-commandper jobEvery job of the matrix passes now, Windows and macOS included, and the parallel run executed the same 2,935 scenarios across the 33 Linux jobs compared here. The run time of the Behat step drops by about 58%, in line with the earlier measurement.
For #391, the run in containers on the same base (3,131 s, -64.2%) still fails the 16 MySQL jobs, for the reasons described there.
schlessera commentedon Oct 31, 2026
Parallel runs across all bundled commands
To find scenarios that only fail when run in parallel, the Behat tests of all 28 bundled commands and of
wp-cli/wp-cliran withWP_CLI_TEST_PARALLEL=4on their full CI matrix, through draft PRs that are not meant to be merged: wp-cli/ability-command#17, wp-cli/ai-command#32, wp-cli/block-command#22, wp-cli/cache-command#139, wp-cli/checksum-command#163, wp-cli/config-command#237, wp-cli/core-command#358, wp-cli/cron-command#154, wp-cli/db-command#356, wp-cli/embed-command#102, wp-cli/entity-command#684, wp-cli/eval-command#104, wp-cli/export-command#150, wp-cli/extension-command#567, wp-cli/i18n-command#516, wp-cli/import-command#116, wp-cli/language-command#193, wp-cli/maintenance-mode-command#46, wp-cli/media-command#271, wp-cli/rewrite-command#85, wp-cli/role-command#86, wp-cli/scaffold-command#404, wp-cli/search-replace-command#252, wp-cli/server-command#108, wp-cli/shell-command#96, wp-cli/site-health-command#23, wp-cli/super-admin-command#79, wp-cli/widget-command#92, wp-cli/wp-cli#6446.Besides the jobs that failed, the first parallel pass of sampled green jobs was checked as well, as
composer behat-reruncan hide a failure in it.Round 1, with #405
wp-cli/wp-cli, awp-cli.ymlwithuser: adminfromruncommand.featurebroke--context=adminand--debugscenarios. Serial runs only avoided this by the order of the scenarios. Fixed in Leave files of the scenario out of the installation cache #411.core-download.feature:524incore-command, fixed earlier in Name the archive of core download --no-extract after the release core-command#355.Round 2, with #405, #411 and the fixes above
All 29 packages pass. Two failures were external:
embed-commandfailed most jobs while a third-party oEmbed provider was unreachable and passed when re-run, as didmainat the same time, andscaffold-commandhit underscores.me failures in two jobs in round 1 that passed when re-run.The first passes also showed workers removing each other's theme downloads: the processes launched by the tests did not receive the worker's
TMPDIR, and WordPress saves downloads in the temporary directory under fixed names such astwentytwelve.4.9.zip. The reruns hid this inscaffold-commandandwidget-command. #405 now passesTMPDIRon.schlessera commentedon Oct 31, 2026
Third round across all 29 bundled packages, now including the temporary directory fix in #405 (10e2ae7) and the follow-ups to #411 (symbolic links, trailing slashes): every package passes its full matrix in parallel mode, with no reruns needed.
The fixes for the hard-coded test database names (wp-cli/config-command#238, wp-cli/db-command#357, wp-cli/wp-cli#6448), #411 and wp-cli/wp-cli#6447 are merged. The draft PRs used for these runs are closed.