Repository navigation
Run Behat in throwaway containers when Docker is available #391
Description
When Docker is available, a run could happen in throwaway containers, one per worker from #390. The container is not faster by itself: PHP ran at the same speed inside and outside of it in local tests. What it allows without root on the host is what makes the difference:
- the temporary directories and the database in memory, with no disk syncs;
- copy-on-write snapshots of the cached WordPress install instead of a full copy for every scenario (about 6 ms instead of about 90 ms to set up and tear down);
- a separate network per worker, so scenarios using fixed ports can run in parallel;
- a known toolchain, so every declared test requirement is present.
Starting a container takes about 0.3 to 0.5 seconds and every docker exec about 50 to 75 ms. Running the whole Behat process of a worker inside one container avoids that cost per command. The container's PHP version has to match the one the vendor folder was installed for.
Steps:
- Provide an image tuned for the test runs, without Xdebug unless coverage is requested.
- Run each worker of Run scenarios in parallel #390 in its own throwaway container, falling back to the regular runner when Docker is not available.
- Restore the cached WordPress install as a snapshot instead of a copy.
To reproduce a failure, it is enough to rerun that single scenario in a container that is kept.
Measuring: take the baseline from #384, then open a draft PR, run the same CI jobs against it and record both results here before we decide.
Activity
schlessera commentedon Oct 30, 2026
Measurement
Measured like #390, on the same baselines, with #408 and
WP_CLI_TEST_PARALLEL=4plusWP_CLI_TEST_DOCKER=1.core-commandtotalcore-commandper jobcache-commandtotalcache-commandper jobFor
cache-command, every job passed and the containers were another 9 percentage points faster than parallel workers on the runner itself, including the time to build the image and start the database servers.For
core-command, 17 jobs failed, all of them against MySQL.In the containers, PHP prints deprecation notices for older WordPress versions on PHP 8 that it does not print on the runner, with the sameerror_reportinganddisplay_errorssettings, and the containers use the MariaDB client against a MySQL server where the runner uses MySQL's own client.Correction: the database client plays no part, and the PHP settings were not the same. The runners set
error_reporting=-1, while the containers had nophp.iniand therefore reportedE_ALL. While testing, WP-CLI keepsE_ALLbut reduces any other value to everything except deprecations (Runner::enable_error_reporting()). So in the containers, scenarios that downgrade to an older WordPress version on PHP 8 failed on deprecation notices that the runners never print. These scenarios are tagged@require-mysql, which is why only the MySQL jobs failed (the MySQL 5.6 and 5.7 jobs run PHP 7 and fell back to running without Docker). One more failure was thecore-download.feature:524issue fixed in wp-cli/core-command#355.#408 now gives the containers the host's PHP error settings. Locally, with the runner's settings, the
core-commandsuite against MySQL 8.0 went from the same 9 failures to none. A new measurement follows below.Locally, where these differences can be kept in mind, containers remain useful for isolated parallel runs without a local database server.
schlessera commentedon Oct 31, 2026
Measurement with the host's PHP settings
With #408 now giving the containers the host's PHP error settings, and
core-commandon top of wp-cli/core-command#355:core-commandtotalcache-commandtotalEvery job passes now, against all MySQL and MariaDB versions and SQLite, and all runs executed the same scenarios (4,567 and 1,162 across the 49 Linux jobs). The containers are another 4 to 6 percentage points faster than parallel workers on the runner itself, including the time to build the image and start the database servers.
The PHP 7.2 and 7.3 jobs still run without Docker, as their base images can no longer be built, and Windows and macOS run without Docker as well.