You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
test(easyengine): add a Behat suite and CI workflow - #1950
Replaces core's old features/ with a Behat suite of 250 scenarios (outline rows counted) and adds a Behat workflow that runs it on every pull request, on pushes to develop and on demand.
It covers ee cli (version, info, check-update and update, dumps, completions, aliases and alias groups, has-command), help for every command, unknown commands, global flags, config files, the requirement checks, migration triggering and the image migration rollback, the phar build and the self-uninstall. features/COVERAGE.md maps each command, flag and behaviour to its scenarios.
behat.yml adds profiles: the default one skips @destructive scenarios, and @destructive, @migration and @uninstall are also refused at runtime unless EE_BEHAT_DESTRUCTIVE, EE_BEHAT_ALLOW_MIGRATION or EE_BEHAT_ALLOW_UNINSTALL is set, since a CLI --tags replaces the profile filters. On a machine without ee.sqlite the suite also needs EE_BEHAT_ALLOW_INIT=1.
Workflow lanes on PHP 7.4 and 8.5: main (everything but the migration and the uninstall, @destructive included since the runner is thrown away), migration (the real upgrade pipeline) and uninstall, each in its own job.
test_and_build.yml: the test job sets EE_BEHAT_ALLOW_INIT=1 (the runner starts without ee.sqlite) and GITHUB_TOKEN (check-update scenarios call the GitHub API), and no longer runs the site-command and site-type-wp suites. Both packages run their own Behat job on every PR, and their new suites, with lanes for known bugs and a local Pebble ACME server, would fail when run here unfiltered. site-type-php's suite still runs here.
Scenarios written for open bugs are left out for now: they assert the fixed behaviour, so each comes back with its fix. Every scenario in this PR gates, and COVERAGE.md lists the bugs that aren't covered yet.
To run it locally (as root, on a test machine): sudo -E vendor/bin/behat runs the default profile, which leaves global state alone; -p destructive with EE_BEHAT_DESTRUCTIVE=1 runs the rest except the migration and the uninstall.
Base the core context on site-command's, fix its unanchored step regexes and inverted container count check, and add generic steps for exit codes, output, json paths, files, EasyEngine options and a per-scenario scratch dir. Commands run through EE\Process so the harness no longer writes them to ee.log, and cleanup only touches bh-easyengine-* sites.
Tag the self-uninstall feature @destructive, exclude that tag in behat.yml (profiles destructive and all run it) and cover declining the uninstall prompt.
When `(not )?` is the last capture group and doesn't match, preg_match drops it, so Behat can't fill `$negate`/`$text`: every positive "STDOUT should contain:" and "should contain ANSI color codes" step failed (all help outline examples among them). A dry run can't catch it. Capture an empty string instead.
…he live stack
- Refuse @destructive, @migration and @uninstall scenarios unless EE_BEHAT_DESTRUCTIVE, EE_BEHAT_ALLOW_MIGRATION or EE_BEHAT_ALLOW_UNINSTALL is set: a CLI --tags replaces the behat.yml filter.
- Split the migration and self-uninstall scenarios into their own profiles; the uninstall ran before later features and broke them (or re-initialised EasyEngine).
- Only the @migration scenario writes an older options.version; the others use 99.0.0 so a regression errors instead of migrating. Restore changed options on shutdown and SIGTERM too.
- Guard the self-uninstall's `cli info` and check networks, volumes and images after it.
- Build the phar from a --no-dev copy under /var/tmp: the dev vendor produced a phar that fatals on start, and the old step wrote and deleted ee.phar/ee-old.phar in the checkout.
…ges and alias groups
- Reference CORE-3..13 in the @known-bug scenarios; drop the "site nosuchcommand" example, which contradicts the passing "site create" rows.
- Add scenarios for per-command config defaults, disabled commands in help lists and whole namespaces, EE_PACKAGES_DIR, site cmd-dump without docker, alias group flags, the self-uninstall refusal, check-update HTTP errors, and known bugs CORE-9..13.
- Check exit codes before negative assertions, make the PAGER check observable, set COLUMNS for cli info tables, clear no_proxy for the dead-proxy checks, and fold check-update --major into the type outline.
… migration fails
A probe container migration and a blocked registry make the upgrade fail after the container migrations ran: the probe's down() runs, its row is removed, and the next attempt runs it again.
…y to be refused
The CORE-4 scenario becomes a regular one and checks the error; new scenarios cover a group and @ALL with an ssh member (a local alias still runs), --ssh/--http, and an ssh: config key that ee config unset can remove. They fail on develop until core fix/refuse-remote-aliases is merged.
Runs core's features/ on a throwaway runner, with every easyengine/* package on dev-develop. The main job runs the default profile plus @destructive on a fresh runner (EE_BEHAT_ALLOW_INIT=1, the first checked command initialises EasyEngine); the real upgrade migration and the self-uninstall each get their own job on an initialised install. @known-bug scenarios run in a separate non-gating job. Needs the suite's behat.yml profiles and guards from test/behat-coverage.
Run PHP 7.4 and 8.5, strip control characters from the junit files before building failures.txt, and cap the Behat step at 75 minutes so a hung run still uploads its logs.
EE's first run downloads docker-compose 2.27.0 without --fail (MIG-10); on a GitHub error it moves the HTML error page over the binary, as on an admin-tools-command runner. Fetch 2.27.0 again with --fail and retries in that case.
…re's test job
Both packages now run their own Behat workflow on every pull request, with tag-filtered lanes: known-bug scenarios never gate, and @le runs only against a local Pebble. Run unfiltered here, their new suites would fail on the known-bug scenarios and on the ones that need Pebble or the local rclone remote.
The test job runs core's Behat suite on a fresh runner with no ee.sqlite, and the suite refuses to start the first-run setup unless EE_BEHAT_ALLOW_INIT is set. GITHUB_TOKEN keeps the check-update scenarios clear of the unauthenticated API rate limit when all PHP versions run at once.
The 17 scenario rows written for open bugs assert the fixed behaviour and fail today, so they stay out of the suite until their fixes land, and COVERAGE.md lists those bugs as not covered yet. Every scenario that ships now gates, and the non-gating known-bug CI lane goes with them.
…d drop the unused site cleanup
After ee config unset ssh the scenario runs a full site list, which passes the requirement checks and could start a migration or the first-run setup; it now starts with the no-pending-migration guard. The @AfterFeature hook deleted bh-easyengine-* sites although the suite creates none, and its site delete could run a migration too, so it is gone.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Root-level migration, host-file modification, and uninstall fixtures need human review, and the rollback test still lacks a deterministic failure setup.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Ensure migration test forces an image pull failure
features/migration.feature:102
Lowering only options.version does not guarantee an image pull. Containers::start_container_migration() skips pulls when the recorded/running image tags already match the target versions. On such an installation, blocking Docker Hub does not fail the image migration, so this scenario fails against correct code rather than testing rollback. Arrange and restore a stale image fixture that is not excluded from downloads, accounting for running-container tags overriding database values, or inject a deterministic failure in the image-migration step.
… an image to pull
With Docker Hub blocked the image migration only fails if it pulls something. On a fresh runner mailhog and php have no recorded tag, but a host where every pullable image already runs at its target tag pulls nothing, so the scenario would fail against correct code. A new step records a stale tag (restored after the scenario) for one image that isn't running when nothing would be pulled.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Root-run migration, host-file modification, and uninstall paths require human safety review, and the subsite helper still needs correction.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Use parent site container and subsite slug when creating subsites
features/bootstrap/FeatureContext.php:213
The new step reverses the parent site and subsite. For I create subsite "blog" in "example.test", it targets blog_php_1 and passes --slug=example.test, rather than creating blog in the parent site's container. Although no current core scenario uses this step, it will target the wrong container when used. Build the container name from $site and pass $subsite as the slug.
Replace internal tracker IDs and local paths in the coverage notes and test comments with short descriptions of each bug or links to the pull requests that fixed it.
compareContents() compared list elements by position, so [1,3,5] wasn't found in [1,2,3,4,5], and a missing key or property raised a notice. The output steps already compare the whole expected text.
…-uninstall prompt
/var/tmp/ee-behat-core was used as found, so a pre-created directory or symlink could redirect root's writes and rm -rf. It is now created 0700 and refused unless it is a real directory owned by the running user and not writable by others. The declined self-uninstall scenario passes --no-yes, so a yes: true default in the config can't skip the prompt.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces core's old
features/with a Behat suite of 250 scenarios (outline rows counted) and adds aBehatworkflow that runs it on every pull request, on pushes todevelopand on demand.It covers
ee cli(version, info, check-update and update, dumps, completions, aliases and alias groups, has-command), help for every command, unknown commands, global flags, config files, the requirement checks, migration triggering and the image migration rollback, the phar build and the self-uninstall.features/COVERAGE.mdmaps each command, flag and behaviour to its scenarios.behat.ymladds profiles: the default one skips@destructivescenarios, and@destructive,@migrationand@uninstallare also refused at runtime unlessEE_BEHAT_DESTRUCTIVE,EE_BEHAT_ALLOW_MIGRATIONorEE_BEHAT_ALLOW_UNINSTALLis set, since a CLI--tagsreplaces the profile filters. On a machine withoutee.sqlitethe suite also needsEE_BEHAT_ALLOW_INIT=1.@destructiveincluded since the runner is thrown away), migration (the real upgrade pipeline) and uninstall, each in its own job.test_and_build.yml: the test job setsEE_BEHAT_ALLOW_INIT=1(the runner starts withoutee.sqlite) andGITHUB_TOKEN(check-update scenarios call the GitHub API), and no longer runs the site-command and site-type-wp suites. Both packages run their own Behat job on every PR, and their new suites, with lanes for known bugs and a local Pebble ACME server, would fail when run here unfiltered. site-type-php's suite still runs here.To run it locally (as root, on a test machine):
sudo -E vendor/bin/behatruns the default profile, which leaves global state alone;-p destructivewithEE_BEHAT_DESTRUCTIVE=1runs the rest except the migration and the uninstall.Green runs on my fork: Behat https://github.com/mrrobot47/easyengine/actions/runs/39292388125, Build + Test https://github.com/mrrobot47/easyengine/actions/runs/39292388227