Repository navigation
Conversation
The Warden environment already brings up nginx, Varnish and Traefik, but nothing installed into it is reachable from the runner itself. Integration tests never noticed, because they drive Magento's own bootstrap rather than making HTTP requests. Three things have to be true before a browser can reach the storefront, and each is added here behind a default that preserves current behaviour: - setup:install needs --base-url, or Magento redirects to its default base URL regardless of where the environment is served - the Traefik hostname needs resolving on the runner, which has no dnsmasq - static content needs deploying, or the storefront returns 200 and renders nothing Also exposes the served URL as an output, so a caller does not have to reconstruct it, and makes the admin frontName and the Warden environment naming configurable rather than hardcoded -- a predictable admin URL is a prerequisite for testing the admin at all. Existing callers pass none of the new inputs and are unaffected.
Adds the canary for the browser-serving inputs on warden/setup-environment: it installs Mage-OS, serves it, deploys Luma's static content and drives a browser at the result, following the _internal-* convention the other actions in this repository use. The suite lives with the demo package, as the dummy e2e test already did. It stays deliberately small: its job is to prove a storefront is being served, not to cover a store's behaviour. The assertions are chosen to fail against things that should not pass. A 200 is not evidence of a storefront -- the admin login page, a CMS no-route page and a Magento error page all answer 200 with a header, a footer and a title -- so the suite checks the resolved route in the body class, Luma's minicart block and whether a stylesheet actually parsed into rules. Without the last one it would pass against a store whose pub/static was never deployed. The suite now refuses to start without PLAYWRIGHT_BASE_URL rather than falling back to a public demo store, since a suite that reports green about a store nobody asked about is worse than one that will not run.
The first run failed in the post phase of the Warden step: the local action could no longer be found. warden/setup-environment checks the Warden CLI out into warden/ at the workspace root, which is exactly where this repository keeps that action, so the checkout landed on top of it. Checking the repository out into actions/ leaves the workspace root free for the Warden CLI, which the steps reach as $(dirname $(pwd))/warden/bin/warden.
The README told the reader to point PLAYWRIGHT_BASE_URL at the admin URL
and watch the test go red. It does not go red: page.goto("/") resolves
against the origin and drops the path, so the run hits the homepage again
and passes -- the exact vacuous green this suite exists to refuse.
Replacing it with a throwaway spec that navigates to the admin path
directly, and the output it produces against a real store.
trace: on-first-retry only produces a trace once a test has failed and been retried. A gate that fails is read by someone deciding whether to ship, and "re-run it and watch" is not an answer at that point, so the first failure now carries its own trace. Report artifacts are kept for 30 days rather than 7, long enough to outlive a release weekend.
The homepage check proves a store is served. It does not prove a store can take money. This walks the deepest path a storefront check can: category listing, product page, cart, shipping address, rate selection, payment, order. It touches the price index, the quote, the shipping and payment method pools, order placement, and the private-content sections Varnish has to punch holes for. That path needs a salable product, which a bare setup:install does not have. fixtures/mageos-e2e.xml seeds the least data it takes: 20 simple products with deterministic SKUs, one configurable, two categories in the main navigation. 12s wall clock on Mage-OS 4.0.0 under Warden, including the reindex the command runs itself, and idempotent. Core ships no profile for functional seeding -- the smallest of the five performance profiles generates 800 products, 100 images and 80 orders, because those profiles exist to benchmark a store under catalog weight. Sample data was the other option and is worse for a gate: its content changes between releases, so an assertion on a product name is an assertion about sample data rather than about Mage-OS. Verified both directions against a Warden store: green with three persisted guest orders carrying Flat Rate and a $6.00 grand total, and red at the rate-selection step with carriers/flatrate/active set to 0.
The checkout spec needs something to buy, and the install the workflow stands up has nothing. The profile is copied into the project directory because the command runs inside the container, which sees only that. It reindexes itself, so no indexer:reindex follows it.
The runner cannot write into the project directories the container has written to, because those are owned by the container's user: cp: cannot create regular file 'var/mageos-e2e.xml': Permission denied Piping the profile through exec avoids host permissions entirely and leaves nothing behind in the project. Keeping the profile outside setup/performance-toolkit means its di.xml is not loaded, which only mocks the mail transport -- noted in the profile, and nothing here generates customers or orders.
Which product a category lists first is not ours to decide, and on a clean install it is the configurable. That will not go into the cart until every attribute has a value, so the first CI run failed waiting for a success message that was never coming -- it had a validation error instead. It passed locally only because a leftover simple product happened to sort ahead of it. Selecting the first available value for whatever the page asks for makes the step indifferent to catalog order, and lets the spec run against a simple product, a configurable, or a merchant's own catalog. Verified against a configurable locally: one dropdown, three values, first selection adds to the cart.
_internal-e2e.yaml proves the actions work, but it cannot be reused: it references them through "./", and a relative path in a reusable workflow resolves against the caller's checkout, which is not this repository when the caller is the release pipeline. So own-repo actions are referenced by full ref here, as integration.yaml and full-integration-tests.yaml already do. test_command is an input rather than a hardcoded "npx playwright test". The workflow's job is to stand a store up and hand over a URL; it should not know what runner points at it, and a caller wanting a grep filter, a shard or a different reporter should not need a new input for each. It arrives through the environment rather than interpolated into the script, so quoting is bash's problem and not the expression engine's. Evidence policy stays in the suite's config, so the default command carries no flags: a CLI flag overrides the config, and --trace on records a trace for every passing test. fixture_profile seeds a catalog before the run, because a bare setup:install has nothing to buy. Empty skips it, which is right for a suite that asserts only on what a bare install provides. Verified green end to end on a throwaway branch, with the Warden action pointed at the stacked serving branch: store served, catalog seeded in 9s, 2 specs passed. At @main it needs the serving inputs, so this cannot go green until mage-os#395 is on main.
Contributor
Author
|
Folded into #396 instead. Stacking this separately bought no review isolation: because the branch targeted
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
e2e.yaml, aworkflow_callworkflow that stands a Mage-OS store up, serves it, and points a browser-based test runner at it. This is the piece the release pipeline can call;_internal-e2e.yaml(#396) cannot be reused, because it references the actions through./and a relative path in a reusable workflow resolves against the caller's checkout.Own-repo actions are therefore referenced by full ref, as
integration.yamlandfull-integration-tests.yamlalready do.Stacked
warden/setup-environmenttest_directoryandfixture_profilepoint at the suite that lands thereThis cannot go green until #395 is on main, because
warden/setup-environment@maindoes not yet havebase_url,configure_host_resolutionorrun_static_content_deploy. Draft until then.Verified
Run on a throwaway branch with the Warden action pointed at #395's branch:
HTTP/2 200on the resolved hostnameTotal execution time: 00:00:092 passed— the storefront smoke test and the guest checkoutTotal 6m31s, of which 2m26s was a runner apt mirror running at 145 kB/s during
playwright install --with-deps. The equivalent canary run is 4m10s. Nothing in the workflow accounts for the difference — the Chromium download itself was 6 seconds — but it is a fair reason to start non-blocking.test_commandtest_commandis an input rather than a hardcodednpx playwright test. The workflow's job is to stand a store up and hand over a URL; it should not know what runner points at it, and a caller wanting a grep filter, a shard or a different reporter should not need a new input for each.Two details worth a reviewer's attention:
trace,video,screenshot) belongs in the suite's config, where it is reviewable — a CLI flag overrides the config, and--trace onrecords a trace for every passing test.The
test_commandinput and the idea of a setup phase are both from @digitalrisedorset's #313. The README documents the setup phase as Playwright projectdependenciesrather than two--grepinvocations, because two invocations write to the same report directory and the second overwrites the first, so a failing setup phase loses its own evidence.Notes for review
fixture_profileseeds a catalog withsetup:performance:generate-fixturesbefore the run, because a baresetup:installhas nothing to buy. Empty skips it.supported-versionmatrix. A release gate has to install the version being released, which by definition is not insupported.jsonyet — somagento_versionandmagento_repositoryare passed directly.integration.yamland that the common parts should be extracted. That is still true and still worth doing separately; this does not make it worse.PR Checklist
e2e-README.md)PR Type
Does this PR introduce a breaking change?