Repository navigation
test(e2e): run the testkit conformance suite in the host lanes - #5078
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test e0d2385 |
|
🌿 Preview your docs: https://nvidia-preview-pr-5078.docs.buildwithfern.com/openshell |
|
Label |
363f081 to
6726561
Compare
The Docker, Podman, Kubernetes and VM host lanes, including external-driver mode and e2e:rust:push, now run the testkit `sandbox` package with nextest instead of the old CLI conformance package. The tests find the lane's gateway the same way the CLI does, through the OPENSHELL_GATEWAY_ENDPOINT and OPENSHELL_GATEWAY values the gateway wrappers export, and `--no-tests fail` makes a lane that selects no tests fail. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Nothing runs tests/suites/conformance/cli now that the host lanes run the testkit suite. Remove it from the conformance workspace and its lockfile, drop the e2e:testkit exclusion for it, and point the TESTING.md manual run at the `sandbox` package. e2e:rust:push also loses its dependency on e2e:conformance:build, which no longer exists, and the unused OPENSHELL_CONFORMANCE_BIN it set. crates/openshell-conformance stays for the features and drivers suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
e0d2385 to
41426f7
Compare
|
/ok to test 41426f7 |
The conformance workspace has one package now that the old CLI suite is gone, so run the whole workspace. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test 81cdd4e |
Conformance is one crate per family, so the family directory is the crate: tests/suites/conformance/sandbox becomes the `conformance` package at tests/suites/conformance, with its suites still in tests/. The workspace manifest path, its lockfile and the scripts that run it by path are unchanged. Nextest's binaries for these tests are now `conformance::<suite>`; test names are unchanged. The tmachine runner rows, the fmt and clippy lines and the docs name the new package. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test b6fad3c |
|
Pushed b6fad3c: renamed the conformance suite crate from |
…to-testkit/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
/ok to test aa7ce2e |
Summary
Run the testkit conformance suite in the host E2E lanes, and delete the old CLI conformance suite it replaces. This is a parity change: every lane keeps its current gateway and driver setup.
Before:
tests/suites/conformanceheld two overlapping suites.e2e:rust:push) ran the oldclipackage withcargo test: 8 tests that run scripted scenarios fromcrates/openshell-conformanceagainst the CLI.sandboxpackage (tests/suites/conformance/sandbox) from test(suites): port the conformance suites onto testkit and run conformance-integration on the runner #4774 (15 tests, covering the old suite's scenarios and more) ran only in theconformance-integrationrunner lanes andmise run e2e:testkit.After: there is one conformance suite.
cargo nextest, one test at a time. A run that selects zero tests fails.tests/suites/conformanceitself is now theconformancepackage (its own workspace), with the suites still intests/*.rs. The manifest path and lockfile location are unchanged; the runner rows, fmt/clippy lines and docs now nameconformance, and nextest binaries areconformance::<suite>.clipackage is deleted.crates/openshell-conformancestays, because other suites still use it: itsOpenShellRunnercommand helper is used by the oldfeatures/oci-image,features/provider-refresh/keycloakanddrivers/podmansuites. A follow-up PR, test(e2e): migrate the features and drivers suites to testkit #5137, moves those to testkit and then deletes the crate.aws-lc-sysand so on), against a handful for the old suite, and each lane compiles them from scratch. A follow-up PR, feat(conformance): add the openshell-conformance program #5138, removes this by building the tests once per architecture as oneopenshell-testprogram that every lane runs. The tests themselves take about as long as the old 8 did. In CI, every lane stays well within its timeout.Based on
main, which now includes #4774.Related Issue
Part of #3954
Changes
e2e/support/conformance.shrunscargo nextest runon thetests/suites/conformanceworkspace with--no-tests fail, so a lane that selects zero tests fails. All host lanes (Docker, Docker external-driver, Podman, Kubernetes and its variants, VM, VM external-driver, ande2e:rust:push) go through this script, so none of the lane wrappers changed.tests/suites/conformance/cli, its workspace member, and its lockfile entries.crates/openshell-conformancestays for the suites that still use it.tasks/test.toml:e2e:testkitno longer excludes the deleted suite.e2e:rust:pushdrops a dependency one2e:conformance:build, a task that no longer exists.TESTING.md: updated the suite and lane descriptions and the manual-run command.Gateway selection per lane:
OPENSHELL_GATEWAYfor an mTLS gateway registered in a temporary config folder.OPENSHELL_GATEWAY. The gateway is registered without TLS, or with mTLS for the TLS and OpenShift variants. Operator workspace mode still skips conformance.OPENSHELL_GATEWAY_ENDPOINT. testkit matches it to the active registered gateway and uses that gateway's mTLS certificates.Testing
cargo fmt,cargo clippy -D warnings(conformance package), lockfile alignment,markdownlint-cli2 TESTING.md,mise run license:check, shellcheck (no new warnings)Local runs, 15/15 conformance tests passing in each:
VM gateway selection was checked by recreating the VM lane's environment on Docker (only
OPENSHELL_GATEWAY_ENDPOINTset). Running with a filter that matches no tests exits non-zero.Not verified locally: the real VM and VM external-driver lanes, Kubernetes external-driver, and the Kubernetes TLS and OpenShift variants. The Docker E2E CI image does not include
cargo-nextest.miseinstalls the pinned version in that image (checked locally), at the cost of one download per job.Checklist