Repository navigation
Open
5 / 115 of 11 issues completedLabels
Description
User Story
I maintain OpenShell's end-to-end tests and CI. I need one place that shows how far the move from the old test system to testkit has come, what is in flight, and what is next, so reviewers and other contributors can follow it without reading every PR.
Description
OpenShell's end-to-end tests are moving from the old system to the new one:
- Old: the
openshell-conformancetest library and its CLI suite (both now deleted), tmachine tests run through Ansible playbooks, and the ~70 tests ine2e/rust. - New: tests written with testkit under
tests/suites/, run inside a tmachine VM by the runner, or against any existing gateway.
This issue tracks the remaining work to retire the old system, plus the flake and CI-time fixes made alongside it. Each piece of work keeps its own issue and PR. Sub-issues are listed below this description.
This issue covers part of the scope of #3954 (testing strategy, closed by the stale bot). RFC 0016 and the broader testing strategy stay out of scope here.
Context
Done:
- test(testkit): add the testkit library for end-to-end suites #4724 (testkit), refactor(tests): move tmachine support files under tests/tmachine #4726 (tmachine layout), feat(tests): add the tmachine Session API and run a conformance smoke test on tmachine #4773 (runner)
- test(features): run feature tests against environment services, starting with Keycloak #4891 (features pattern, Keycloak example)
- test(suites): port the conformance suites onto testkit and run conformance-integration on the runner #4774 (conformance tests on testkit, required on 5 VM setups)
- test(e2e): run the testkit conformance suite in the host lanes #5078 (host lanes run the new tests; old CLI suite deleted)
- test(testkit): add cleanup hooks that run after pass, failure, or panic #5219 and test(e2e): migrate the features and drivers suites to testkit #5137 (testkit cleanup hooks; OCI image, Podman user namespace and Keycloak tests moved; old library deleted)
In progress:
- test(testkit): add the openshell-conformance test program #5249 (with docs(conformance): drop playbook mention from adding-a-test section #5247 and test(testkit): name unsupported gateway auth modes in the error #5248): package the conformance tests as one program,
openshell-conformance, built once per CI run - test(tmachine): prepare runner environments for the e2e/rust migration #5277 (test(tmachine): prepare runner environments for the e2e/rust migration #5275): prepare the runner environments for the
e2e/rustmigration:host.openshell.internalon k3s, the Python workload image in every VM, runner timeouts and test groups - Flakes: test(e2e): split credential_gating into independent parallel tests #5264 (test(e2e): credential_gating end-to-end test times out on the Kubernetes Rust smoke lanes #5263,
credential_gatingtimeouts on Kubernetes), fix(podman): keep a polled stale exit from failing an in-progress start #5265 (bug(podman): polled snapshot moves a restarting sandbox from Starting to Error (ContainerExited) #5258, Podman restart race) - CI time: ci(e2e): start Linux and tmachine jobs without waiting on macOS or package builds #5270 (ci: start Branch E2E Linux and tmachine jobs without waiting on macOS builds or package builds #5268, start Linux and tmachine jobs earlier), then ci: start Release Dev Linux jobs without waiting on macOS builds #5269 (the same change for Release Dev)
Next:
- Move the
e2e/rusttests in batches: Podman, Docker, the common tests, gateway reconfiguration, Kubernetes, then VM, GPU, performance and edge tunnel. Each batch deletes what it moves. - Run the prebuilt
openshell-conformanceprogram on the runner rows, Release Dev, qualification and GPU lanes. - Delete
e2e/rustand the old tmachinee2e-podmantestsuite. Move each remaining host-only CI lane to a tmachine environment, or keep it on the host with a written reason.
Definition of Done
-
e2e/rustis deleted, and its tests run on testkit. - The tmachine
e2e-podmantestsuite and the Rust steps in the e2e wrapper scripts are deleted. - CI, Release Dev and release qualification run only testkit suites.
- Each remaining host-only CI lane has a written reason it can't run on tmachine.
Activity