Skip to content

test: retire the old e2e test system onto testkit #5281

Open
5 of 11 issues completed
Open

Description

@matthewgrossman

User Story

I maintain OpenShell's end-to-end tests and CI. I need one place that shows how far the move from the old test system to testkit has come, what is in flight, and what is next, so reviewers and other contributors can follow it without reading every PR.

Description

OpenShell's end-to-end tests are moving from the old system to the new one:

  • Old: the openshell-conformance test library and its CLI suite (both now deleted), tmachine tests run through Ansible playbooks, and the ~70 tests in e2e/rust.
  • New: tests written with testkit under tests/suites/, run inside a tmachine VM by the runner, or against any existing gateway.

This issue tracks the remaining work to retire the old system, plus the flake and CI-time fixes made alongside it. Each piece of work keeps its own issue and PR. Sub-issues are listed below this description.

This issue covers part of the scope of #3954 (testing strategy, closed by the stale bot). RFC 0016 and the broader testing strategy stay out of scope here.

Context

Done:

In progress:

Next:

  1. Move the e2e/rust tests in batches: Podman, Docker, the common tests, gateway reconfiguration, Kubernetes, then VM, GPU, performance and edge tunnel. Each batch deletes what it moves.
  2. Run the prebuilt openshell-conformance program on the runner rows, Release Dev, qualification and GPU lanes.
  3. Delete e2e/rust and the old tmachine e2e-podman testsuite. Move each remaining host-only CI lane to a tmachine environment, or keep it on the host with a written reason.

Definition of Done

  • e2e/rust is deleted, and its tests run on testkit.
  • The tmachine e2e-podman testsuite and the Rust steps in the e2e wrapper scripts are deleted.
  • CI, Release Dev and release qualification run only testkit suites.
  • Each remaining host-only CI lane has a written reason it can't run on tmachine.

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions