[matthewgrossman]

Summary

The required conformance-integration job moves from the old tmachine conformance path to the new test runner, and the conformance tests it runs move onto testkit.

	Old state	New state (this PR)
Tests	The old conformance/cli Cargo suite (crates/openshell-conformance scenarios), built as a nextest archive.	The same coverage rewritten on testkit in tests/suites/conformance/sandbox: cli, lifecycle, file-transfer, persistent-workspace and policy-advisor, next to smoke.
How conformance-integration runs them	Through the tmachine conformance testsuite (an Ansible playbook) via integration-runner.yml.	Through openshell-test-runner, via e2e-suites.yml. Same job ID, still required.
Environments	Docker/deb, Podman rootful and rootless/rpm, k3s, HA k3s.	The same five.
Testsuites in tmachine	conformance and policy-advisor.	Deleted, along with their playbooks and the old conformance test archive.
Branch E2E	A separate e2e-suites job running the smoke row and #4891's Keycloak row.	That job keeps running, but only the features family (the Keycloak row). The conformance rows run in conformance-integration, so neither run duplicates the other.
k3s failure evidence	The playbook saved k3s and controller diagnostics.	The runner saves the same diagnostics, and checks the Agent Sandbox controller after the guest boots.

What does not change in this PR: the old CLI suite (tests/suites/conformance/cli, e2e/support/conformance.sh) still runs in the Docker, Podman, Kubernetes and VM host lanes and mise run e2e:rust:push. It moves in the next PR (PR 5b). Nor do crates/openshell-conformance, the features and drivers suites, or e2e/rust.

The lifecycle, persistent-workspace, file-transfer and policy-advisor ports start from @elezar's draft #4629.

Context

Builds on the runner merged in #4773 and on #4891 (the features pattern, with the Keycloak example), which is now merged; this PR is based on main with it.

Related Issue

Part of #3954

Changes

• testkit: cx.track and cx.adopt, for sandboxes a test creates through the CLI.
• Suites: tests/suites/conformance/sandbox gains lifecycle, cli, persistent_workspace, file_transfer and policy_advisor next to smoke. persistent_workspace_survives_stop_start stays in conformance: every conformant driver keeps the workspace across stop and start. TESTING.md says what a conformance test may assert.
• tmachine: Git in every test environment (test-tools.yaml), for the .gitignore filtering test; an Agent Sandbox readiness check in the k3s installers (k3s-controller-ready.yaml, moved out of the old conformance playbook), so tests don't wait for the controller. The runner repeats the check after it boots the guest (wait-k3s-controller.sh), because a booted guest restarts k3s and the controller; a failure there is an environment error, not a test failure.
• e2e-suites.yml: a select job holds six rows: the five conformance rows (Docker deb, Podman rootful and rootless rpm, k3s and k3s HA, running the sandbox package) and test(features): run feature tests against environment services, starting with Keycloak #4891's Keycloak row (features, keycloak-provider-refresh). A gateway column says which gateway each installer registers (tmachine for the k3s installers, openshell for the others). A family input filters the rows, so each caller gets only its own rows and never a run where every row is skipped. The disk cache uses test(features): run feature tests against environment services, starting with Keycloak #4891's key (environment, installer and the tmachine setup inputs, no restore-keys fallback); the k3s environment still skips it, as integration-runner.yml does (its history doesn't say why, so that is still to find out).
• Required lane: conformance-integration (same job ID; Branch E2E, Release Dev and Release Qualification depend on it) now runs the conformance rows through the runner (family: conformance) and passes CACHIX_AUTH_TOKEN. Branch E2E keeps test(features): run feature tests against environment services, starting with Keycloak #4891's e2e-suites job and its core-e2e-result entry, now with family: features.
• Deleted: the tmachine conformance and policy-advisor testsuites, their playbooks and the old conformance test archive. The integration-runner and integration-test defaults no longer name conformance.
• Parity, k3s diagnostics: on k3s and HA k3s, after a failed run the runner runs the old playbook's collect-k3s.sh (carried over unchanged except for its output handling) and saves diagnostics/k3s.txt; the systemd gateway journal is still saved for every run. Without it the newly required k3s rows would fail with no gateway logs. The script runs under a time limit in the guest, and what it wrote is copied out even when it ran out of time. A failure to collect never changes the test result. It has not yet run on a real k3s failure.
• Docs: CI.md, the policy-advisor README and the watch-github-actions skill point at the runner and its e2e-results-* artifacts.

Old test to new test

Old	New	Notes
smoke (scenario: status, create, get Ready, list, exec, delete, list empty)	cli.rs::cli_create_exec_list_delete	Same CLI steps. smoke.rs::create_exec_delete is the SDK variant (in #4773).
sandbox_lifecycle stop_start_preserves_workspace	persistent_workspace.rs::persistent_workspace_survives_stop_start (SDK) and cli.rs::cli_stop_start_delete (CLI)	Also lifecycle.rs::restart_stopped_sandbox.
sandbox_lifecycle stopped_can_be_deleted	lifecycle.rs::stopped_sandbox_can_be_deleted	Also at the end of cli_stop_start_delete.
file_transfer::round_trip	file_transfer.rs::upload_download_round_trip
file_transfer::git_filtering	file_transfer.rs::upload_respects_gitignore	Needs git where the tests run.
file_transfer::path_safety	file_transfer.rs::download_stays_inside_workspace
policy_local	policy_advisor.rs::policy_local_accepts_proposals
mechanistic_proposal	policy_advisor.rs::denied_address_produces_draft_rule
new_hostname_proposal	policy_advisor.rs::denied_hostname_produces_draft_rule

New with no old counterpart (from #4629): lifecycle.rs::long_running_exec_delete, run_to_completion_exit_delete, default_command_sandbox_is_listed_ready.

Not here

• The old CLI suite (tests/suites/conformance/cli, e2e/support/conformance.sh) stays until the host lanes move to the new tests in the next PR (PR 5b). The required Docker, Podman, Kubernetes and VM lanes and mise run e2e:* still use it.
• features and drivers (oci-image, Podman userns, e2e-podman) and deleting crates/openshell-conformance: later.
• arm64: needs an arm64 KVM runner and arm64 build inputs.

Testing

• Checks appropriate to the affected code and behavior pass
  • Locally: fmt (runner, tmachine, testkit), clippy -D warnings, runner and tmachine tests, lockfile alignment, license headers, and Nix evaluation of the runner, tmachine config and test archives.
  • On CI: Branch E2E run 40932543672 on an earlier head (4e6a9c4d8) passed: all five conformance-integration rows ran in tmachine VMs (Docker/deb, Podman rootful and rootless/rpm, k3s, HA k3s), along with the Select E2E suites and result jobs.
  • The k3s diagnostics, the post-boot controller wait and the persistent-workspace poll were added after that run. Branch E2E run 41098411898 on an earlier head (363f08150) passed.
  • Flake stress test: with the build artifacts from that run, e2e-suites.yml ran with family: conformance five times in parallel (run 41114006500, concurrency limit removed on a throwaway branch only). All 25 row-runs passed, 5 of 5 for each of Docker/deb, Podman rootful/rpm, Podman rootless/rpm, k3s and HA k3s. Counting the earlier green run, each row has passed 6 times. This includes the controller wait, the diagnostics code and the poll. Not covered: the k3s diagnostics only run after a failure, so they haven't run on a real failure, and five runs can't show a flake rarer than about 1 in 5.
  • This head (6726561d3) is rebased onto main after test(features): run feature tests against environment services, starting with Keycloak #4891 and fix(test): drop IPv6 loopback redirect from the Keycloak fixture realm #5089 merged. Branch E2E run 41387833584 and Branch Checks run 41387833616 passed on it: all five conformance-integration rows and the Keycloak e2e-suites row.
• Unit tests added/updated
• E2E tests added/updated

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, CI.md)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[matthewgrossman]

/ok to test 4e6a9c4

[matthewgrossman]

/ok to test 363f081

[matthewgrossman]

/ok to test 6726561

[elezar]

Thanks for taking this over @matthewgrossman. LGTM.

[elezar]

Non-blocking suggestion for the testkit tracking API.

[elezar]

Could track return a tracked handle, with adopt(self) on that handle, rather than requiring callers to repeat the name across two independent context methods?

let pending = cx.track(&name);
cx.cli(["sandbox", "create", "--name", &name, "--detach"])
    .await?
    .success()?;
let sandbox = pending.adopt().await?;

track would still register cleanup immediately, before the CLI can partially succeed. The handle would carry the name and context needed for adoption, and adopt(self) would consume it to return the Sandbox. Cleanup would remain owned by Context, so a CLI failure, failed adoption, or an unused handle would still leave the sandbox registered for cleanup; no async Drop is needed. This makes the sequencing explicit and prevents tracking one name and accidentally adopting another. Non-blocking API suggestion.