[matthewgrossman]

Summary

Port the remaining suites that still use crates/openshell-conformance (features/oci-image, features/provider-refresh/keycloak and the Podman user-namespace suite in drivers/podman) to testkit, and run them through e2e-suites.yml with explicit CI rows. This is a parity change: each suite keeps its current environments and installers. The last commit deletes crates/openshell-conformance.

Related Issue

Part of #3954

Built on #5219 (testkit cleanup hooks), the base of this PR until it merges. #5219's branch is merged in, not rebased, so its commits keep their SHAs.

Changes

• Support crates (tests/support, packages named <name>-support): container-engine-support builds images into the gateway's image store through OPENSHELL_TEST_CONTAINER_ENGINE. podman-support reads OPENSHELL_TEST_PODMAN_USERNS and computes Podman's own UID map for a mode. testkit gains Context::expect_compute_driver and testkit::base_image() (OPENSHELL_TEST_BASE_IMAGE, or the default sandbox image). The support unit tests run in the Rust test lists. The existing Keycloak fixture is renamed to keycloak-support to match; its directory doesn't change.
• OCI image (tests/suites/features/oci-image, package renamed to oci-image): rewritten as #[testkit::test] tests. The Docker, rootful Podman and rootless Podman environments publish OPENSHELL_TEST_CONTAINER_ENGINE (docker, sudo -n podman, podman). mise run e2e:oci-image(:podman) and the ARM64 Docker lane use the new package name.
• Keycloak provider refresh: the testkit suite (keycloak-provider-refresh, from test(features): run feature tests against environment services, starting with Keycloak #4891) already matches the old test. This PR adds the fedora-podman-{rootful,rootless}-keycloak environments for it and deletes the old provider-refresh/keycloak crate. On Podman, the keycloak role now starts the container with --restart always and enables podman-restart.service (the system unit when rootful, tmachine's user unit when rootless), so Keycloak comes back when a cached disk boots. Docker keeps unless-stopped.
• Podman user namespaces (tests/suites/drivers/podman, package podman; driver suites are named after the driver, and each is its own workspace): one test compares the sandbox's UID map with Podman's for the configured mode and checks that the non-root workload owns and can write /sandbox. As in the old playbooks, the reference container runs and is removed before the sandbox is created: in auto mode Podman gives each running container its own range, so the sandbox gets the range the reference released.
  • Eight environments, fedora-podman-{rootful,rootless}-userns-{default,auto,keep-id,private}, run playbooks/drivers/podman/userns.yaml. It writes the mode's [openshell.drivers.podman] settings to /etc/openshell-test/podman-driver.toml and publishes OPENSHELL_TEST_PODMAN_USERNS. In default mode it removes that file, so gateway.toml has no userns, uidmap or gidmap settings (the old baseline asserted this).
  • The openshell_gateway role (binaries installer) adds those settings to gateway.toml before the gateway starts. The packaged installers fail on such an environment instead of testing without the mode.
  • This replaces the in-VM chaining of default-userns-baseline.yaml, tests.yaml and the userns-*.yaml playbooks.
• CI:
  • feature-specific-integration (Branch E2E, Release Dev, Release Qualification; job ID kept) now calls e2e-suites.yml with family: features. Branch E2E's separate e2e-suites job is folded into it.
  • driver-specific-integration (Branch E2E) calls it with family: drivers, which selects every driver workspace (drivers/<driver>).
  • The tmachine e2e-podman row is unchanged and moves to its own podman-e2e-integration job, which is added to the Core E2E result.
  • The manual Integration Tests workflow defaults to the e2e-podman row.
• Removed: the tmachine oci-image, provider-refresh and driver-podman testsuites, their playbooks, fixtures and test archives.
• Workspaces: conformance and features are one workspace each, and each driver suite is its own workspace with its own lockfile (tests/suites/drivers/podman), so one driver's tests and dependencies can't affect another's build or cache. There is no drivers workspace. The runner's --family is the path of a workspace under tests/suites/ (drivers/podman) and rejects absolute paths and ... Driver rows in e2e-suites.yml name that path and no package; job and artifact names are unchanged.
• tests/suites/drivers/podman is added to ALIGNED_WORKSPACES. The fmt/clippy lists (Branch Checks Format and Lint steps, rust:lint, rust:format, rust:format:check) now run once per test workspace (conformance, drivers/podman, features) instead of once per package, so a new feature suite needs no lint-list edit; a new driver suite adds one line to each list. This also lints features/kubernetes-updates for the first time; it raises no warnings. TESTING.md and CI.md are updated.

CI rows after this PR (e2e-suites.yml):

Family	Package	Environments	Installer
features	keycloak-provider-refresh	ubuntu-docker-rootful-keycloak	deb
features	keycloak-provider-refresh	fedora-podman-{rootful,rootless}-keycloak	rpm
features	oci-image	ubuntu-docker-rootful	deb
features	oci-image	fedora-podman-{rootful,rootless}	rpm
drivers/podman	(all)	fedora-podman-{rootful,rootless}-userns-{default,auto,keep-id,private}	binaries

The driver suites run only in Branch E2E, as driver-specific-integration does today. The Docker Keycloak row now also runs in Release Dev and Release Qualification, because it belongs to the features family.

Still to come in this PR: nothing. The last commit deletes crates/openshell-conformance, its Cargo.lock entry and its AGENTS.md architecture row. No other code used the crate. tmachine's testsuites list is now just shell (interactive) and e2e-podman. nextest.yaml stays because the runner rows still run nextest in the guest.

Cleanup uses the testkit cleanup hooks from #5219, so it also runs after a failure or panic. The OCI suite registers an image-removal hook right after each build; hooks run last registered first, so the sandboxes created from an image are deleted before it is removed. The Keycloak suite's provider and profile deletes moved to hooks in #5219.

Testing

• Checks appropriate to the affected code and behavior pass: cargo fmt and cargo clippy -D warnings for every list entry, including each whole test workspace, cargo check --workspace --locked, lockfile checks and alignment, mise run rust:deny:policy, mise run license:check, markdownlint-cli2, actionlint (no new findings), zizmor (no new findings against main), ansible-playbook --syntax-check on the changed playbooks, and nix build .#tmachine-config
• Unit tests added/updated (if applicable): tests/support UID-map normalization
• E2E tests added/updated (if applicable)

Local runs:

• mise run e2e:oci-image against a local Docker gateway, before the move to cleanup hooks: 6/6 passed, with no images or sandboxes left behind. Changing an expected UID made the test fail and still clean up. The hook-based cleanup is covered by the OCI CI rows.
• The gateway-podman.toml.j2 template rendered for the default and private modes and parsed as TOML.
• sudo -n podman compared with the old playbooks' context, in a Fedora 44 container: sudo -n sets HOME=/root and drops XDG_RUNTIME_DIR, but rootful Podman's graph root, run root, temp directory and socket are identical to the old HOME=/root XDG_RUNTIME_DIR=/run/user/0 context. Rootless runs as tmachine over SSH, where PAM sets XDG_RUNTIME_DIR.

Not verified locally (tmachine needs an amd64 KVM host): every tmachine row in the table above, including the Podman OCI rows, the eight user-namespace rows, the Fedora Keycloak rows, and Keycloak restarting after a cached-disk boot.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5137.docs.buildwithfern.com/openshell

[matthewgrossman]

/ok to test b07fbc4

[github-actions]

Label test:e2e applied for b07fbc4. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[matthewgrossman]

/ok to test 78af387

[matthewgrossman]

/ok to test 7a8d936

[matthewgrossman]

/ok to test e9281a5

[matthewgrossman]

/ok to test d0d6e10

[matthewgrossman]

/ok to test 988d5c3

[elezar]

Proposal for discussion, not a request to change this PR: let a testkit test register custom cleanup for resources testkit doesn't track.

Problem. Testkit deletes tracked sandboxes after a pass, failure or panic, but the OCI suite (Oci::run and Engine::remove_images) and the Keycloak provider suite clean up inline after the body. A panic in the body skips that and leaks images, providers and profiles. The PR description notes this as a known limitation.

Sketch.

cx.on_cleanup("remove images", async move || engine.remove_images().await);

• Context keeps one ordered hook list, registered through &self like track.
• Hooks run in LIFO order after the body, whether it passes, fails or panics. Each has CLEANUP_TIMEOUT and is wrapped in catch_unwind.
• A failed or panicking hook is reported alongside the existing cleanup failures, the remaining hooks still run, and a passing test fails.
• Sandbox tracking becomes the first user of the same list: track and the builder register a delete-and-wait hook keyed by sandbox name (so track then adopt registers once, and Sandbox::delete removes it). Then LIFO gives dependency order for free: an image built before a sandbox is removed after it, which is what Oci::run does by hand today.
• Trade-off: sandbox deletes are concurrent today. Sequential hooks could take up to N x CLEANUP_TIMEOUT if the gateway stalls. Start sequential; batch adjacent sandbox hooks later only if it matters.

Effect here. Oci::run would drop its sandboxes vec and delete_sandboxes, and register image removal once. Keycloak would register the provider and profile deletes right after creating them (failures would then fail the test, which is stricter than today). Engine and Cli would need to be cheap to clone or wrapped in Arc.

Tests. Unit tests for ordering, failure and panic reporting. A deliberate-failure test that registers a hook which writes a marker, then panics, and asserts the test fails and the marker exists (the idea from #4464).

Where. Since this PR is what introduces the leaking cleanup, it seems best to land the testkit change first as its own small PR and rebase this one onto it, rather than ship the inline cleanup and rewrite it right after. Happy to discuss, @matthewgrossman.

[matthewgrossman]

/ok to test c41046c

[matthewgrossman]

/ok to test 6e0ec21

[matthewgrossman]

/ok to test 6e0ec21

[elezar]

Two small follow-ups, neither blocking.

• userns.rs compares only the UID map, as default_userns.rs did before. Comparing /proc/self/gid_map against the reference too would cover the gidmap side of private mode.
• This adds eight drivers rows to Branch E2E, each booting its own VM (90 minute timeout), and eight feature rows to Release Dev and Qualification. Do we know the typical duration and whether runner capacity is fine? Could the four userns modes share one rootful and one rootless VM by reconfiguring the gateway between runs?

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[matthewgrossman]

/ok to test 3941a00

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. state — No explicit risk acceptance found

oidc_token.json bundles using expiry/expires_in are replaced by expires_at bundles requiring issuer and client_id, with no legacy-reader migration.

• Impact: Existing Go SDK logins cause gateway.NewClient RPC authentication to fail, while oidc.Login starts interactive authentication instead of reusing an unexpired token.
• Action: Continue accepting legacy bundles with their original expiry semantics and provide backward-compatible migration rather than requiring a fresh login on this no-break train.
• PR: test(e2e): migrate the features and drivers suites to testkit #5137

2. go-sdk — No explicit risk acceptance found

RefreshableToken.GetRequestMetadata previously returned cached bearer metadata after refresh failures even past expiry; it now returns the refresh error once the cached token expires.

• Impact: Consumers relying on documented stale-token fallback during token-source outages now receive errors instead of authorization metadata, including during refresh backoff.
• Action: Make fail-on-expiry behavior opt-in while preserving the existing default, or defer the default change to a break-permitting train with actionable migration guidance.
• PR: test(e2e): migrate the features and drivers suites to testkit #5137

Review details

• Execution: complete
• Candidate: PR #5137 (bd9ce9e4678f1040ebf4c1c5935710a0652bc46d)
• Baseline: 057441b3bd59704f8d4658ac779a6301c33d542f (057441b3bd59704f8d4658ac779a6301c33d542f)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

[matthewgrossman]

/ok to test 5749613

[elezar]

/ok to test 7c646f5

[elezar]

Thanks @matthewgrossman. I rebased a and applied some nits. Looks good.