feat(conformance): add the openshell-conformance program by matthewgrossman · Pull Request #5138 · NVIDIA/OpenShell

matthewgrossman · 2026-11-02T18:14:34Z

Summary

Turn the conformance suite into one program, openshell-conformance, that anyone can run against their own gateway without a Rust toolchain, cargo or nextest. #[testkit::test] now also registers each test, and testkit::main() turns a binary into a test program with list, ready and run. The conformance suite becomes a library, and the conformance crate builds the program as a [[bin]]. Existing suites still run under cargo test and nextest.

The program promises only the conformance contract: passing it means a gateway meets the rules in tests/suites/conformance/README.md. Feature and driver suites are not promised to join it. The Docker, Podman, VM and Kubernetes host lanes now run conformance through the program, built once per architecture. Switching the tmachine rows and publishing the program are later PRs.

Related Issue

Part of #3954

Changes

Five commits (the crate rename to conformance landed in #5078):

feat(testkit): register tests and add a standalone test program.
- #[testkit::test] keeps its #[cfg(test)] #[tokio::test] wrapper and also adds a link-time registry entry (linkme) with the test's family, name, description and #[ignore] reason.
  - The family is the directory under tests/suites/. It groups tests in the JUnit report. The name is the same name libtest and nextest use.
  - The hidden function gets only cfg and lint attributes (allow, warn, deny, forbid). #[should_panic], #[cfg_attr] and #[expect] are rejected with an error explaining why.
- testkit::main():
  - With no subcommand, the program takes libtest's arguments (via libtest-mimic), so it also works as a harness = false target under cargo test and nextest.
  - list [FILTER]... prints name and description, with ignored tests marked.
  - ready [--timeout SECS] waits for the gateway, sharing its check with testkit-ready. It exits 2 if the gateway isn't ready in time.
  - run [FILTER]... [--include-ignored] [--timeout SECS] [--grace SECS] [--junit PATH] runs tests one at a time. Each test is a new process (<name> --exact --nocapture) in its own process group, so nothing a test starts outlives it. A test that runs past its timeout (900s by default) gets SIGTERM, then SIGKILL after the grace period (10s). Interrupting run stops the current test and runs no more tests; a second interrupt skips the grace period. Output is shown live and kept (the last 4 MiB per stream) for the JUnit report. run exits 1 when a test failed and 2 when tests couldn't run or the run was interrupted. It is Unix-only; on other platforms it says to use cargo test or nextest.
- tests/testkit/README.md documents the program and its limits.
refactor(conformance): move the conformance suites into a library. Cargo builds each file in tests/ as its own test executable, which no other program can link. tests/suites/conformance/tests/*.rs move to src/ as modules of a lib.rs; test code is unchanged. cargo test and nextest run the same 15 tests as the library's unit tests.
- Nextest names change. conformance::smoke create_exec_delete (binary conformance::smoke) becomes binary conformance, test smoke::create_exec_delete. Filters that picked a suite by binary now pick it by test name, e.g. --filter 'test(/^policy_advisor::/)'. CI.md, e2e/policy-advisor/README.md, the runner's --filter help and the suite README are updated. This break is intentional: these filters are internal test tooling, not a published interface, and every in-repo use is updated here. It is what the advisory compatibility review flags.
- The host lanes (e2e/support/conformance.sh), mise run e2e:testkit and the tmachine runner select the suite by workspace or package, so they still select every test. The kind(test) archive filters in tests/tmachine/artifacts.nix aren't used for conformance.
feat(conformance): add the openshell-conformance program.
- tests/suites/conformance/Cargo.toml adds [[bin]] name = "openshell-conformance" (src/main.rs, test = false). Its main calls testkit::main() and names the library with use conformance as _;.
- Rust only links a dependency that is named, so a suite the program forgets to name would silently go missing. Any testkit program that links no tests now exits 2 with an error under every command.
- tasks/scripts/check-conformance-program.sh checks that openshell-conformance list names the same tests as cargo nextest list for the crate. Branch Checks (Rust test, all three systems) and mise run test:rust run it. Neither command needs a gateway.
- TESTING.md, the testkit README and the conformance README describe the program. The conformance README says what it promises.
test(e2e): run the conformance host lanes with openshell-conformance.
- e2e/support/conformance.sh runs openshell-conformance ready, then run --junit, instead of cargo nextest run. It uses OPENSHELL_CONFORMANCE_BIN when set, and otherwise builds the program from the checkout, so local mise run e2e:docker and the other runners work as before.
- The JUnit report goes to OPENSHELL_CONFORMANCE_JUNIT, else results/conformance-<gateway>.xml. The Kubernetes runner names it after its report, so the two credential-driver runs keep both.
- The lanes now get the program's 900s per-test timeout; nextest had none.
ci(e2e): build openshell-conformance once per architecture in Branch E2E.
- A new build-conformance job builds a static musl binary for x86_64 and aarch64, checks it with verify-static-binary.sh and list, and uploads openshell-conformance-<triple>.
- setup-e2e-cli gets a conformance input that downloads the runner's binary and sets OPENSHELL_CONFORMANCE_BIN. The four lane workflows pass it through a prebuilt-conformance input, and all 15 Branch E2E lane callers set it and depend on build-conformance.
- The job isn't in build-binaries.yml, because the release workflows call that too and the program isn't released. Callers that don't set prebuilt-conformance (Release Dev, release qualification, GPU) build the program in the lane, as commit 4 does.

Why a [[bin]] in the conformance crate, not its own workspace:

The program is conformance-only, so it links one crate. A separate workspace only paid off if it linked suites from several family workspaces. Dropping it removes a 4.5k-line lockfile, an ALIGNED_WORKSPACES entry, and fmt, clippy and nextest lines in Branch Checks and the mise tasks.
cfg(test) wrappers: the bin links the library built without cfg(test), so it gets the linkme registrations and none of the #[tokio::test] wrappers. The library's own unit-test build is unchanged.
test = false: mise run e2e:testkit and the tmachine runner run plain cargo nextest run on the whole crate. A bin test target would add an empty test binary to each. Without one, nextest still sees one conformance binary with 15 tests.
Binary name vs package name: the package stays conformance, so nextest binary IDs and filters such as test(/^policy_advisor::/) don't change again. The binary is named openshell-conformance for the people who run it.
Empty-program check: the old tests/openshell-test/tests/list.rs integration test would have been a test target in the crate. The list-vs-nextest script replaces it, and the runtime guard in testkit::main() catches a program that links nothing.

Name note: crates/openshell-conformance, the existing CLI conformance library, still exists on main until #5137 merges. The new openshell-conformance is a binary in the separate tests/suites/conformance workspace (package conformance), so cargo doesn't conflict. Once #5137 lands, the name has one meaning.

Limits:

Plaintext and mTLS gateways only.
A test stopped by timeout or signal doesn't run its cleanup yet, so its sandboxes stay on the gateway.
Not published, and the tmachine conformance rows still run nextest archives; moving them needs a runner change.

Release Notes

Design agreed with @elezar

This reverses two decisions from #4629: "no custom harness" and "one binary per suite". The reason is that the program should run where the user has no cargo or nextest. It keeps the standard tooling (the libtest interface, so nextest and cargo test work as before), per-test process isolation, and nextest archives in CI. Agreed with @elezar: the libtest-mimic harness, list/ready/run, one process per test and linkme registration, with the program narrowed to conformance and named openshell-conformance.

Testing

Checks appropriate to the affected code and behavior pass:
- fmt and clippy (--all-targets -D warnings) are clean for testkit, conformance (including the bin), features, support and the runner.
- nextest: testkit 31 passed, 3 skipped (ignored macro fixtures); runner 10/10. testkit doctests pass.
- tasks/scripts/check-conformance-program.sh: openshell-conformance list names the same 15 tests as cargo nextest list. The libtest interface (--list --format terse) lists 15 too.
- cargo nextest list on the crate shows one binary, conformance, with 15 tests, and test(/^policy_advisor::/) selects 3.
- Cargo lockfile alignment and the lockfile check pass, and license headers are in place.
Unit tests added/updated: the program checks (testkit/tests/program.rs) run fixture tests through the subcommands. They cover the list format, pass, fail, timeout, SIGTERM ignored, interrupt during and between tests, leftover helper processes, JUnit output, and a program with no tests. testkit/tests/attribute.rs checks that the registry agrees with libtest's --list and --list --ignored.
E2E (as openshell-test, before the move into the conformance crate): against a live mTLS Docker gateway (e2e/with-docker-gateway.sh), nextest ran the conformance workspace with --no-tests fail: 15/15 passed. The program's ready exited 0, and run passed 15/15 and exited 0. Its JUnit report has one conformance suite with 15 test cases. The program code is unchanged by the move; only where it is built changed.
Commits 4 and 5, locally: e2e/support/conformance.sh builds the program when OPENSHELL_CONFORMANCE_BIN is unset, uses it when set, fails with exit 2 when it isn't executable, and passes on ready's exit 2 for an unreachable gateway. actionlint is clean on the changed workflows; zizmor (high) reports no new findings.
CI, commit 4 alone (62e1f57): all 15 conformance host lanes ran openshell-conformance (built in the lane) and passed 15/15. Branch Checks green.
CI, commit 5 (b8a1ef3): build-conformance built static-pie musl binaries for both architectures in about 3 minutes. All 15 host lanes downloaded the prebuilt binary, built nothing, and passed 15/15. Branch Checks and Workflow Security Reports green.

CI coverage: Rust format, Rust lint and Rust test (x86_64-linux, aarch64-linux, aarch64-darwin) in Branch Checks build and test tests/testkit, lint the conformance crate including the bin, and run the list check. Branch E2E builds the program and runs it in every Docker, Podman, VM and Kubernetes host lane. The tmachine conformance rows run the moved suite under its new names with nextest.

Known red, not from this PR: both ubuntu-k3s conformance rows fail before any test runs, because get.k3s.io no longer matches the checksum pinned in tests/tmachine/ansible/playbooks/k3s.yaml. On commit 4's run, fedora-podman-rootless / rpm also failed once: persistent_workspace_survives_stop_start hit a sandbox container that exited on create (ContainerExited). That row runs nextest archives, which commit 4 doesn't touch, and it passed on commit 5's run without a rerun.

Not verified: the non-Unix run path, and the HA, credential-driver and GPU lanes (label-gated, not run here).

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable): tests/testkit/README.md, TESTING.md, tests/suites/conformance/README.md

copy-pr-bot · 2026-11-02T18:14:39Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

matthewgrossman · 2026-11-02T18:53:58Z

/ok to test ca412d5

github-actions · 2026-11-02T18:54:04Z

Label test:e2e applied for ca412d5. Open Branch E2E Checks, find the run for commit ca412d5, and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

github-actions · 2026-11-02T18:54:05Z

🌿 Preview your docs: https://nvidia-preview-pr-5138.docs.buildwithfern.com/openshell

matthewgrossman · 2026-11-02T20:40:45Z

/ok to test 592a748

github-actions · 2026-11-02T20:47:12Z

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

openshell-test-runner replaces --family conformance --package sandbox --filter 'binary_id(=sandbox::policy_advisor)' with --family conformance --package conformance --filter 'test(/^policy_advisor::/)'.

Impact: Existing contributor and downstream CI invocations fail archive creation because the sandbox package no longer exists, while updating only the package leaves legacy suite filters selecting no tests.
Action: Preserve the sandbox package selector and legacy suite filters for v0.1.7, or defer their removal to a minor train with explicit migration notice.
PR: feat(conformance): add the openshell-conformance program #5138

Review details

Execution: complete
Candidate: PR #5138 (2e1b00deb0a0b0a3e8f07214380be9fb6bddf768)
Baseline: 81cdd4ef4c100e701e5172f60538774fe9533f8c (81cdd4ef4c100e701e5172f60538774fe9533f8c)
Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

matthewgrossman · 2026-11-02T21:10:14Z

/ok to test 2fecde6

matthewgrossman · 2026-11-02T23:06:20Z

/ok to test f945820

github-actions · 2026-11-02T23:10:13Z

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. cli — No explicit risk acceptance found

With unchanged environment, installer, family and package arguments, openshell-test-runner --filter 'binary_id(=conformance::policy_advisor)' must become openshell-test-runner --filter 'test(/^policy_advisor::/)' because the suite binary becomes a library module.

Impact: Contributor and downstream CI scripts using the documented binary selector no longer select the policy-advisor tests and fail with runner exit code 2.
Action: Preserve legacy conformance suite selection in v0.1.7, or defer its removal to a minor train with explicit notice and actionable migration guidance.
PR: feat(conformance): add the openshell-conformance program #5138

Review details

Execution: complete
Candidate: PR #5138 (116373046b31e9d809523502be42254bf9f54f27)
Baseline: aa7ce2e0bc1a4e1e69a3dba762f76b3dedf5f376 (aa7ce2e0bc1a4e1e69a3dba762f76b3dedf5f376)
Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run

matthewgrossman · 2026-11-03T02:29:38Z

/ok to test 8835ce6

#[testkit::test] now also registers each test in a link-time registry, alongside the existing #[tokio::test] wrapper, so suites keep running under cargo test and nextest unchanged. testkit::main() runs the registered tests a binary links. Without a subcommand it speaks libtest's interface, so a harness = false target works under cargo test and nextest. Its subcommands are: - list: print each test's name and description - ready: wait for the gateway, sharing its check with testkit-ready, and exit 2 if it is not ready in time - run (Unix only): run tests one at a time, each in its own process group with a timeout (SIGTERM, grace, SIGKILL), write a JUnit report, and exit 1 when a test failed or 2 when tests could not run A stopped test does not clean up its sandboxes yet; the README notes this. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Cargo builds each file in tests/ as its own test executable, which no other program can link. Move the conformance suites into src/ as modules of a library, so a test program can link them. Test code is unchanged; cargo test and nextest run the same 15 tests as the library's unit tests. Nextest now names them as binary `conformance`, test `<suite>::<fn>`, so filters that selected a suite by binary select it by test name instead. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-03T19:08:50Z

/ok to test c03061b

The conformance crate now also builds openshell-conformance, a binary whose main calls testkit::main() and links the crate's library. It lists, waits for the gateway, and runs the conformance tests without cargo or nextest. It runs only the conformance suite and promises only the conformance contract; feature and driver suites are not part of it. The binary has test = false, so cargo test and nextest still see one conformance test binary, the library's unit tests, and the e2e lanes and the runner's filters are unchanged. The library is linked without cfg(test), so the program gets the registrations and none of the tokio test wrappers. tasks/scripts/check-conformance-program.sh checks that openshell-conformance list names the same tests as nextest; Branch Checks and mise test:rust run it. A dependency that is never named is not linked, so a suite the program forgets to name would silently go missing. testkit::main() now fails with exit 2 when a program links no tests, under every command. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-03T19:26:41Z

/ok to test 1d3ed2d

The Docker, Podman, VM and Kubernetes host lanes ran the conformance suite with cargo nextest. They now run it with openshell-conformance: ready, then run --junit, so CI exercises the program that gateway owners use. The script uses OPENSHELL_CONFORMANCE_BIN when set and otherwise builds the program from the checkout. The JUnit report goes to OPENSHELL_CONFORMANCE_JUNIT or results/conformance-<gateway>.xml; the Kubernetes runner names it after its report so credential-driver runs keep both. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Each Docker, Podman, VM and Kubernetes host lane built the conformance suite itself. Branch E2E now builds openshell-conformance once per architecture as a static musl binary, checks that it is static and lists its tests, and uploads it. setup-e2e-cli downloads the one that matches the runner when a lane sets prebuilt-conformance, and the lane runs it through OPENSHELL_CONFORMANCE_BIN. The job is separate from build-binaries.yml, which the release workflows also call: the program has its own workspace and is not released. Lanes called without prebuilt-conformance, such as the release qualification lanes, still build the program from the checkout. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-03T21:17:37Z

/ok to test 62e1f57

matthewgrossman · 2026-11-03T23:02:31Z

/ok to test b8a1ef3

…t-program/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-11-04T00:42:38Z

/ok to test 7531d8a

matthewgrossman added the test:e2e Requires end-to-end coverage label Nov 2, 2026

matthewgrossman mentioned this pull request Nov 2, 2026

test(e2e): run the testkit conformance suite in the host lanes #5078

Merged

4 of 6 tasks

matthewgrossman force-pushed the test/3954-testkit-test-program/matthewgrossman branch from 592a748 to 2fecde6 Compare November 2, 2026 21:09

matthewgrossman force-pushed the test/3954-testkit-test-program/matthewgrossman branch from 2fecde6 to f945820 Compare November 2, 2026 23:06

Base automatically changed from test/3954-host-lanes-to-testkit/matthewgrossman to main November 3, 2026 02:27

matthewgrossman force-pushed the test/3954-testkit-test-program/matthewgrossman branch from f945820 to 8835ce6 Compare November 3, 2026 02:29

matthewgrossman marked this pull request as ready for review November 3, 2026 06:12

matthewgrossman requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners November 3, 2026 06:12

matthewgrossman mentioned this pull request Nov 3, 2026

bug(kubernetes): stopped sandbox regresses to Provisioning after stop completes #5195

Closed

matthewgrossman added 2 commits November 3, 2026 11:07

matthewgrossman force-pushed the test/3954-testkit-test-program/matthewgrossman branch from 8835ce6 to c03061b Compare November 3, 2026 19:08

matthewgrossman changed the title ~~feat(testkit): register tests and add the openshell-test program~~ feat(conformance): add the openshell-conformance program Nov 3, 2026

matthewgrossman force-pushed the test/3954-testkit-test-program/matthewgrossman branch from c03061b to 1d3ed2d Compare November 3, 2026 19:27

matthewgrossman added 2 commits November 3, 2026 13:11

matthewgrossman mentioned this pull request Nov 3, 2026

fix(tests): pin the k3s installer to a release #5223

Merged

4 of 8 tasks

Merge remote-tracking branch 'origin/main' into test/3954-testkit-tes…

7531d8a

…t-program/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Conversation

matthewgrossman commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Release Notes

Design agreed with @elezar

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Compatibility review (advisory)

1. cli — No explicit risk acceptance found

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

matthewgrossman commented Nov 2, 2026

Uh oh!

github-actions Bot commented Nov 2, 2026

Compatibility review (advisory)

1. cli — No explicit risk acceptance found

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 3, 2026

Uh oh!

matthewgrossman commented Nov 4, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

matthewgrossman commented Nov 2, 2026 •

edited

Loading