Repository navigation
feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI - #4773
Draft
matthewgrossman wants to merge 9 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
matthewgrossman
added this pull request to stack #4776
October 20, 2026 23:38
|
🌿 Preview your docs: https://nvidia-preview-pr-4773.docs.buildwithfern.com/openshell |
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
3 times, most recently
from
October 21, 2026 20:41
e06a0eb to
a23e140
Compare
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
2 times, most recently
from
October 22, 2026 04:51
be4af48 to
63f4141
Compare
matthewgrossman
removed this pull request from stack #4776
October 22, 2026 04:51
matthewgrossman
changed the base branch from
test/3954-land-testkit/matthewgrossman
to
test/3954-testkit-ci-lane/matthewgrossman
October 22, 2026 04:51
matthewgrossman
added this pull request to stack #4824
October 22, 2026 04:51
5 tasks done
…tkit-spec `#[testkit::test]` takes two optional arguments. `id = "t-0123abcd"` gives a test a stable identity that survives renames and moves. `requires = [gpu]` names the capabilities the environment must provide, read from OPENSHELL_TEST_CAPABILITIES. A test with requirements fails instead of passing when the environment lacks them; a test without them never reads the variable. The attribute's parsing moves into a new testkit-spec crate, which the test runner's catalog also uses to list tests without compiling them. `#[should_panic]` stays rejected; the requirement tests call `testkit::run` directly. The testkit lockfile is also realigned with the root lockfile. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add a library API that boots an environment, runs commands in it with timeouts, copies files in and out, collects diagnostics and tears it down, so a caller can drive a machine without writing an Ansible playbook per suite. SSH and exec timeouts and QEMU cleanup on SIGTERM keep a hung guest or a cancelled CI job from leaving QEMU running. Add the `test-runner` flake package, which runs tests/runner with the toolchains and VM tools it needs. The Session API and test-runner package are adapted from #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…like the root `cargo --locked` only checks a lockfile against its own manifest. A workspace outside the root that builds root crates through path dependencies, such as the testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two lockfiles can resolve a shared crate to different versions without any check failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link against the AWS-LC that Nix provides. check_cargo_lockfile_alignment.py fails when an aligned workspace locks a registry crate that the root lockfile also contains at a version the root lockfile does not have. With --fix it re-locks the workspace starting from the root lockfile. tests/testkit is the first aligned workspace. The script and its tests come from #4629 (a37ebe9). Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add the `sandbox` crate to the conformance workspace with one testkit test that creates a sandbox, runs `echo` in it with exec, deletes it and checks it is gone. It is deliberately small: it exists so a plan can run a real test through the runner on CI before the existing suites move to testkit. The workspace's other crate, the CLI conformance suite, keeps running as it does today. The conformance workspace joins the lockfile alignment check, so its lockfile is realigned with the root lockfile. The test is adapted from the `sandbox-lifecycle` suite in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
`openshell-test` expands a plan (tests/plans/*.yaml) into one job per environment, builds the family's nextest archive for the guest, boots the environment through the tmachine Session API, runs nextest in it and brings back the JUnit report and logs. No per-suite Ansible playbook is involved. The runner points the tests at the system under test with environment variables: OPENSHELL_GATEWAY (the gateway the installer registered, from the environment's `gateway` field), OPENSHELL_BIN (the installed CLI) and OPENSHELL_TEST_CAPABILITIES (from the environment definition). Nextest runs with `--test-threads 1` and a slow-timeout that terminates a hung test. The JUnit report is checked against the catalog, so a missing or skipped expected test fails the job; `#[ignore]` tests are not expected and nextest's skipped case for them is accepted. `--strict` fails on an unsupported environment, an environment with nothing left to run, or an empty selection. `--emit-github-matrix` prints the GitHub Actions matrix for a plan, so the plan is the only list of what runs where. Environments are defined for amd64 and arm64 Ubuntu with Docker, k3s and Fedora Podman; tests/plans/cpu-tests.yaml selects only the amd64 Ubuntu rootful Docker + Debian package environment, the one CI can provision today. Most of this comes from #4629, adapted to testkit finding the gateway the way the CLI does instead of reading a target file. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Run fmt, clippy and nextest for tests/tmachine and tests/runner, and fmt and clippy for the new `sandbox` suite crate, wherever tests/testkit already runs them: the mise tasks and Branch Checks. The suite's tests need a gateway, so they run through `mise run e2e:testkit` against a local Docker gateway and, on CI, through the test runner rather than the unit-test job. The lockfile task and the Cargo lockfiles job also run the alignment check, and its own tests run with `mise run test`. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add a reusable workflow that takes a plan. A planning job runs `openshell-test test --plan <plan> --emit-github-matrix`, and one job per environment, on the runner label the matrix gives, runs `nix run .#test-runner -- test --plan <plan> --env <id> --strict`. The jobs use the same integration inputs as the existing tmachine lanes, cache the tmachine disks keyed on the Ansible tree, upload the JUnit reports and logs whether or not the run passes, and an aggregator job reports the overall result. Branch E2E calls it for tests/plans/cpu-tests.yaml when the test:e2e label is set. The job is not in the Core E2E result's needs yet, so it starts out advisory. The existing conformance, feature and driver lanes are unchanged. Modeled on the E2E Suites workflow in #4629. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Describe how a plan runs on CI and locally in TESTING.md and a new tests/plans/README.md, add a README for the sandbox suite, point the testkit README at the runner, and add rows for tests/suites, tests/tmachine, tests/runner, tests/plans and tests/environments to the project-structure tables in AGENTS.md and CONTRIBUTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The conformance sandbox suite replaces the standalone tests/suites/sandbox smoke suite from the Docker lane, so there is one smoke test. `mise run e2e:testkit` and the testkit-e2e job now run `cargo test -p sandbox` in tests/suites/conformance. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
matthewgrossman
force-pushed
the
test/3954-first-testkit-lane/matthewgrossman
branch
from
October 22, 2026 05:03
63f4141 to
df44e02
Compare
matthewgrossman
changed the title
matthewgrossman
removed this pull request from stack #4824
October 22, 2026 05:29
matthewgrossman
added this pull request to stack #4825
October 22, 2026 05:30
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the test runner, plans, and the tmachine
SessionAPI, and runs a plan on CI. Acpu-testsplan runs the conformance smoke suite on the Ubuntu, rootful Docker, Debian package environment through a new reusablee2e-suites.ymlworkflow. The plan is the only list of what runs where: nothing in a workflow or playbook repeats it.Most of the code comes from @elezar's draft #4629: the
SessionAPI,testkit-spec, the runner, environments, and waivers, and the lockfile alignment check (a37ebe917). It is adapted to testkit finding the gateway the way the CLI does, so the runner setsOPENSHELL_GATEWAYandOPENSHELL_BINinstead of writing a target file.Stack
Part of a stack of four; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its
tests/tmachine/paths.SessionAPI, with a plan running on CI.Still to come, on top of #4774: moving the release lanes onto plans and deleting the old suites, once the plan has run the ported suites green.
Related Issue
Part of #3954
Changes
tests/runner(openshell-test): plan resolution, environments intests/environments, waivers, JUnit result checking (a missing, skipped, or unselected test fails the lane), strict mode, and--emit-github-matrix. The catalog leaves#[ignore]tests out of what a lane expects.tests/plans/cpu-tests.yaml: the plan.tests/environmentsalso defines the k3s, Podman and arm64 environments, so adding one to a plan is one line once it has a runner path.SessionAPI; SSH, scp, and exec timeouts;down()never panics; QEMU is cleaned up on SIGTERM; the SSH port isTMACHINE_SSH_PORT. Atest-runnerflake package.#[testkit::test(id = "t-…", requires = [...])], atestkit-speccrate shared by the macro and the runner, and capabilities throughOPENSHELL_TEST_CAPABILITIES. Existing tests don't change.tests/suites/conformance/sandbox: the smoke suite. It replaces test(sandbox): run a testkit suite in Branch E2E #4822's standalonetests/suites/sandbox, so there is one smoke test.mise run e2e:testkitand thetestkit-e2ejob run it on a Docker gateway..github/workflows/e2e-suites.yml(reusable) and aplan-cpu-testsjob in Branch E2E (labeltest:e2e): one job per environment from the plan, runningnix run .#test-runner -- test --plan ... --env <id> --strict, with JUnit and logs uploaded. It is not part of the Core E2E result yet.Testing
plan-cpu-testsjob passed:tmachine.ubuntu2404-amd64.docker-rootful.deb / conformance: PASS, 1 passed, withsmoke: create_exec_deleteselected from the plan. Thetestkit-e2ejob also passed.Not done yet
Checklist
🤖 Generated with Claude Code