[matthewgrossman]

Summary

Adds the test runner, plans, and tmachine Session API on top of testkit. This is step 5 of the test harness plan, stacked on the suite migration.

Most of the code comes from @elezar's draft #4629: the Session API, testkit-spec, and the runner, plans, environments, and waivers. It is adapted to testkit finding the gateway the way the CLI does, so the runner sets OPENSHELL_GATEWAY and OPENSHELL_BIN instead of writing a target file.

Stack

Part of a stack of four; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

1. test(testkit): add the testkit library for end-to-end suites #4724: the testkit library.
2. test(sandbox): add the first testkit suite and run it in Branch E2E #4773: the first testkit suite and lane.
3. test(suites): move the conformance, oci-image, provider-refresh and podman suites to testkit #4774: migrates the old suites to testkit and deletes them.
4. feat(runner): add the test runner, plans and tmachine Session API #4775: the test runner, plans, and tmachine Session API.

Related Issue

Part of #3954

Changes

• testkit: optional #[testkit::test(id = "t-…", requires = [...])], a testkit-spec crate that the macro and the runner share, and capabilities that reach a test through OPENSHELL_TEST_CAPABILITIES. Existing tests don't change. The macro rejects #[should_panic] and moves other attributes such as #[ignore] to the generated test, as test(testkit): add the testkit library for end-to-end suites #4724 now does; the requirement tests call testkit::run directly instead of using should_panic.
• tests/runner (openshell-test): plan resolution, environments in tests/environments, waivers, JUnit result checking (a missing, skipped, or unselected test fails the lane), a strict CI mode, and --emit-github-matrix. The catalog leaves #[ignore] tests out of what a lane expects, and the JUnit check accepts nextest's skipped case for a test it never expected.
• tests/plans/testkit.yaml plans the suites in the environments the conformance and feature lanes run today. tests/expected-matrix.json is checked in, so a change to what runs where shows up as a diff. tasks/scripts/check_plan_workflow_matrix.py fails when the release and Branch E2E workflow matrices drift from the plan, or when a testsuite's suite_expected_tests in tests/tmachine/config.nix differs from the number of tests the plan expects.
• tmachine: the Session API; SSH, scp, and exec timeouts; down() never panics; QEMU is cleaned up on SIGTERM; the SSH port is TMACHINE_SSH_PORT.
• fmt, clippy, and nextest for tests/runner and tests/tmachine in Branch Checks and the mise tasks.

persistent_workspace_survives_stop_start doesn't declare requires yet: the tmachine playbooks and mise run e2e:testkit don't pass OPENSHELL_TEST_CAPABILITIES, so a test with requires would fail there. Add it when the runner drives those lanes.

Not in this PR: generating the release workflow matrices from --emit-github-matrix, which needs a reusable workflow like the one in #4629. The drift check covers it meanwhile.

Testing

• Checks appropriate to the affected code and behavior pass
  • fmt, clippy, and nextest pass for the runner (58 tests), testkit (23), and tmachine (8). The plan dry run, the matrix check, the drift check, and the lockfile and alignment checks pass. The sandbox suites still pass against a local Docker gateway (14/14), and the flake outputs evaluate.
  • Not run: nothing here booted a VM, the Nix build of the runner, or Branch Checks in CI.
• Unit tests added/updated
• E2E tests added/updated (n/a)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (TESTING.md, AGENTS.md, CONTRIBUTING.md)

🤖 Generated with Claude Code

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4775.docs.buildwithfern.com/openshell

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.