[matthewgrossman]

TL;DR

• Adds testkit, a small library for writing end-to-end tests against a running OpenShell gateway through the Rust SDK. A test is a documented async fn marked #[testkit::test] that runs with plain cargo nextest.
• Tests find the gateway the same way the openshell CLI does, so they run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh with no extra config.
• Each test gets unique sandbox names, and the sandboxes it creates are cleaned up whether it passes, fails, or panics.
• cx.cli(...) runs the CLI against the same gateway and workspace as the SDK.
• A test can carry an optional stable id and requires = [...] capabilities, parsed by a small testkit-spec crate that tools such as a test runner can reuse to list tests without compiling them.
• Branch Checks now formats, lints, and unit-tests testkit on all three platforms.
• No suites use it yet and existing e2e lanes are unchanged. This lands the foundation from test(conformance): run conformance suites from test plans in tmachine environments #4629 so the next PR can move suites onto it.

Stack

Part of a stack of three; each PR targets the one below it. The tmachine move (#4726) has merged, so the stack uses its tests/tmachine/ paths.

1. test(testkit): add the testkit library for end-to-end suites #4724 (this PR): the testkit library.
2. feat(runner): add the test runner, plans and tmachine Session API, and run a plan on CI #4773: the test runner, plans, and tmachine Session API, with a plan running a smoke test on CI.
3. test(suites): port the old suites onto testkit and run them from the plan #4774: ports the old suites onto testkit and adds them to the plan.

Still to come, on top of #4774: moving the release lanes onto plans and deleting the old suites, once the plan has run the ported suites green.

Summary

Adds tests/testkit/, a support library for SDK-driven end-to-end suites, as its own Cargo workspace. This is a small first step to get the crate into main and checked in CI. Nothing uses it yet. The existing suites move onto it in one follow-up migration (#4774, stacked on this PR) that also deletes the old suites. Suites added after that arrive one at a time.

The code comes from @elezar's draft #4629 (head 673928804). This PR takes the testkit portion, adapted to current main. It keeps #4629's optional test IDs, capability requirements, and testkit-spec crate, and drops the target file (see "Deferred and follow-ups" below). The test runner in the next PR is their first consumer.

Related Issue

Part of #3954

Changes

• tests/testkit/: the testkit, testkit-macros, and testkit-spec crates, plus the testkit-ready binary, taken from test(conformance): run conformance suites from test plans in tmachine environments #4629.
  • A test is an async fn marked #[testkit::test] with a required doc comment. Other attributes such as #[ignore] apply to the generated test. #[should_panic] is rejected, because a testkit test panics on any failure.
  • Tests find the gateway the same way the openshell CLI does: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint and mTLS material come from its CLI config directory. Like the CLI, an endpoint takes its credentials from OPENSHELL_GATEWAY or the registered gateway with that endpoint. cx.cli runs OPENSHELL_BIN against the same gateway and workspace. Each test logs the gateway it used.
  • Cleanup deletes a test's sandboxes concurrently, each with its own timeout.
  • id = "t-…" and requires = [...] are optional arguments, so a test without them needs nothing. An ID gives a test an identity that survives renames. requires names capabilities (gpu, persistent_workspace, rootless, service_exposure) that the environment lists in OPENSHELL_TEST_CAPABILITIES; a test that requires one fails instead of passing untested when it is missing, and a test without requires never reads the variable. testkit-spec is the single parser the macro and a runner's catalog share.
  • Cargo.lock is re-seeded from the root lockfile, because main's openshell-sdk has more dependencies than when test(conformance): run conformance suites from test plans in tmachine environments #4629 was cut.
• Branch Checks and the rust:lint, rust:format, rust:format:check, and test:rust mise tasks now run fmt, clippy, and nextest for the testkit workspace. This follows test(conformance): run conformance suites from test plans in tmachine environments #4629's cf72f1108, restricted to testkit.
• tests/testkit/README.md covers writing a test, what Context provides, and running against a gateway. TESTING.md, AGENTS.md, and CONTRIBUTING.md point to it.

Deferred and follow-ups

Dropped from #4629's testkit

What is left out:

• The target file (OPENSHELL_TEST_TARGET). It held a copy of the CLI's registered gateway plus capabilities, so testkit now reads the CLI's gateway config directly. As a result, step 3 no longer needs test(conformance): run conformance suites from test plans in tmachine environments #4629's openshell_client role change or the runner code that writes target.yaml. The runner only sets OPENSHELL_GATEWAY (or OPENSHELL_GATEWAY_ENDPOINT) and OPENSHELL_BIN.
• Guidance on what a conformance test may assert. It moves to the conformance suites.

Rest of #4629, for later steps

• The runner, plans, environments, waivers, and matrix.
• The tmachine Session library, and the Ansible and config.nix changes.
• The new suites and e2e-suites.yml.
• The lockfile alignment script (test(conformance): run conformance suites from test plans in tmachine environments #4629's a37ebe917). It matters once testkit is built into the Nix test archives. Until then, check-cargo-lockfiles.sh already validates tests/testkit/Cargo.lock.
• Changes to existing e2e lanes.

Follow-ups found while landing this

• Share gateway resolution with the CLI. testkit's gateway.rs repeats the CLI's lookup: the endpoint-to-registered-gateway match, metadata.json auth modes, and the mtls/ files. It also builds the mTLS channel itself, because the SDK leaves mTLS out. The copies can drift apart. When step 3 lands or tmachine needs the same lookup, move it into openshell-bootstrap or an SDK helper so the CLI, testkit, and tmachine share one implementation.
• Other auth modes. testkit supports plaintext and mTLS gateways. A gateway registered with OIDC or edge-token auth is rejected with a clear error. Add support when a lane needs it.
• Small differences from the CLI. Both are documented or harmless for today's wrappers, and both go away if resolution is shared:
  • For an HTTPS URL that no registered gateway uses, testkit connects without a client certificate. The CLI guesses a certificate directory from the hostname instead.
  • testkit doesn't sanitize gateway names when building the mtls/ path, as the CLI does. Gateway names with only letters, digits, ., -, and _ are unaffected.
• Cleaning up more than sandboxes. Cleanup only deletes sandboxes created through cx.sandbox(...). Tests must delete anything else they create, such as providers. This is documented in the README. Extend the cleanup registry when suites create other resources often.
• CI build cache. testkit is its own Cargo workspace, so its build output in tests/testkit/target isn't covered by the Rust cache. This adds roughly 1–2 minutes to each Rust lint and test job. Caching it would add about 2 GB per platform cache. Revisit if suites make the build slower.
• Local runs on Docker Desktop. Sandboxes can't reach a gateway on the Mac's loopback unless Docker Desktop's host networking is on, or the gateway uses supervisor_network_mode = "bridge". e2e/with-docker-gateway.sh sets neither. Linux CI isn't affected. Consider setting bridge mode in the wrapper on macOS when the first testkit lane lands.

Raised in review and left as is

• CLI error messages include the full command line, so a credential passed as an argument would appear in test output. Arguments are already visible in process listings, and suites use throwaway credentials. Hiding values would mean guessing which arguments are secret.
• Some Sandbox helpers have no caller yet (for example config, draft_policy, and set_setting). They came from test(conformance): run conformance suites from test plans in tmachine environments #4629 for the suites in the next PR, so removing them now would only mean adding them back.

Testing

• cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
• cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
• cargo nextest run --locked --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml (23 tests: the spec parser, capability parsing and requirement checks, and the macro's misuse checks. Gateway resolution, the CLI wrapper, and cleanup are exercised against a real gateway below and by the suites that adopt testkit.)
• tasks/scripts/check-cargo-lockfiles.sh covers tests/testkit/Cargo.lock
• mise run license:check
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable). Not applicable: no suite uses testkit yet.

Against a real gateway

To check that testkit works end to end, I built a throwaway suite crate outside the repo. It depends on this branch's testkit and contains the six sandbox_lifecycle tests from #4629, plus a test that creates three sandboxes, checks that cx.cli(["sandbox", "list"]) shows them, and leaves them for cleanup. I ran it unchanged under e2e/with-docker-gateway.sh on macOS with Docker Desktop. The wrapper's own OPENSHELL_GATEWAY and OPENSHELL_BIN were the only configuration:

e2e/with-docker-gateway.sh sh -c 'testkit-ready 120 && cargo nextest run --manifest-path <suite>/Cargo.toml --test-threads 1'

testkit-ready resolved the wrapper's mTLS gateway, reported it healthy, and confirmed the CLI reaches it. All 7 tests passed in about 15s, and openshell sandbox list showed nothing left behind. Two #[ignore] tests, with the attribute above and below #[testkit::test], were skipped. With only OPENSHELL_GATEWAY_ENDPOINT set, as e2e/rust/e2e-vm.sh does, testkit found the registered gateway's mTLS material, and testkit-ready plus two tests passed.

On Docker Desktop, sandboxes need either Docker Desktop's host networking turned on (Settings → Resources → Network) or supervisor_network_mode = "bridge". Otherwise the supervisor cannot reach a gateway on the Mac's loopback. For this run I set bridge mode with a local, uncommitted change to the wrapper's gateway config.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4724.docs.buildwithfern.com/openshell

[drew]

Reviewed commit 40cc70740c0441fd1d862bf570b9c0610a46777a. Two findings:

1. Readiness and cleanup still lack enforced timeouts. wait_ready checks elapsed time only after health returns; cleanup awaits deletion before starting its polling timeout. An unresponsive gateway can therefore hang either operation. Reproduced again: testkit-ready 1 remained stuck after four seconds against an HTTP/2 peer that never answers health. Please wrap the complete operations in tokio::time::timeout. See gateway.rs:181 and context.rs:156.

2. The new catalog scanner silently misses nested tests. The macro works inside mod nested { ... }, and Cargo discovers those tests, but scan examines only top-level functions. I confirmed Cargo discovered three testkit tests while the scanner returned two. A runner using this catalog would miss nested tests and their requirements. Please traverse inline modules and preserve qualified names, or explicitly reject that layout. See testkit-spec/src/lib.rs:201.

The earlier attribute-forwarding issue is fixed: probes confirmed #[ignore] works above and below the macro. IDs and capability requirements are now supported through the shared parser.

All 23 existing tests passed locally. No real-gateway E2E run was performed.

[drew]

Could we add mTLS configuration to the SDK and have testkit use OpenShellClient::connect here?

ClientConfig already accepts a CA certificate. Adding an optional client identity (certificate and private key contents) would let the SDK configure mTLS through its existing transport path. Testkit would still resolve the registered gateway and load its credential files; filesystem paths and gateway discovery can stay outside the SDK.

That would remove the duplicate channel construction and endpoint settings from testkit, keeping TLS behavior, connection timeouts, and keepalive settings consistent across SDK consumers. This seems worth a small prerequisite SDK change, followed by simplifying this PR to use it. from_parts can remain the escape hatch for other transport customization.

The current comment also overstates the limitation: the SDK can use an mTLS channel via from_parts; it is ClientConfig that cannot configure a client identity today.