Testy

A Go test running framework.

Please use the reference badge above to find the full documentation of this package.

We couldn't find a framework that addressed our API acceptance testing desires, so we're making one ourselves. We had some specific design goals in mind when we started this project:

• Maintain the ability to run tests via go test during development of tests, including its ability to run specific tests only.
• Provide the ability to run tests as a service, so that they may be run on a schedule and as part of CI/CD.
• The same tests should be able to be run both ways.
• Tests should be written in a familiar manner.
• Historical test results should be stored somewhere.

As this is intended for external API acceptance tests, it is expected that the test code itself does not reside in the same repository as the code under test. We have the tests for all of our external APIs in a single test repository, even though those tests are testing several different APIs. This makes it easier to run tests over all external APIs at the same time, to ensure a change to an internal service that is used by multiple APIs does not cause any such API to fail.

Examples

Please see the Example directory.

Test-scoped resources

Use testy.Cleanup(t, cleanup) immediately after acquiring a resource in a Test or t.Run callback. Callbacks run last-in, first-out after subtests, even when the test fails fatally or panics. A failing or panicking cleanup does not prevent earlier callbacks from running. Cleanup errors should fail the test; logging and ignoring them reports a misleading pass.

testy.Context(t) returns a context canceled immediately before cleanup starts. Subtests inherit parent cancellation; native runs also honor the Go test deadline. Use a separate, bounded context for cleanup I/O. These helpers are optional capabilities, so existing custom TestingT implementations still compile, but must implement Cleanup(func()) and Context() context.Context to use them. Legacy Before/After hooks are not resource scopes: migrate resource acquisition into a Test/Run callback before using the lifecycle helpers.

Use ordinary named t.Run cases (or TestEach) for repeated contracts. Hosted execution keeps steps serial within each case; Parallel is only effective with RunAsTest. Independent top-level cases can explicitly opt in as described below. Do not make acceptance correctness depend on subtest scheduling, and do not use a parent's defer to release resources needed by native parallel children. Cleanup waits for those children in native runs. These callbacks cannot recover resources after process termination; persistent fixture providers must retain ownership and support external reconciliation.

Skipped prerequisites

testy.Skipf(t, "reason: %s", detail) stops a case whose prerequisites cannot be established, while running defers and registered cleanup. Use only at explicit prerequisite boundaries, not to suppress product assertion failures. The native runner uses Go's skipped status; hosted results retain SkipReason, including when a prior or later failure overrides the skip. testy.Skipped(t) lets cleanup adapters identify an explicit skip; it does not mean that the test has not failed. These optional capabilities do not add methods to TestingT.

Hosted skipped leaves are skipped; nonfailed containers with skipped coverage are incomplete, never passed. SumTestStatsWithSkipped returns total, passed, failed and skipped counts. The existing SumTestStats signature is preserved, but its total includes skipped tests without counting them as passed. Result consumers must handle incomplete coverage explicitly before enabling skips. Any Fail, Errorf, fatal error or panic, including during cleanup, remains a failure. A prerequisite-specific best-effort cleanup policy must report warnings explicitly instead of weakening these generic failure semantics.

Bounded independent cases

Register an independent case with ConcurrentTest(name, tester) instead of Test. This is an author declaration that its fixture IDs, mutations and hooks are safe to overlap with other opted-in cases. Never split dependent steps merely to get concurrency: t.Run children still execute sequentially in hosted mode.

Create one executor per hosted worker and share it in RunOptions{CaseExecutor: executor} across all RunWithContext or RunPackageWithOptions calls. Choose the admission boundary explicitly:

• NewPackageCaseExecutor(4, 2) admits up to four lifecycles across the worker, with at most two independent cases within a package. Ordinary cases and hooks exclude work only within their package, preserving overlap with other packages. Invocations of the same package serialize through AfterPackage, including across runs, unless the package opts in as described below. Packages that conflict on shared resources must not be scheduled together; the executor does not infer resource ownership.
• NewCaseExecutor(2) retains worker-wide exclusivity for ordinary cases and hooks. NewCaseExecutor(1) serializes every shared lifecycle.
• A nil executor retains serial-in-package behavior and existing package overlap, without a shared lifecycle bound or same-package invocation lock.

A case owns admission from BeforeTest through body, children, cleanup and AfterTest. Package teardown waits for all of its admitted work. Hooks themselves are still legacy non-resource scopes; migrate fixture acquisition into cases. With NewPackageCaseExecutor(4, 1), cases stay serial within each package while up to four different packages can progress. The two limits are not multiplied. Native RunAsTest uses Go's -parallel limit for ConcurrentTest; existing manual Parallel semantics are unchanged. Native AfterPackage now runs in enclosing Go test cleanup, after parallel children and their cleanup, without changing test names. A native bootstrap should contain one registered package, as before.

RunPackageWithOptions(ctx, packageName, opts) propagates caller cancellation, then waits for admitted cases, their cleanup and hooks before returning ctx.Err(). It cannot preempt non-cooperative callbacks or clean up after a killed process. Cleanup I/O needs a fresh bounded context. Infra retries can still repeat side effects; orchestration must not claim exactly-once execution.

The executor is not a fleet-wide limit. Deployment must bound simultaneous worker replicas, surge and other test runners before increasing capacity. Keep a serial rollback and drain old workers before enabling overlap. ListCases exposes a copy of canonical case identities/indices and opt-in flags; reports retain this order regardless of completion order. TestResult.QueueDur records delay from case discovery after package setup to admission; Dur covers the admitted case lifecycle, including its test hooks, separately from that queue delay.

A hosted caller may supply RunOptions.CaseOrder[pkg] as an immutable complete permutation of current case names (for example, a duration-based snapshot recorded by an external workflow). Testy does no history I/O and reads no scheduling clock. This changes admission order only: returned cases retain canonical indices/names, children remain ordered, and the shared executor still covers each entire lifecycle. Missing hints use canonical order. Invalid/duplicate/stale hints are ignored as a whole with a package diagnostic, never by dropping or duplicating selected tests.

Overlapping invocations of one package

By default, two runs that dispatch the same package to one worker wait for each other: the second invocation starts only after the first finishes AfterPackage. A package can declare that its invocations may overlap instead:

var _ = testy.ConcurrentPackage()

The declaration takes effect only when both of these hold; otherwise invocations keep serializing and each package result carries an informational concurrent package declaration ignored message:

• the package has no BeforePackage and no AfterPackage, so there is no package-scoped setup or teardown to interleave;
• every registered case is a ConcurrentTest.

Only NewPackageCaseExecutor has a same-package gate to skip. NewCaseExecutor, a nil executor and native RunAsTest behave as before.

ConcurrentTest promises a case is safe beside its siblings. ConcurrentPackage additionally promises each case is safe beside another copy of itself, so its fixtures must be generated per invocation, never fixed IDs, users or suppliers. This is a separate audit, which is why it is an explicit declaration rather than inferred from ConcurrentTest.

Each overlapping invocation gets its own package limit: with NewPackageCaseExecutor(4, 2), two invocations of one package can each run two cases, and all of them still share the worker's four lifecycle permits. Within an invocation, admission, CaseOrder, BeforeTest/AfterTest, cleanup and cancellation rollback behave as for any other package.