Skip to content

ci: reuse test and benchmark builds across runners - #558

Draft
shuvroroy wants to merge 1 commit into
PHPantom-dev:mainfrom
shuvroroy:ci/optimize-workflow-runtime
Draft

shuvroroy wants to merge 1 commit into
PHPantom-dev:mainfrom
shuvroroy:ci/optimize-workflow-runtime

Conversation

@shuvroroy

@shuvroroy shuvroroy commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

CI previously took 23m 44s, with repeated Rust compilation and benchmark work on the critical path. This PR shares compiled artifacts, parallelizes independent builds, and reuses exact matching builds. The final upstream PR run passed in 10m 08s; a controlled same-commit warm run passed in 5m 14s.

Every existing check still runs on every PR. The 5–6 minute target is demonstrated for a warm rerun; a new commit still exceeds it.

Before and after

Area Before After
Linux tests and coverage All six shards compiled the complete instrumented test suite independently, then ran their partition. One job builds a Nextest archive; six workers execute their partitions from it without compiling. All six LCOV reports are uploaded together after successful completion.
Windows tests One runner compiled and executed the entire suite. The first optimization still took 11m 38s, including 5m 17s compiling and 5m 31s executing tests. One job builds an archive; four workers execute the complete suite in partitions. An exact-commit cache avoids rebuilding on a matching rerun. The aggregate verifies all four success markers belong to the current revision.
Benchmark compilation The original workflow built base and head sequentially in one job. Dependency caching still left project compilation to repeat. Independent base and head jobs run in parallel. A shared action builds the release benchmarks and server together, or restores exact matching artifacts.
Benchmark execution Four measurement workers followed the combined build. The first optimization used sixteen workers and a separate preparation job, which queued for 4m 17s in the measured run. Twelve workers by default, with manifest pairing performed inside each worker. There is no separate preparation job. Manual runs can select eight, twelve, or sixteen workers.
Scaling and memory Each job built its own release server. Both reuse the head release binary and start as soon as that build completes, independently of the baseline build.
Semantic-export tests Running ten integration cases required compiling the large integration test executable. Those same cases have a dedicated feature-gated target. The check still runs all ten integration cases and the existing unit case. Both all-targets Clippy configurations remain.
Runner setup Floating Rust/tool versions and runner families; each coverage shard deleted preinstalled software before building. Rust 1.99.1, Nextest 0.9.151, llvm-cov 0.9.1, Ubuntu 24.04, and Windows 2025 runner families. Smaller test binaries remove the disk-cleanup step, which previously cost up to 128 seconds per shard despite 88 GiB being free.
Superseded runs and diagnostics Earlier PR revisions continued consuming runners; build and queue costs were harder to distinguish. New PR revisions cancel obsolete runs. Cargo timing artifacts and the comparison summary expose compilation, queue time, job execution time, and exact artifact cache hits. Main-branch publishing retains its serialization lock.

Build reuse and cache boundaries

Benchmark artifact keys include the exact source revision, build recipe, compiler, runner image, native toolchain, and build flags. They store benchmark executables, their manifest, the release server, and build timings. A cache miss builds the requested revision; there are no approximate artifact restore keys. Main-branch builds publish entries that later PRs can reuse for a matching base, while PR-created entries remain in their GitHub cache scope.

The Windows archive key includes the exact commit, compiler, runner image, target, and profile settings. Cache hits reuse compilation only: every test partition and every benchmark measurement still executes. Linux coverage continues to build its instrumented archive on each run. Dependency caches remain separate from these exact artifact caches, and Composer downloads are cached for memory measurements.

PR benchmarks still use the actual first parent of GitHub's test merge as the baseline. Each worker measures base and head sequentially on the same VM, using each revision's own fixtures. Criterion sampling, case accounting, and the 30% latency and memory regression thresholds are unchanged. The existing non-blocking Windows policy is also preserved.

Memory measurement correction

The previous fixed delay could sample RSS while background diagnostics still held temporary parse data. The same code produced a 161.2 MiB WordPress sample in one run and 73.1 MiB in another. Measurements now wait for both native diagnostic publications for the opened file before the existing settling period and RSS sampling. This changes synchronization in the benchmark helper; it leaves server diagnostics and regression thresholds intact. Tests cover waiting for the slow pass, ignoring unrelated files, and enforcing the timeout.

Measured results

Workflow run Total duration Result
Original successful PR workflow 23m 44s Passed
First optimization 16m 57s Passed
Final new-commit upstream PR run 10m 08s Passed
Final new-commit controlled run in the fork 8m 27s Passed
Same-commit warm controlled run in the fork 5m 14s Passed

The final three runs use commit 6430430913e062abef5bb9f3555cc66bfe8f907d. Their cache state and queue conditions differ, so these are observed outcomes rather than a guarantee for every PR.

In the warm run, each benchmark build job took 41s and the Windows archive job took 33s, with Windows compilation skipped on the exact cache hit. The complete Windows path, measured from workflow creation through its aggregate check, fell from 11m 41s in the first optimization to 7m 51s in the final upstream run and 3m 01s in the warm run. The warm run still rebuilt Linux coverage (3m 11s) and ran lint (4m 56s).

New source revisions must compile changed project code, and cache misses, runner image updates, and queue contention can increase duration. The manual fork runs execute the complete test and benchmark workload and generate coverage; only the external Codecov upload is skipped when that repository has no token. Normal PR and push uploads retain their existing behavior. All checks on the final upstream PR run are green.

Validation

The successful upstream workflow covers formatting, default and semantic-export Clippy, Linux coverage, Windows tests, semantic-export tests, Python helpers, WebAssembly Clippy/build/smoke test, PHP lint and existing runtime assertions, scaling, and latency and memory comparisons.

Additional validation completed during development:

  • Workflow validation with actionlint and composite action YAML parsing.
  • All 41 benchmark-helper tests and all 11 semantic-export tests.
  • Required cargo +1.99.1 clippy --fix --allow-dirty -- -D warnings, followed by cargo +1.99.1 fmt; default and semantic-export all-targets Clippy also passed without warnings.
  • A full instrumented suite archive was built locally; an archived shard passed 3,423 tests, emitted source coverage, and performed no compilation during execution.
  • A Windows harness comparison executed and passed the identical 20,096 tests under both harnesses; four additional tests remained ignored. Standard execution took 236.52s and Nextest took 378.367s on the same runner. Four Nextest partitions distribute execution while retaining process isolation.
  • Release artifact validation verified nine benchmark executables and a runnable archived server; git diff --check passed.

Checklist

  • I have updated CHANGELOG.md
  • I have updated the documentation (README.md, docs/, examples/)
  • I have updated the config schema (config-schema.json) — not applicable
  • I have added/updated tests to cover my changes
  • If I used AI tools, I have reviewed what they produced and can explain the changes and why they are needed.
  • I have read through any AI-generated text, especially documentation, the changelog, and other user-facing content, and removed or corrected anything inaccurate, unclear, or unnecessary.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@shuvroroy
shuvroroy force-pushed the ci/optimize-workflow-runtime branch from d5faae9 to da7f952 Compare November 8, 2026 19:04
@shuvroroy shuvroroy changed the title ci: parallelize builds and reuse performance artifacts ci: reuse test and benchmark builds across runners Nov 8, 2026
@shuvroroy
shuvroroy force-pushed the ci/optimize-workflow-runtime branch 2 times, most recently from d80559a to 48e9c4d Compare November 8, 2026 20:26
Build the Linux coverage suite and Windows suite once, then distribute
Nextest archives across six Linux and four Windows runners. Cache Windows
archives for the exact revision, compiler, image, and profile to avoid
recompilation on reruns while still executing every partition. Preserve
the existing aggregate checks and Windows policy.

Cache release benchmark artifacts for exact source and build inputs,
reuse matching baselines, and share the head server with scaling and
memory checks. Pair manifests in measurement workers rather than queueing
a preparation job. Keep normal Criterion sampling and 30% thresholds.

Pin Rust, test tools, and runner families; populate scoped PR caches; and
report cache hits, compiler timings, and runner queue time. Add manual
8/12/16-worker timing runs. Wait for native diagnostic passes before
memory sampling so temporary parses do not enter the result. Give the
optional semantic-export integration tests their own target so lint does
not compile the full integration suite a second time for ten tests.

Validated with actionlint, 41 benchmark-helper tests, default and feature
Clippy checks, formatting, all 11 semantic-export tests, and archived
coverage execution. Both Windows harnesses passed the same 20,096 cases.
The archive-based revision passed the full PR workflow in 9m 29s, with
Windows completing in 7m 22s before adding its rerun cache and isolating
the feature tests. Further cold and warm timings are recorded in the PR.
@shuvroroy
shuvroroy force-pushed the ci/optimize-workflow-runtime branch from 48e9c4d to 6430430 Compare November 8, 2026 20:30
@AJenbo AJenbo added this to the Sprint 9 milestone Nov 8, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants