Skip to content

Collect managed coverage across CI test workers - #2147

Draft
Tyrie Vella (tyrielv) wants to merge 5 commits into
microsoft:masterfrom
tyrielv:tyrielv/aot-coverage-probe
Draft

Tyrie Vella (tyrielv) wants to merge 5 commits into
microsoft:masterfrom
tyrielv:tyrielv/aot-coverage-probe

Conversation

@tyrielv

@tyrielv Tyrie Vella (tyrielv) commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Problem and Context

CI does not publish full managed-source coverage for the deployed NativeAOT product.
Unit coverage can measure linked source inside a test assembly instead of the product executable.

The functional suite already uses twelve workers per architecture.
A separate coverage suite duplicates that work.
This change collects coverage during the existing test runs.

Changes

  • Instrument the non-release CI payloads in the two existing builds. Remove the separate coverage build.
  • Collect coverage during the existing unit runs, 24 functional shards, and six upgrade scenarios. Keep their test selections unchanged.
  • Use the Visual Studio collector with access limited to LocalSystem. Preserve real Windows-service execution.
  • Include a build manifest with the installer drop. The manifest identifies the compiled source, architecture, and collector session.
  • Use the manifest to select the source and session for reused installer artifacts. Keep uninstrumented installer drops on the existing path.
  • Require every expected worker report before the aggregate report claims complete-suite coverage.
  • Reject mixed source revisions, duplicate workers, failed test outcomes, and empty coverage.
  • Map build-machine source paths to the tested checkout. Exclude generated and external documents explicitly.
  • Publish separate reports for unit, functional, and upgrade execution by architecture.
  • Publish changed-line evidence that keeps product execution separate from linked-source unit execution. Show absent coverage entries without inferring coverage.
  • Add unit tests for worker completeness, source identity, source-path mapping, and misleading coverage evidence.

Coverage Scope

The reports measure managed source in the existing CI test suite.
Native C++ remains unmeasured.
The changed-line report identifies unsupported native files and unmeasured C# lines.

This change adds no coverage threshold.
The release pipeline does not enable instrumentation.
Workers that consume instrumented installers start a collector before the product runs.

Build an isolated instrumented installer for x64 shard 0 and collect coverage before Windows services start. Keep ordinary installers and test selections unchanged for the other workers.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv

Copy link
Copy Markdown
Contributor Author

Do not complete - I marked this ready for review so we could verify #2153 is a problem.

@tyrielv
Tyrie Vella (tyrielv) marked this pull request as draft November 4, 2026 15:36
Grant LocalSystem access through the Visual Studio collector. Verify service startup coverage before the selected functional shard runs.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Instrument non-release payloads in the existing builds. Collect unit, functional, and upgrade evidence without a second test suite, and preserve source identity and separate unit-copy attribution.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv Tyrie Vella (tyrielv) changed the title Probe NativeAOT coverage on one functional shard Collect managed coverage across CI test workers Nov 5, 2026
Strip argument quotes before the batch script compares or assigns configuration and architecture. Preserve the existing defaults and test command.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Resolve local relative paths without weakening source-path checks. Accept empty optional scenario fields and valid NUnit skipped roots. Compare PR coverage against the tested merge base.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants