AccessViolationException on Linux: instrumentation probe writes to a coverage buffer whose mapping is invalidated mid-method #238

Summary

On Linux, test hosts intermittently die with AccessViolationException. A crash dump shows the faulting instruction is an injected code-coverage probe, not user code. Three probes in the same straight-line method target the same 4 KB page of the probe buffer; the first two succeed and the third faults, so the mapping is invalidated while instrumented code is executing.

The managed GC heap is completely intact (verifyheap: 243,846 objects, 0 errors), which rules out heap corruption in the test's own code.

Environment

Failing CI run with full artifacts (crash dump + crash report + sequence log):
https://github.com/dotnet/Nerdbank.GitVersioning/actions/runs/31400455859

Frequency: AVs appeared in 17 of the 18 failed Linux CI runs following the commit noted below, while the 109 failed runs before it contain none. Never reproduced on Windows. Never reproduced locally on a large multi-core dev box despite several dozen full-suite attempts, which is consistent with a timing/memory-pressure-sensitive race.

The faulting instruction

The AV surfaces inside Nerdbank.GitVersioning.SemanticVersion.TryParse. !ip2md shows two native code versions for the method:

Method Name:  Nerdbank.GitVersioning.SemanticVersion.TryParse(System.String, SemanticVersion ByRef)
Version History:
     CodeAddr: 0000000000000000  (QuickJitted + Instrumented)
     CodeAddr: 00007f1a64721060  (QuickJitted)

Disassembling the executing code around the faulting IP 0x7F1A64721115 (extracted straight from the ELF core's PT_LOAD segments and run through objdump -b binary -m i386:x86-64 -M intel):

0x647210a3:  mov  rax, QWORD PTR [rip+0x...]   ; -> global slot 0x7f1a6124b2c8 (probe buffer base)
0x647210aa:  mov  BYTE PTR [rax+0x428], 0x1    ; probe A -- SUCCEEDED
0x647210bf:  call Requires.NotNullOrEmpty
0x647210c5:  mov  rax, QWORD PTR [rip+0x...]   ; re-read same slot
0x647210cc:  mov  BYTE PTR [rax+0x429], 0x1    ; probe B -- SUCCEEDED
0x64721104:  call Regex.Match
0x6472110e:  mov  rax, QWORD PTR [rip+0x...]   ; re-read same slot
0x64721115:  mov  BYTE PTR [rax+0x42a], 0x1    ; probe C -- ACCESS VIOLATION

Key points:

• All three probes write to offsets 0x428, 0x429, 0x42a — the same 4 KB page, in the same invocation, microseconds apart.
• The slot at 0x7f1a6124b2c8 held 0x7f1a60000000. The buffer length stored at slot+8 is 0x1d04 (7428), so offset 0x42a is well in bounds — this is not a buffer overrun.
• Therefore the page went from writable to inaccessible between two instruction groups. Only munmap, mprotect, or truncation of the backing file (SIGBUS) can do that — or the slot being re-published to a base whose mapping is not (yet / no longer) valid.

The buffer is a shared memory mapping

Walking the core for references to 0x7f1a60000000 finds a managed object at 0x7f126f8b85b8 whose MethodTable resolves to:

Name:        Microsoft.Win32.SafeHandles.SafeMemoryMappedViewHandle
MethodTable: 00007f1a6483bdf8

Strings in the dump confirm the backing store:

/tmp/CodeCoverage.<sessionGuid>.<moduleGuid>
ShmOpen / ShmUnlink
CreateSharedBackingObjectUsingMemoryShmOpen

Loaded modules include libInstrumentationEngine.so, libCoverageInstrumentationMethod.so, and Microsoft.CodeCoverage.{Core,Instrumentation,Instrumentation.Core,Interprocess}.dll.

Notably, exactly two SafeMemoryMappedViewHandle instances exist in the heap, and both have state = 0x4 (refcount 1, not closed, not disposed). So the managed SafeHandle did not release the mapping — whatever invalidated it did so from outside that object's lifetime tracking.

strace evidence: the same buffer is mapped ~174 times and never released

Tracing a full test run (strace -f -e trace=mmap,munmap,ftruncate,openat) on a dev box shows:

• The buffer files are created once and sized with ftruncate:

  ftruncate(158</tmp/CodeCoverage.<sessionA>.<moduleX>>, 0)
  ftruncate(158</tmp/CodeCoverage.<sessionA>.<moduleX>>, 7428)
  

  7428 = 0x1d04, matching the buffer length recorded in the crashing process.

• Across the run there are 356 mmap calls for these buffer files, in this repeating pair:

  openat(..., "/tmp/CodeCoverage.<session>.<module>", O_RDWR) = 262
  mmap(NULL, 1,    PROT_READ|PROT_WRITE, MAP_SHARED, 262, 0) = 0x...
  mmap(NULL, 7428, PROT_READ|PROT_WRITE, MAP_SHARED, 262, 0) = 0x7b2989cff000
  

• In a single test-host process, the same module's 7428-byte buffer is mapped 171–174 times, and /proc/<pid>/maps confirms all of them remain resident simultaneously (342 rw-s mappings of just 2 files at the end of the run). Only ~8 munmap calls for these buffers occur in the whole run, all during shutdown in the report-writing processes.

So each re-instrumentation creates a fresh mapping of the same shared file; earlier mappings are never unmapped during the run. That looks like a leak on its own, and it means the module's global probe-base slot is being re-published repeatedly while instrumented code on other threads is actively dereferencing it.

What triggers the repeated re-mapping

The trigger is repeated loading of the same instrumented assembly into new AssemblyLoadContexts.

This test suite has 180 tests that each run MSBuild in-process (new BuildManager() + new ProjectCollection() from Microsoft.Build), and each build loads the product assembly into a fresh MSBuildLoadContext. Correlating each coverage-buffer openat with the most recent preceding .dll openat on the same thread:

  174   Microsoft.Build.Tasks.Core.dll
  162   Validation.dll
    5   Nerdbank.GitVersioning.dll

174 re-mappings vs. 180 in-proc MSBuild builds — essentially one re-mapping per build.

The crash happens on a different thread running an unrelated test, executing the instrumented copy of the same module from the default load context. Its probes read the shared global slot for that module, so it is exposed to the re-publish/teardown churn driven by the MSBuild load contexts.

Timing corroborates this. The AVs in this repo began at the exact commit that removed an explicit GC.Collect() + GC.WaitForPendingFinalizers() from the test base class teardown. That change did not introduce the defect; it moved GC/finalization (and therefore load-context teardown) from deterministic quiet points to arbitrary moments while other tests are running in parallel — which is exactly when the race can be observed.

Minimal characterization

Ingredients that appear necessary:

1. Linux (shm-backed probe buffers via CreateSharedBackingObjectUsingMemoryShmOpen).
2. An assembly instrumented for coverage that is loaded many times into distinct collectible AssemblyLoadContexts (here, MSBuild's MSBuildLoadContext, once per in-proc build).
3. Concurrent execution of instrumented methods from that same module on other threads.
4. Memory pressure / GC activity sufficient to tear down those load contexts mid-run (readily met on a 4 vCPU hosted runner; hard to hit on a large dev box, which is why this does not reproduce locally).

Things we tried that did not help

• <CollectFromChildProcesses>False</CollectFromChildProcesses> — no change; the mapping count stayed at exactly 342. The churn is not caused by child processes.
• Disposing the BuildManager / ProjectCollection per test (they were genuinely leaked) — correct hygiene, but the re-mapping count was unchanged at 356.
• Upgrading — 18.9.0 is already the latest published version.
• We found no configuration knob to opt out of the shm-backed probe buffers.

Our workaround

We had to disable code coverage entirely on our Linux CI leg. That is a real loss of coverage signal for a cross-platform product, so a fix would be very welcome.

Suggested areas to look at

• Whether the per-module probe-base global is re-published (and any prior view released) without synchronization against threads currently executing instrumented code from that module.
• Whether module unload / AssemblyLoadContext teardown releases a mapping that is still reachable from JITted probe sequences belonging to other live instances of the same module (same MVID, different load context).
• The unbounded accumulation of MAP_SHARED views of the same buffer file (171 per module in one process) looks like a leak worth fixing independently.

Our runsettings

<CodeCoverage>
  <ModulePaths>
    <Include>
      <ModulePath>\.dll$</ModulePath>
      <ModulePath>\.exe$</ModulePath>
    </Include>
    <Exclude>
      <ModulePath>xunit\..*</ModulePath>
    </Exclude>
  </ModulePaths>
  <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
  <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
  <CollectFromChildProcesses>True</CollectFromChildProcesses>
  <CollectAspDotNet>False</CollectAspDotNet>
  <EnableStaticNativeInstrumentation>False</EnableStaticNativeInstrumentation>
  <EnableDynamicNativeInstrumentation>False</EnableDynamicNativeInstrumentation>
  <EnableStaticNativeInstrumentationRestore>True</EnableStaticNativeInstrumentationRestore>
</CodeCoverage>

We still have the 405 MB crash dump and the crash report JSON and are happy to share them or run further experiments if that helps.