Instrumented probe stores through an unguarded raw pointer into a file-backed mapping — any loss of page backing kills the process with an AccessViolation attributed to innocent user code #232

Activity

1. Alexandru Ciobanu (pavkam)

   added 8 commits that reference this issue on Aug 4, 2026

   Identify the writer that shrinks the mapped coverage buffer (#32)

   ...

   4f5f493

   Instrument the Linux coverage lane to identify the AccessViolation so…

   ...

   481a544

   Identify the writer that shrinks the mapped coverage buffer (#32)

   ...

   e0f64bb

   Instrument the Linux coverage lane to identify the AccessViolation so…

   ...

   f33990d

   Identify the writer that shrinks the mapped coverage buffer (#32)

   ...

   a3780e9

   Instrument the Linux coverage lane to identify the AccessViolation so…

   ...

   edfeb0c

   Identify the writer that shrinks the mapped coverage buffer (#32)

   ...

   1581524

   Make the issue-reference ban enforceable, and wire two unrun quality …

   ...

   b7c681b
2. fhnaseer commented on Aug 12, 2026

   

   fhnaseer

   on Aug 12, 2026

   Member

   More actions

   Please use 18.10.0 version and see if it fixes the issue for you or not.
3. pcshrosbree commented on Aug 13, 2026

   

   Peter Shrosbree (pcshrosbree)

   on Aug 13, 2026

   Author

   More actions

   Thanks — I've tested 18.10.0.

   Short answer: no, it does not fix it. The two changes 18.10.0 does make are real and address parts of remedies 1 and 4, but the reported failure mode is unchanged: losing page backing for the mapped buffer still kills the test host with an AccessViolationException attributed to innocent user code. Reproduced 3/3 on net8.0 and 1/1 on net10.0 with 18.10.0, against 0/4 for the same runs without the induced truncation.

   What 18.10.0 changed

   Decompiling lib/netstandard2.0/Microsoft.CodeCoverage.Instrumentation.Core.dll, these are the only two functional differences between 18.9.0 and 18.10.0:

   -Begin = (byte*)_view.SafeMemoryMappedViewHandle.DangerousGetHandle().ToPointer();
   +byte* pointer = null;
   +_view.SafeMemoryMappedViewHandle.AcquirePointer(ref pointer);
   +Begin = pointer;

   -MemoryMappedFiles[i] = MemoryMappedFile.CreateFromFile(text, FileMode.Create, null, BufferSizes[i], MemoryMappedFileAccess.ReadWrite);
   +using (FileStream fileStream = new FileStream(text, FileMode.Create, FileAccess.ReadWrite, FileShare.ReadWrite))
   +{
   +    byte[] array = new byte[65536];
   +    int num = BufferSizes[i];
   +    while (num > 0)
   +    {
   +        int num2 = ((num < array.Length) ? num : array.Length);
   +        fileStream.Write(array, 0, num2);
   +        num -= num2;
   +    }
   +    fileStream.Flush();
   +}
   +MemoryMappedFiles[i] = MemoryMappedFile.CreateFromFile(text, FileMode.Open, null, BufferSizes[i], MemoryMappedFileAccess.ReadWrite);

   Both are genuine improvements and I appreciate them:

   • Remedy 1 is addressed. AcquirePointer takes the ref (DangerousAddRef), so the mapping can no longer be released out from under a live Begin. (FWIW there is no matching ReleasePointer anywhere in the assembly — AcquirePointer appears once, ReleasePointer zero times — so the ref is held permanently. That is the right lifetime for this use, but it may be worth an explicit comment so it isn't "fixed" later as a leak.)
   • Remedy 4 is half-addressed. The buffer is now preallocated with real blocks rather than created at length by FileMode.Create, which should genuinely reduce the filesystem-exhaustion path — an unwritten hole in a sparse file is exactly the case where a dirty page can't be allocated and you get SIGBUS. That is the cause I suspected for our naturally-occurring CI crashes, so this may well reduce their frequency. The other half is missing: fileStream.Flush() is the managed overload and does not fsync (Flush(flushToDisk: true) would), and there is still no SIGBUS handling.

   What did not change

   Remedy 2 — the buffer is still a shared, world-writable, guessable path. Still Path.Combine(Path.GetTempPath(), …). Observed in all 10 runs below, e.g.:

   /tmp/CodeCoverage.14f67fa5-1417-454e-a53a-66464fe0c569.be97176d2-c35e-412a-8bf2-72bb6baeb402   (1633 bytes)
   

   Remedy 3 — the probe store is still unguarded. Capturing the rewritten LibA.dll mid-run under 18.10.0 and disassembling, the injected prologue is byte-for-byte the shape originally reported:

   IL_0000: ldsfld uint8* 'Microsoft.CodeCoverage.Instrumentation.Static.Tracker.StaticManagedTrackerTemplate_61950162-d3c5-49c6-822a-a8d8675bb5dc'::Begin
   IL_0005: ldc.i4 1609
   IL_000a: add
   IL_000b: ldc.i4.1
   IL_000c: stind.i1
   IL_000d: ldarg.2          // user code starts here
   

   (Static rewriting in place is also unchanged — LibA.dll.orig and LibA.pdb.orig appear in the output directory during every run.)

   So AcquirePointer guarantees the mapping object stays alive, but that was never the only way to lose page backing. Anything that unbacks the mapped page still turns the next basic block of any instrumented method into a fatal AV, and it is still reported against whatever user method happened to be executing.

   Results

   Same solution, same harness, only the package version and the induced truncation varying. truncate = truncate -s 0 on the buffer file 6 s into a run while instrumented code is executing continuously; observe = the identical run with no truncation (the specificity control).

   version	TFM	mode	exit	AV?
   18.10.0	net8.0	truncate	7	YES
   18.10.0	net8.0	truncate	7	YES
   18.10.0	net8.0	truncate	7	YES
   18.10.0	net8.0	observe	0	no
   18.10.0	net10.0	truncate	7	YES
   18.10.0	net10.0	observe	0	no
   18.9.0	net8.0	truncate	134	YES
   18.9.0	net8.0	observe	0	no
   18.9.0	net10.0	truncate	134	YES
   18.9.0	net10.0	observe	0	no

   Representative failure under 18.10.0:

   Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
      at BigRepro.Tests.SlowSuite.ChurnsInstrumentedCodeForThirtySeconds()
      at System.RuntimeMethodHandle.InvokeMethod(System.Object, Void**, System.Signature, Boolean)
      at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(System.Object, System.Reflection.BindingFlags)
   

   One incidental behaviour change worth knowing about: the host still FailFasts and still prints Fatal error. System.AccessViolationException, but the exit code the platform reports moved from 134 (SIGABRT) on 18.9.0 to 7 on 18.10.0, consistently. I haven't chased why. If anything that makes it slightly harder to spot in CI, since 134 is recognisably a crash.

   Environment

   • .NET SDK 10.0.301; runtimes Microsoft.NETCore.App 8.0.26 and 10.0.10
   • Ubuntu 26.04, x64, ext4 (not the disk-pressured CI runner — this is induced, not the spontaneous variant)
   • xunit.v3 3.2.2 on Microsoft.Testing.Platform

   Reproducer

   Self-contained — the script below generates the whole solution (ordinary safe C# only; no unsafe, no pointers, no interop), and the second one runs the probe and the attack.

   gen_repro.py — generates the solution

   #!/usr/bin/env python3
   """Generate the minimal reproducer solution used for
   microsoft/codecoverage#232 and coverlet-coverage/coverlet#1990.
   
   Ordinary safe C# only: no unsafe, no pointers, no P/Invoke, no interop.
   Layout:
     BigRepro/
       NuGet.config          -- nuget.org only
       global.json           -- MTP test runner
       Directory.Build.props -- shared settings
       LibA/LibA.csproj      -- net8.0;net10.0 class library
       Tests/Tests.csproj    -- net8.0;net10.0 xunit.v3 MTP test project
   """
   import os, shutil, pathlib, textwrap
   
   ROOT = pathlib.Path("/data/tmp/cov-repro/BigRepro")
   N_WIDGETS = 24
   N_SUITES = 24
   
   if ROOT.exists():
       shutil.rmtree(ROOT)
   (ROOT / "LibA").mkdir(parents=True)
   (ROOT / "Tests").mkdir(parents=True)
   
   # ---------------------------------------------------------------- solution files
   (ROOT / "NuGet.config").write_text("""<?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <packageSources>
       <clear />
       <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
     </packageSources>
   </configuration>
   """)
   
   (ROOT / "global.json").write_text("""{
     "sdk": {
       "version": "10.0.301",
       "rollForward": "latestFeature"
     },
     "test": {
       "runner": "Microsoft.Testing.Platform"
     }
   }
   """)
   
   (ROOT / "Directory.Build.props").write_text("""<Project>
     <PropertyGroup>
       <Nullable>enable</Nullable>
       <LangVersion>latest</LangVersion>
       <ImplicitUsings>enable</ImplicitUsings>
       <AllowUnsafeBlocks>false</AllowUnsafeBlocks>
       <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
       <DebugType>portable</DebugType>
       <ManagePackageVersionsCentrally>false</ManagePackageVersionsCentrally>
     </PropertyGroup>
   </Project>
   """)
   
   # ---------------------------------------------------------------- LibA
   (ROOT / "LibA" / "LibA.csproj").write_text("""<Project Sdk="Microsoft.NET.Sdk">
     <PropertyGroup>
       <TargetFrameworks>net8.0;net10.0</TargetFrameworks>
       <RootNamespace>BigRepro.LibA</RootNamespace>
       <AssemblyName>LibA</AssemblyName>
     </PropertyGroup>
   </Project>
   """)
   
   for i in range(1, N_WIDGETS + 1):
       src = textwrap.dedent(f"""\
           using System.Collections.Concurrent;
   
           namespace BigRepro.LibA;
   
           /// <summary>Ordinary safe type; many small basic blocks so instrumentation
           /// has plenty of probe sites.</summary>
           public sealed class Widget{i}
           {{
               private readonly ConcurrentDictionary<int, long> _cells = new();
               private readonly double _scale;
               private readonly double _bias;
               private readonly long _seed;
               private readonly bool _strict;
   
               public Widget{i}(double scale, double bias, long seed, bool strict)
               {{
                   if (double.IsNaN(scale)) throw new ArgumentException("scale", nameof(scale));
                   if (double.IsNaN(bias)) throw new ArgumentException("bias", nameof(bias));
                   _scale = scale;
                   _bias = bias;
                   _seed = seed;
                   _strict = strict;
               }}
   
               public double Scale => _scale;
               public double Bias => _bias;
               public long Seed => _seed;
               public bool Strict => _strict;
               public int Count => _cells.Count;
   
               public long Touch(int a, int b, int c, long d)
               {{
                   var key = (a * 31 + b) * 31 + c;
                   if (key < 0) key = -key;
                   var v = _cells.AddOrUpdate(key % 64, d, (_, old) => old + d);
                   if (_strict && v < 0) return 0;
                   return v;
               }}
   
               public int Evict(int a, int b, int c)
               {{
                   var removed = 0;
                   for (var k = a; k < a + 8; k++)
                   {{
                       if ((k + b) % 3 == 0) continue;
                       if (_cells.TryRemove(k % 64, out _)) removed++;
                       if (removed > c) break;
                   }}
                   return removed;
               }}
   
               public double Calc(int n)
               {{
                   var acc = _bias;
                   for (var k = 0; k < n; k++)
                   {{
                       acc = k % 2 == 0 ? acc + _scale * k : acc - _scale / (k + 1);
                       if (acc > 1e12) {{ acc = 1e12; break; }}
                       if (acc < -1e12) {{ acc = -1e12; break; }}
                   }}
                   return acc;
               }}
   
               public string Describe(int n) => n switch
               {{
                   < 0 => "negative",
                   0 => "zero",
                   < 10 => "small",
                   < 100 => "medium",
                   _ => "large",
               }};
   
               public bool Validate(int n, string? tag)
               {{
                   if (tag is null) return false;
                   if (tag.Length == 0) return false;
                   if (n < 0) return false;
                   return tag.Length + n < 4096;
               }}
           }}
           """)
       (ROOT / "LibA" / f"Widget{i}.cs").write_text(src)
   
   # ---------------------------------------------------------------- Tests
   (ROOT / "Tests" / "Tests.csproj").write_text("""<Project Sdk="Microsoft.NET.Sdk">
     <PropertyGroup>
       <TargetFrameworks>net8.0;net10.0</TargetFrameworks>
       <RootNamespace>BigRepro.Tests</RootNamespace>
       <AssemblyName>Tests</AssemblyName>
       <OutputType>Exe</OutputType>
       <IsPackable>false</IsPackable>
       <UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
       <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>
     </PropertyGroup>
     <ItemGroup>
       <PackageReference Include="XUNIT_PKG" Version="XUNIT_VERSION" />
       <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" Version="COVERAGE_VERSION" />
     </ItemGroup>
     <ItemGroup>
       <ProjectReference Include="../LibA/LibA.csproj" />
     </ItemGroup>
   </Project>
   """)
   
   for i in range(1, N_SUITES + 1):
       w = (i % N_WIDGETS) + 1
       src = textwrap.dedent(f"""\
           using BigRepro.LibA;
           using Xunit;
   
           namespace BigRepro.Tests;
   
           public class Suite{i}
           {{
               private static Widget{w} Make() => new Widget{w}(1.5, 0.25, {i}L, strict: {'true' if i % 2 == 0 else 'false'});
   
               [Theory]
               [InlineData(1)] [InlineData(2)] [InlineData(3)] [InlineData(4)] [InlineData(5)]
               public void Calc10Works(int n)
               {{
                   var w = Make();
                   var v = w.Calc(n * 10);
                   Assert.False(double.IsNaN(v));
                   Assert.True(w.Touch(n, n + 1, n + 2, n) != long.MinValue);
               }}
   
               [Fact]
               public void RejectsBadConfig()
               {{
                   Assert.Throws<ArgumentException>(() => new Widget{w}(double.NaN, 0, 1, false));
               }}
   
               [Fact]
               public async Task ConcurrentTouchVsEvict()
               {{
                   var w = Make();
                   var t1 = Task.Run(() => {{ for (var k = 0; k < 200; k++) w.Touch(k, k + 1, k + 2, k); }});
                   var t2 = Task.Run(() => {{ for (var k = 0; k < 200; k++) w.Evict(k, k + 1, 3); }});
                   await Task.WhenAll(t1, t2);
                   Assert.True(w.Count >= 0);
               }}
   
               [Theory]
               [InlineData(-1, "negative")] [InlineData(0, "zero")] [InlineData(5, "small")]
               [InlineData(50, "medium")] [InlineData(500, "large")]
               public void DescribeBuckets(int n, string expected)
               {{
                   Assert.Equal(expected, Make().Describe(n));
               }}
   
               [Theory]
               [InlineData(1, "x", true)] [InlineData(-1, "x", false)] [InlineData(1, "", false)]
               public void ValidateWorks(int n, string tag, bool expected)
               {{
                   Assert.Equal(expected, Make().Validate(n, tag));
               }}
           }}
           """)
       (ROOT / "Tests" / f"Suite{i}.cs").write_text(src)
   
   print(f"Generated {ROOT}")
   print(f"  LibA:  {N_WIDGETS} widget classes")
   print(f"  Tests: {N_SUITES} suites (~{N_SUITES * 15} test cases)")
   print("  Tests.csproj carries the literal token COVERAGE_VERSION -- substitute per run.")

   probe_live.py — runs it, observes the buffer, and performs the truncation

   #!/usr/bin/env python3
   """Observe the coverage buffer / rewritten-assembly state WHILE an instrumented
   run is in flight, then attempt the microsoft/codecoverage#232 truncation repro.
   
   Adds a deliberately slow test to the generated solution so there is a window to
   observe in, launches `dotnet test --coverage`, and polls:
     - /tmp/CodeCoverage.*            (the shared world-writable buffer files)
     - the test output dir            (*.orig -> rewrite-in-place evidence)
     - the process maps               (is the buffer mmap'd into the test host?)
   
   Usage: probe_live.py <coverage-version> <tfm> <xunit-pkg> [truncate|observe]
   """
   import os, shutil, subprocess, sys, pathlib, time, threading, glob, textwrap
   
   VERSION = sys.argv[1] if len(sys.argv) > 1 else "18.10.0"
   TFM = sys.argv[2] if len(sys.argv) > 2 else "net8.0"
   XPKG = sys.argv[3] if len(sys.argv) > 3 else "xunit.v3.mtp-v2"
   MODE = sys.argv[4] if len(sys.argv) > 4 else "observe"
   
   ROOT = pathlib.Path("/data/tmp/cov-repro/BigRepro")
   WORK = pathlib.Path(f"/data/tmp/cov-repro/live-{VERSION}-{TFM}-{MODE}")
   LOG = []
   
   
   def say(m):
       print(m, flush=True)
       LOG.append(str(m))
   
   
   def sh(cmd, **kw):
       e = dict(os.environ); e["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1"; e["DOTNET_NOLOGO"] = "1"
       return subprocess.run(cmd, capture_output=True, text=True, env=e, **kw)
   
   
   # ----------------------------------------------------------------- prepare
   if WORK.exists():
       shutil.rmtree(WORK)
   shutil.copytree(ROOT, WORK)
   csproj = WORK / "Tests" / "Tests.csproj"
   csproj.write_text(csproj.read_text()
                     .replace("COVERAGE_VERSION", VERSION)
                     .replace("XUNIT_PKG", XPKG)
                     .replace("XUNIT_VERSION", "3.2.2"))
   
   # a slow test so instrumented code is live long enough to observe / attack
   (WORK / "Tests" / "SlowSuite.cs").write_text(textwrap.dedent("""\
       using BigRepro.LibA;
       using Xunit;
   
       namespace BigRepro.Tests;
   
       public class SlowSuite
       {
           // Keeps instrumented user code executing continuously for ~30s so the
           // coverage buffer is mapped and being written the whole time.
           [Fact]
           public void ChurnsInstrumentedCodeForThirtySeconds()
           {
               var deadline = DateTime.UtcNow.AddSeconds(30);
               var w1 = new Widget1(1.5, 0.25, 7L, false);
               var w2 = new Widget2(2.5, 0.5, 9L, true);
               long acc = 0;
               while (DateTime.UtcNow < deadline)
               {
                   for (var k = 0; k < 5000; k++)
                   {
                       acc += w1.Touch(k, k + 1, k + 2, k);
                       acc += w2.Evict(k, k + 1, 3);
                       acc += (long)w1.Calc(8);
                       acc += w2.Describe(k % 200).Length;
                       acc += w1.Validate(k, "t") ? 1 : 0;
                   }
               }
               Assert.True(acc != long.MinValue);
           }
       }
       """))
   
   say(f"=== live probe: CodeCoverage {VERSION}, {TFM}, {XPKG}, mode={MODE} ===")
   r = sh(["dotnet", "build", "Tests/Tests.csproj", "-c", "Release", "-f", TFM], cwd=WORK, timeout=900)
   say(f"[build] rc={r.returncode}")
   if r.returncode != 0:
       say(r.stdout[-4000:]); say(r.stderr[-3000:]); sys.exit(1)
   
   BIN = WORK / "Tests" / "bin" / "Release" / TFM
   pre_tmp = set(glob.glob("/tmp/CodeCoverage.*"))
   
   # ----------------------------------------------------------------- launch
   proc = subprocess.Popen(
       ["dotnet", "test", "--project", "Tests/Tests.csproj", "-c", "Release", "-f", TFM,
        "--coverage", "--coverage-output-format", "cobertura"],
       cwd=WORK, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
       env={**os.environ, "DOTNET_CLI_TELEMETRY_OPTOUT": "1", "DOTNET_NOLOGO": "1"})
   
   observations = {"tmp_bufs": {}, "orig_files": set(), "maps": [], "host_pids": set(), "cmdlines": {}}
   stop = threading.Event()
   attacked = {"done": False, "path": None, "size": None}
   
   
   def poller():
       t0 = time.time()
       while not stop.is_set() and time.time() - t0 < 120:
           # 1. buffer files
           for f in glob.glob("/tmp/CodeCoverage.*"):
               if f in pre_tmp:
                   continue
               try:
                   observations["tmp_bufs"].setdefault(f, []).append(os.path.getsize(f))
               except OSError:
                   pass
           # 2. rewrite-in-place evidence
           for f in BIN.glob("*.orig"):
               observations["orig_files"].add(f.name)
           # 3. is a buffer mapped into a live process? (scan every readable /proc/<pid>/maps)
           if not observations["maps"]:
               for pid in os.listdir("/proc"):
                   if not pid.isdigit():
                       continue
                   try:
                       maps = open(f"/proc/{pid}/maps").read()
                   except (OSError, PermissionError):
                       continue
                   if "CodeCoverage" not in maps:
                       continue
                   observations["host_pids"].add(pid)
                   try:
                       cl = open(f"/proc/{pid}/cmdline", "rb").read().decode(errors="replace").replace("\x00", " ")
                   except OSError:
                       cl = "?"
                   observations["cmdlines"][pid] = cl.strip()[:200]
                   for line in maps.splitlines():
                       if "CodeCoverage" in line:
                           observations["maps"].append(f"pid {pid}: {line}")
           # 4. attack, once, after the buffer is live and being written
           if MODE == "truncate" and not attacked["done"]:
               live = [f for f, sizes in observations["tmp_bufs"].items() if sizes and sizes[-1] > 0]
               if live and observations["maps"] and time.time() - t0 > 6:
                   target = sorted(live, key=lambda p: os.path.getsize(p))[-1]
                   attacked["path"] = target
                   try:
                       attacked["size"] = os.path.getsize(target)
                       os.truncate(target, 0)
                       attacked["done"] = True
                   except OSError as e:
                       attacked["done"] = True
                       attacked["size"] = f"truncate failed: {e}"
           time.sleep(0.02)
   
   
   th = threading.Thread(target=poller, daemon=True)
   th.start()
   try:
       out, _ = proc.communicate(timeout=300)
   except subprocess.TimeoutExpired:
       proc.kill(); out, _ = proc.communicate()
   stop.set(); th.join(timeout=5)
   
   # ----------------------------------------------------------------- report
   say(f"\n[dotnet test] rc={proc.returncode}")
   for line in out.strip().splitlines()[-30:]:
       say("  " + line)
   
   say(f"\n[buffer files seen in /tmp during the run] {len(observations['tmp_bufs'])}")
   for f, sizes in sorted(observations["tmp_bufs"].items())[:12]:
       uniq = sorted(set(sizes))
       say(f"  {f}  sizes seen: {uniq[:8]}{' ...' if len(uniq) > 8 else ''}  samples={len(sizes)}")
   
   say(f"\n[mmap of a buffer into the test host] {len(observations['maps'])} lines")
   for m in observations["maps"][:10]:
       say("  " + m)
   
   say(f"\n[*.orig in the output dir during the run] {sorted(observations['orig_files'])}")
   say(f"[pids with a CodeCoverage mapping] {sorted(observations['host_pids'])}")
   for pid, cl in observations["cmdlines"].items():
       say(f"  pid {pid}: {cl}")
   
   if MODE == "truncate":
       say(f"\n[attack] truncated={attacked['done']} path={attacked['path']} size_before={attacked['size']}")
   
   say(f"\n[AccessViolation in output?] {'YES' if 'AccessViolation' in out else 'no'}")
   say(f"[exit 134 / SIGABRT?] {'YES' if proc.returncode in (134, -6) else 'no (rc=%s)' % proc.returncode}")
   
   pathlib.Path(f"/data/tmp/cov-repro/log-live-{VERSION}-{TFM}-{MODE}.txt").write_text("\n".join(LOG) + "\n\n--- full output ---\n" + out)
   say(f"\n[log] /data/tmp/cov-repro/log-live-{VERSION}-{TFM}-{MODE}.txt")

   python3 gen_repro.py
   python3 probe_live.py 18.10.0 net8.0 xunit.v3.mtp-v2 truncate    # faults
   python3 probe_live.py 18.10.0 net8.0 xunit.v3.mtp-v2 observe     # control, clean

   Note for anyone reproducing: with 18.9.0/18.10.0 the plain xunit.v3 3.2.2 package no longer works — the run dies at startup with TypeLoadException: Could not load type 'Microsoft.Testing.Platform.Extensions.TestHost.IDataConsumer' from assembly 'Microsoft.Testing.Platform, Version=2.3.3.0', because xunit.v3 pulls xunit.v3.mtp-v1 while the coverage package requires Microsoft.Testing.Platform 2.3.x. Referencing xunit.v3.mtp-v2 instead fixes it. That is a separate packaging matter, not part of this issue, but it will block a naive repro attempt.

   What would actually close this

   Remedy 3 is the one that matters, and it is independent of how the buffer is created or referenced: a coverage probe failing should never terminate the process, and should never be attributed to user code. Even a one-time validity check with a graceful disable, or a marker frame that makes the probe's provenance visible in the stack, would have saved us the investigation that produced this report. Remedy 2 (a private or anonymous mapping — memfd_create, or at minimum a private directory rather than a path any process on the machine can truncate) removes the most easily triggered cause.

   Happy to test any prerelease build against the same harness.
4. Mateusz Różański (matthewrosse)

   added 2 commits that reference this issue on Aug 29, 2026

   Retry the test run when the coverage probe crashes

   ...

   e4fb929

   A user who exists, and cannot be assigned (#125) (#135)

   ...

   4098d28
5. fhnaseer commented on Sep 2, 2026

   

   fhnaseer

   on Sep 2, 2026

   Member

   More actions

   Please use 18.11.0 version and see if it fixes the issue for you or not.
6. pcshrosbree commented on Sep 3, 2026

   

   Peter Shrosbree (pcshrosbree)

   on Sep 3, 2026

   Last edited by pcshrosbree

   Author

   More actions

   Thanks — I've tested 18.11.0 against the same harness.

   Short answer: the case I originally reported no longer kills the host, but the new guard is a check-then-write race, and a tight loop of truncations still takes the process down — 3/3 — now from inside the coverage code rather than user code. Two further things surfaced that you may want to know about: a truncated buffer now produces a green run with an empty report, and 18.11.0 leaves about 52 six-byte files per test run in /tmp that are not removed when the run ends.

   Edited 2026-09-03 — three corrections, marked inline below. (1) The collector's read at host exit is guarded by HasIntactBacking (it was in 18.10.0 too); I wrongly called it unguarded. It shares the same check-then-read race as the flush. (2) The net10.0 hammer run faulted in both processes: the host flush thread first, then the collector. (3) Environment: /tmp on this host is tmpfs, so the buffer was fully page-backed during these runs; the sparse-file case was not exercised here.

   Results

   Same solution, same probe, one added mode. truncate = a single truncate -s 0 on the buffer 6 s into a run; hammer = from 6 s, loop truncate(0) / truncate(size) as fast as possible for 15 s; observe = no attack (control). The 18.10.0 row is the positive control for the harness.

   version	TFM	mode	exit	AV?	faulting frame	cobertura
   18.11.0	net8.0	truncate ×3	0	no	—	178-byte empty report, line-rate="1"
   18.11.0	net8.0	hammer ×2	7	YES	StaticManagedTrackerTemplate.Flush() ← PeriodicFlushLoop()	empty report
   18.11.0	net8.0	observe	0	no	—	line-rate="0.909", 341 KB
   18.11.0	net10.0	truncate	0	no	—	empty report
   18.11.0	net10.0	hammer	134	YES	host: StaticManagedTrackerTemplate.Flush() ← PeriodicFlushLoop(); then collector: SharedMemoryStream.GetCoverageBuffer ← LoggerBase.Shutdown ← TestingPlatformCoverageProcessMonitor.OnTestHostProcessExitedAsync (corrected: both, not collector only)	none produced
   18.11.0	net10.0	observe	0	no	—	line-rate="0.909"
   18.10.0	net8.0	truncate	7	YES	user test method (as before)	empty report

   Both hammer runs died within about 1–2 s of the loop starting (40k–91k truncate cycles logged before exit).

   What 18.11.0 changed (decompiled Microsoft.CodeCoverage.Instrumentation.Core, 18.10.0 → 18.11.0)

   On non-Windows, StaticManagedTrackerTemplate no longer points Begin at the mapping. Begin is now a GC-pinned private byte[] (_cachedHits), and a background thread (AboveNormal, every 10 ms) copies set bytes into the MemoryMappedViewAccessor — the same copy runs on ProcessExit, and SharedBufferReconciler in the extension host repeats it. Each copy is gated on HasIntactBacking(), which is new FileInfo(BufferName).Length >= BufferSize:

   if (view == null || cachedHits == null || !HasIntactBacking(cachedHits.Length)) return;
   for (int i = 0; i < cachedHits.Length; i++)
       if (cachedHits[i] != 0) { view.Write((long)i, (byte)1); cachedHits[i] = 0; }

   That is exactly what the results show:

   • The injected probe is now fail-safe. The IL is byte-for-byte unchanged (ldsfld Begin / ldc.i4 / add / ldc.i4.1 / stind.i1), but it targets private memory, so a probe can no longer fault, whatever happens to the file. This is the substance of remedy 3 and it is what makes the single-truncation rows pass. Thank you.
   • Attribution is fixed. When something does fault it is now the flush thread or the collector, not whichever user method was executing.
   • The guard is a TOCTOU. HasIntactBacking stats the file, then the loop writes through the mapping; nothing stops the file shrinking in between. The hammer arm just makes the window easy to hit — a single truncation lands in it with probability roughly (flush duration ÷ 10 ms), which is why the truncate rows are green and the hammer rows are not. The same shape applies to the ProcessExit copy and to SharedBufferReconciler.
   • The collector's read has the same race. (Corrected: I originally wrote that this read has no guard. It does — SharedMemoryStream.GetCoverageBuffer checks HasIntactBacking and returns a cleared array before CreateViewAccessor/ReadArray, and that check already existed in 18.10.0.) On net10.0, after the host's flush thread had faulted, the collector also died: LoggerBase.Shutdown passed the length check, then UnmanagedMemoryAccessor.ReadArray hit the truncated page. That path is taken exactly when the host has already gone wrong, so it compounds the failure and — because it is the collector — no report is written at all.

   The other change in this release — DataStoreTrackerTemplate.Initialize gaining a 0700 session directory, 0600 files and Flush(flushToDisk: true) — does not apply to the probe path this issue is about (corrected: the static instrumenter does clone that template, for a module's dependency data store when one is present; the probes never store through it, they store through StaticManagedTrackerTemplate::Begin). The buffer the static tracker maps is created by the collector, in Microsoft.CodeCoverage.Interprocess's MemoryMappedFileFactory.CreateBackingFile (FileMode.Create + SetLength, i.e. still sparse), directly under Path.GetTempPath() with the CodeCoverage.<session>.<module> name, and that code is identical between 18.10.0 and 18.11.0. One correction to my previous comment while I am here: that file was already 0600 in 18.10.0 (my umask is 002, so something sets it explicitly), so "any process on the machine" overstated it — it is any process running as the same user, which in a CI job is every process.

   Two things worth a look

   1. Data loss is reported as success. Every truncate row above printed Test run summary: Passed! and produced a 178-byte cobertura file that says line-rate="1" for zero coverable lines. A threshold gate would pass it. Since HasIntactBacking already knows when it has declined to flush, that seems like the place to at least log a warning, and arguably to fail the coverage step.

   2. /tmp litter. Each 18.11.0 run leaves about 52 files named /tmp/CodeCoverage.<session>.<guid>, 6 bytes each, contents 02 00 00 00 00 00, mode 0600, and they are never removed — 674 of them after 13 runs here. 18.10.0 created one buffer file per run and removed it at exit (corrected: I first wrote "left one", which was an in-run count; a measured 18.10.0 run left nothing behind). I have not chased which component writes them.

   What would actually close this

   The probe half is done. What remains is that the host and the collector still both write to, and read from, a file-backed mapping whose backing they do not control, behind a check that cannot be made atomic from managed code. The options I can see, in order of how much they change:

   1. Stop writing through the mapping from the host. The private array already exists; at exit (and on a timer if you want incremental data) send it over the pipe the tracker already has (PipeHelper.ConnectAndSendData), and let the collector own the file. Then nothing in the test host touches file-backed memory and the whole class goes away.
   2. If the mapping stays, back it with memfd_create / an anonymous shared mapping rather than a path under $TMPDIR, so there is no file to truncate or run out of blocks for.
   3. At minimum, surface a skipped flush (or a cleared read in the collector) as a warning rather than a green run. (Corrected: I originally suggested adding the length check to the collector's read; it is already there, and it races the same way.)

   Reproducer

   gen_repro.py is unchanged from my previous comment. The probe needed two changes for 18.11.0: match the buffer's /proc/<pid>/maps entry on /tmp/CodeCoverage rather than on the token CodeCoverage (the coverage DLL paths contain it too, so the old check was matching the wrong mapping), and the hammer mode. Full script below.

   probe_live2.py

   #!/usr/bin/env python3
   """probe_live.py, revised for the 18.11.0 layout of microsoft/codecoverage#232.
   
   18.11.0's DataStoreTrackerTemplate gained a per-session directory
     /tmp/CodeCoverage.<session>/CodeCoverage.<session>.buffer<guid>
   but the static-instrumentation buffer this issue is about is still created by the
   collector directly under /tmp as CodeCoverage.<session>.<module>. This version:
     - globs both /tmp/CodeCoverage.* and /tmp/CodeCoverage.*/* and keeps files only,
       so it works for either layout
     - records the st_mode of the buffer file (and of a session directory if one appears)
     - matches the host's /proc/<pid>/maps on "/tmp/CodeCoverage" rather than on the
       token "CodeCoverage", which the coverage DLL paths also contain
     - adds mode `hammer`: after the buffer is live, loop truncate(0)/truncate(size)
       for HAMMER_SECS to race 18.11.0's HasIntactBacking() length check against the
       background flush thread's MemoryMappedViewAccessor.Write (TOCTOU probe)
     - parses the cobertura line-rate so data loss vs host death is visible
   
   Usage: probe_live2.py <coverage-version> <tfm> <xunit-pkg> [truncate|observe|hammer]
   """
   import os, shutil, subprocess, sys, pathlib, time, threading, glob, textwrap, re, stat
   
   VERSION = sys.argv[1] if len(sys.argv) > 1 else "18.11.0"
   TFM = sys.argv[2] if len(sys.argv) > 2 else "net8.0"
   XPKG = sys.argv[3] if len(sys.argv) > 3 else "xunit.v3.mtp-v2"
   MODE = sys.argv[4] if len(sys.argv) > 4 else "observe"
   HAMMER_SECS = 15.0
   
   ROOT = pathlib.Path("/data/tmp/cov-repro/BigRepro")
   WORK = pathlib.Path(f"/data/tmp/cov-repro/live2-{VERSION}-{TFM}-{MODE}")
   LOG = []
   
   
   def say(m):
       print(m, flush=True)
       LOG.append(str(m))
   
   
   def sh(cmd, **kw):
       e = dict(os.environ); e["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1"; e["DOTNET_NOLOGO"] = "1"
       return subprocess.run(cmd, capture_output=True, text=True, env=e, **kw)
   
   
   def buffer_candidates():
       out = []
       for f in glob.glob("/tmp/CodeCoverage.*") + glob.glob("/tmp/CodeCoverage.*/*"):
           if os.path.isfile(f):
               out.append(f)
       return out
   
   
   # ----------------------------------------------------------------- prepare
   if WORK.exists():
       shutil.rmtree(WORK)
   shutil.copytree(ROOT, WORK)
   csproj = WORK / "Tests" / "Tests.csproj"
   csproj.write_text(csproj.read_text()
                     .replace("COVERAGE_VERSION", VERSION)
                     .replace("XUNIT_PKG", XPKG)
                     .replace("XUNIT_VERSION", "3.2.2"))
   
   (WORK / "Tests" / "SlowSuite.cs").write_text(textwrap.dedent("""\
       using BigRepro.LibA;
       using Xunit;
   
       namespace BigRepro.Tests;
   
       public class SlowSuite
       {
           // Keeps instrumented user code executing continuously for ~30s so the
           // coverage buffer is mapped and being written the whole time.
           [Fact]
           public void ChurnsInstrumentedCodeForThirtySeconds()
           {
               var deadline = DateTime.UtcNow.AddSeconds(30);
               var w1 = new Widget1(1.5, 0.25, 7L, false);
               var w2 = new Widget2(2.5, 0.5, 9L, true);
               long acc = 0;
               while (DateTime.UtcNow < deadline)
               {
                   for (var k = 0; k < 5000; k++)
                   {
                       acc += w1.Touch(k, k + 1, k + 2, k);
                       acc += w2.Evict(k, k + 1, 3);
                       acc += (long)w1.Calc(8);
                       acc += w2.Describe(k % 200).Length;
                       acc += w1.Validate(k, "t") ? 1 : 0;
                   }
               }
               Assert.True(acc != long.MinValue);
           }
       }
       """))
   
   say(f"=== live probe v2: CodeCoverage {VERSION}, {TFM}, {XPKG}, mode={MODE} ===")
   r = sh(["dotnet", "build", "Tests/Tests.csproj", "-c", "Release", "-f", TFM], cwd=WORK, timeout=900)
   say(f"[build] rc={r.returncode}")
   if r.returncode != 0:
       say(r.stdout[-4000:]); say(r.stderr[-3000:]); sys.exit(1)
   
   BIN = WORK / "Tests" / "bin" / "Release" / TFM
   pre_tmp = set(buffer_candidates()) | set(glob.glob("/tmp/CodeCoverage.*"))
   
   # ----------------------------------------------------------------- launch
   proc = subprocess.Popen(
       ["dotnet", "test", "--project", "Tests/Tests.csproj", "-c", "Release", "-f", TFM,
        "--coverage", "--coverage-output-format", "cobertura"],
       cwd=WORK, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
       env={**os.environ, "DOTNET_CLI_TELEMETRY_OPTOUT": "1", "DOTNET_NOLOGO": "1"})
   
   observations = {"tmp_bufs": {}, "orig_files": set(), "maps": [], "host_pids": set(),
                   "cmdlines": {}, "modes": {}, "dirs": set()}
   stop = threading.Event()
   attacked = {"done": False, "path": None, "size": None, "cycles": 0, "errors": 0, "first_at": None}
   
   
   def poller():
       t0 = time.time()
       while not stop.is_set() and time.time() - t0 < 150:
           # 1. buffer files (top level or one directory down), plus their modes
           for f in buffer_candidates():
               if f in pre_tmp:
                   continue
               try:
                   st = os.stat(f)
                   observations["tmp_bufs"].setdefault(f, []).append(st.st_size)
                   observations["modes"].setdefault(f, oct(stat.S_IMODE(st.st_mode)))
                   d = os.path.dirname(f)
                   if d != "/tmp" and d not in observations["dirs"]:
                       observations["dirs"].add(d)
                       observations["modes"][d + "/"] = oct(stat.S_IMODE(os.stat(d).st_mode))
               except OSError:
                   pass
           # 2. rewrite-in-place evidence
           for f in BIN.glob("*.orig"):
               observations["orig_files"].add(f.name)
           # 3. is the BUFFER FILE mapped into a live process? (the coverage DLL paths
           #    also contain "CodeCoverage", so match on the /tmp buffer path, not the token)
           if not observations["maps"]:
               for pid in os.listdir("/proc"):
                   if not pid.isdigit():
                       continue
                   try:
                       maps = open(f"/proc/{pid}/maps").read()
                   except (OSError, PermissionError):
                       continue
                   if "/tmp/CodeCoverage" not in maps:
                       continue
                   observations["host_pids"].add(pid)
                   try:
                       cl = open(f"/proc/{pid}/cmdline", "rb").read().decode(errors="replace").replace("\x00", " ")
                   except OSError:
                       cl = "?"
                   observations["cmdlines"][pid] = cl.strip()[:200]
                   for line in maps.splitlines():
                       if "/tmp/CodeCoverage" in line:
                           observations["maps"].append(f"pid {pid}: {line}")
           # 4. attack
           if MODE in ("truncate", "hammer") and not attacked["done"]:
               live = [f for f, sizes in observations["tmp_bufs"].items() if sizes and sizes[-1] > 0]
               if live and observations["maps"] and time.time() - t0 > 6:
                   target = sorted(live, key=lambda p: os.path.getsize(p))[-1]
                   attacked["path"] = target
                   attacked["first_at"] = round(time.time() - t0, 2)
                   try:
                       size = os.path.getsize(target)
                       attacked["size"] = size
                       if MODE == "truncate":
                           os.truncate(target, 0)
                           attacked["cycles"] = 1
                       else:
                           t1 = time.time()
                           while time.time() - t1 < HAMMER_SECS and proc.poll() is None:
                               try:
                                   os.truncate(target, 0)
                                   os.truncate(target, size)
                                   attacked["cycles"] += 1
                               except OSError:
                                   attacked["errors"] += 1
                                   if attacked["errors"] > 1000:
                                       break
                           # leave it truncated at the end so the exit-time flush sees a short file
                           try:
                               os.truncate(target, 0)
                           except OSError:
                               pass
                       attacked["done"] = True
                   except OSError as e:
                       attacked["done"] = True
                       attacked["size"] = f"truncate failed: {e}"
           time.sleep(0.02)
   
   
   th = threading.Thread(target=poller, daemon=True)
   th.start()
   try:
       out, _ = proc.communicate(timeout=300)
   except subprocess.TimeoutExpired:
       proc.kill(); out, _ = proc.communicate()
   stop.set(); th.join(timeout=5)
   
   # ----------------------------------------------------------------- report
   say(f"\n[dotnet test] rc={proc.returncode}")
   for line in out.strip().splitlines()[-30:]:
       say("  " + line)
   
   say(f"\n[buffer files seen during the run] {len(observations['tmp_bufs'])}")
   for f, sizes in sorted(observations["tmp_bufs"].items())[:12]:
       uniq = sorted(set(sizes))
       say(f"  {f}  mode={observations['modes'].get(f)}  sizes seen: {uniq[:8]}{' ...' if len(uniq) > 8 else ''}  samples={len(sizes)}")
   for d in sorted(observations["dirs"]):
       say(f"  [dir] {d}  mode={observations['modes'].get(d + '/')}")
   
   say(f"\n[mmap of a buffer into the test host] {len(observations['maps'])} lines")
   for m in observations["maps"][:10]:
       say("  " + m)
   
   say(f"\n[*.orig in the output dir during the run] {sorted(observations['orig_files'])}")
   say(f"[pids with a CodeCoverage mapping] {sorted(observations['host_pids'])}")
   for pid, cl in observations["cmdlines"].items():
       say(f"  pid {pid}: {cl}")
   
   if MODE in ("truncate", "hammer"):
       say(f"\n[attack] mode={MODE} truncated={attacked['done']} path={attacked['path']} size_before={attacked['size']} "
           f"first_at={attacked['first_at']}s cycles={attacked['cycles']} errors={attacked['errors']}")
   
   # cobertura: did coverage data survive?
   rates = []
   for f in WORK.rglob("*.cobertura.xml"):
       m = re.search(r'line-rate="([0-9.]+)"', f.read_text(errors="replace")[:2000])
       rates.append((str(f.relative_to(WORK)), m.group(1) if m else "?", f.stat().st_size))
   say(f"\n[cobertura] {rates if rates else 'NONE produced'}")
   
   say(f"\n[AccessViolation in output?] {'YES' if 'AccessViolation' in out else 'no'}")
   fm = re.search(r"Fatal error\. ([^\n]+)", out)
   say(f"[fatal error line] {fm.group(1) if fm else 'none'}")
   say(f"[exit] rc={proc.returncode}")
   
   pathlib.Path(f"/data/tmp/cov-repro/log-live2-{VERSION}-{TFM}-{MODE}.txt").write_text("\n".join(LOG) + "\n\n--- full output ---\n" + out)
   say(f"\n[log] /data/tmp/cov-repro/log-live2-{VERSION}-{TFM}-{MODE}.txt")

   python3 gen_repro.py
   python3 probe_live2.py 18.11.0 net8.0 xunit.v3.mtp-v2 truncate   # green, empty report
   python3 probe_live2.py 18.11.0 net8.0 xunit.v3.mtp-v2 hammer     # AV in PeriodicFlushLoop
   python3 probe_live2.py 18.11.0 net8.0 xunit.v3.mtp-v2 observe    # control
   python3 probe_live2.py 18.10.0 net8.0 xunit.v3.mtp-v2 truncate   # positive control, AV in user code

   Environment: .NET SDK 10.0.301, runtimes 8.0.26 / 10.0.10, Ubuntu 26.04 x64, xunit.v3 3.2.2 (xunit.v3.mtp-v2). (Corrected: /tmp here is tmpfs, not ext4 — the buffer file showed st_blocks=8 for 1633 bytes while live, i.e. fully page-backed. The truncation results stand; the sparse-file/ENOSPC path was not exercised by these runs.)

   Happy to run the same matrix against a prerelease.
7. fhnaseer commented on Sep 11, 2026

   

   fhnaseer

   on Sep 11, 2026

   Member

   More actions

   Please use 18.11.2 version and see if it fixes the issue for you or not.
8. pcshrosbree commented on Sep 11, 2026

   

   Peter Shrosbree (pcshrosbree)

   on Sep 11, 2026

   Author

   More actions

   Thanks — I've tested 18.11.2 against the same harness.

   Short answer: this fixes it. No arm faulted — 0/10 attack runs, against controls in the same matrix that still die. The truncate/extend loop that killed 18.11.0 after 40k–91k cycles now survives about 8 million cycles per run with no fault and no lost host. Removing the mapping on non-Windows entirely, in both processes, closes the class by construction rather than narrowing it, and I appreciate that you went that way rather than tightening the length check.

   Two things are worth your time below: the truncated-buffer case still produces a green run with an empty report, which I can now root-cause to a single byte rather than just report; and the /tmp litter from 18.11.0 is unchanged.

   Results

   Same solution and same probe as before, plus a delete arm. truncate = a single truncate -s 0 on the buffer 6 s into a run; hammer = from 6 s, loop truncate(0) / truncate(size) as fast as possible for 15 s; delete = unlink the buffer; observe = no attack (control). The last two rows are positive controls: the same probe, against the two releases known to fault.

   version	TFM	mode	exit	AV?	faulting frame	cobertura
   18.11.2	net8.0	truncate ×3	0	no	—	178-byte empty report, line-rate="1"
   18.11.2	net8.0	hammer ×2	0	no	—	empty report (8,007,277 and 8,020,930 cycles)
   18.11.2	net8.0	delete ×2	0	no	—	empty report
   18.11.2	net8.0	observe	0	no	—	line-rate="0.909", 341 KB
   18.11.2	net10.0	truncate	0	no	—	empty report
   18.11.2	net10.0	hammer	0	no	—	empty report (7,901,884 cycles)
   18.11.2	net10.0	delete	0	no	—	empty report
   18.11.2	net10.0	observe	0	no	—	line-rate="0.909"
   18.11.0	net8.0	hammer (control)	134	YES	StaticManagedTrackerTemplate.Flush() ← PeriodicFlushLoop()	none produced
   18.10.0	net8.0	truncate (control)	7	YES	user test method	empty report

   A harness note, because the obvious reuse would have given you a false pass

   My previous probe gated its attack on first seeing the buffer file in some process's /proc/<pid>/maps. 18.11.2 maps nothing on Linux, so that gate can never open — every 18.11.2 row would have come back green with the attack never fired. The revised probe triggers on the buffer file existing instead, prints FIRED=True/False on every row, and measures liveness from buffer content (count of non-zero bytes over time) rather than file size, since the collector SetLengths the file before the first hit.

   With that fixed, the mapping count is itself a measurement: 0 mapped lines on all 12 of the 18.11.2 rows, against 1 on both control rows.

   What 18.11.2 changed (decompiled, 18.11.0 → 18.11.2)

   HasIntactBacking is deleted from both assemblies, and every non-Windows path now goes through ordinary FileStream I/O:

   • Host — InitializeWithSharedMemory now branches on OS. On non-Windows it calls the new ValidateFileBackedBuffer(), then disposes the view and the MemoryMappedFile and nulls both, before PinCachedHitsAndSetBegin(). Flush() delegates to the new FlushFileBackedBuffer(hits), which opens the buffer, re-extends with SetLength if it has shrunk, and writes runs of set bytes at their offsets. If the write throws it restores the hits it had already cleared and rethrows, so a failed publish loses nothing — that is a nice touch.
   • Collector — SharedMemoryStream's WriteMessage, both GetCoverageBuffer overloads, IsInitialized and MarkInitialized all take a non-Windows branch through the new ReadFileBuffer / WriteFileBuffer, which handle a short read by clearing the remainder.

   So there is no longer a mapping to lose backing for, in either process. A truncated file is a short read; a filesystem that cannot allocate a block is a caught IOException. Both were SIGBUS before. I also noticed FailFileBackedPublishAfterRangeCount, which suggests the new publish path has fault-injection tests of its own.

   The injected IL is unchanged in shape, as expected — captured from a rewritten LibA.dll under 18.11.2, byte-for-byte what I first reported:

   IL_0000: ldsfld uint8* 'Microsoft.CodeCoverage.Instrumentation.Static.Tracker.StaticManagedTrackerTemplate_5704b294-8990-4110-bc6a-5ff093649812'::Begin
   IL_0005: ldc.i4 1609
   IL_000a: add
   IL_000b: ldc.i4.1
   IL_000c: stind.i1
   IL_000d: ldarg.2          // user code starts here
   

   What changed is only what Begin points at.

   Still open 1: a lost buffer is reported as a successful run, and it is one byte

   Every attacked row above printed Test run summary: Passed! and wrote a 178-byte cobertura file with line-rate="1" over an empty <packages />. That is not caused by the data loss. It is caused by the initialization byte at offset 0, which SharedMemoryStream.IsInitialized() reads and which gates whether a module enters the report at all — if (logStream.IsInitialized()) guards AddPackage, AddStream and PopulateCoverageStatus. Truncation zeroes it, and the host's flush only writes indices where a probe fired, so index 0 is never restored.

   Two runs isolate it, rather than inferring it from a truncation that perturbs everything at once:

   what was done to the buffer	cobertura
   write a single zero byte to offset 0; file length and all 1,356 probe bytes left intact	178 bytes, line-rate="1"
   truncate to 0, re-extend to the original length, then set offset 0 back to 1	337,297 bytes, line-rate="0.027"

   Note which direction is dangerous. When the flag survives, you report a low rate that honestly reflects the destroyed data, and any coverage threshold catches it. When the flag is lost, you report a perfect rate over zero lines, which every threshold passes. So the silent-success path is exactly the one where the module is dropped.

   Since the collector knows it created the buffer and handed it to the host, a module that comes back uninitialized is a detectable anomaly rather than a normal skip. Logging a warning — or failing the coverage step — at that point would close this. It is cheap and it is the last thing standing between this and a coverage run I would trust in CI.

   Still open 2: /tmp litter

   Unchanged from 18.11.0: each run leaves 52 six-byte files on net8.0 and 54 on net10.0, named /tmp/CodeCoverage.<session>.<guid>, contents 02 00 00 00 00 00, mode 0600, never removed. TryDeleteBackingFile does clean up the real coverage buffer — the 1,633-byte one — so it is only these small files that leak. For comparison, 18.10.0 left zero behind in this same matrix.

   One scoping note

   The private-memory change is non-Windows only: on Windows, with ForcePrivateProbeMemory unset, the probe still stores through the raw pointer from AcquirePointer. That mapping is a named section rather than a file, so the attack I have been using does not transfer to it, and nothing in this matrix says whether it is reachable some other way. Flagging it only so the fix is not remembered as broader than it is.

   Reproducer

   gen_repro.py is unchanged from my first comment. The probe needed the trigger change described above; full script below.

   probe_live3.py

   #!/usr/bin/env python3
   """probe_live3.py — revised for the 18.11.2 layout of microsoft/codecoverage#232.
   
   WHY A THIRD REVISION. probe_live2.py gated its attack on seeing the buffer file in
   some process's /proc/<pid>/maps. 18.11.2 stops memory-mapping the buffer on non-Windows
   altogether (both the host tracker and the collector now use plain FileStream I/O), so
   that gate can never open and every 18.11.2 row would come back green with the attack
   never fired — a vacuous pass measuring the probe's own trigger rather than the product.
   
   Changes from probe_live2.py:
     - the attack trigger no longer requires a mapping. It fires once the buffer file
       exists with non-zero length and >=6 s have elapsed. Whether anything maps the file
       is now an OBSERVATION, reported either way, not a precondition.
     - every run reports `fired=True/False` explicitly, so a non-firing attack cannot be
       read as a pass.
     - liveness is measured from buffer CONTENT (count of non-zero bytes over time), which
       positively shows hits landing in the file, rather than from file size (18.11.2's
       collector SetLength()s the file up front, so size is non-zero before any hit).
     - new mode `delete`: unlink the buffer file mid-run. 18.11.2's FlushFileBackedBuffer
       and ValidateFileBackedBuffer both open FileMode.Open with no guard, so a missing
       file is a distinct path from a short one.
     - /tmp litter is counted before the run and after it, to re-measure the 18.11.0
       finding that ~52 six-byte files per run are left behind.
   
   Usage: probe_live3.py <coverage-version> <tfm> <xunit-pkg> [observe|truncate|hammer|delete]
   """
   import glob
   import os
   import pathlib
   import re
   import shutil
   import stat
   import subprocess
   import sys
   import textwrap
   import threading
   import time
   
   VERSION = sys.argv[1] if len(sys.argv) > 1 else "18.11.2"
   TFM = sys.argv[2] if len(sys.argv) > 2 else "net8.0"
   XPKG = sys.argv[3] if len(sys.argv) > 3 else "xunit.v3.mtp-v2"
   MODE = sys.argv[4] if len(sys.argv) > 4 else "observe"
   HAMMER_SECS = 15.0
   
   ROOT = pathlib.Path("/data/tmp/cov-repro/BigRepro")
   WORK = pathlib.Path(f"/data/tmp/cov-repro/live3-{VERSION}-{TFM}-{MODE}")
   LOG = []
   
   
   def say(m):
       print(m, flush=True)
       LOG.append(str(m))
   
   
   def sh(cmd, **kw):
       e = dict(os.environ)
       e["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1"
       e["DOTNET_NOLOGO"] = "1"
       return subprocess.run(cmd, capture_output=True, text=True, env=e, **kw)
   
   
   def buffer_candidates():
       out = []
       for f in glob.glob("/tmp/CodeCoverage.*") + glob.glob("/tmp/CodeCoverage.*/*"):
           if os.path.isfile(f):
               out.append(f)
       return out
   
   
   def litter_inventory():
       """Every /tmp entry the collector might leave behind, with sizes."""
       inv = {}
       for f in glob.glob("/tmp/CodeCoverage.*"):
           try:
               inv[f] = os.path.getsize(f) if os.path.isfile(f) else "dir"
           except OSError:
               pass
       for f in glob.glob("/tmp/CodeCoverage.*/*"):
           try:
               inv[f] = os.path.getsize(f)
           except OSError:
               pass
       return inv
   
   
   def nonzero_bytes(path):
       try:
           with open(path, "rb") as fh:
               return sum(1 for b in fh.read() if b)
       except OSError:
           return -1
   
   
   # ----------------------------------------------------------------- prepare
   if WORK.exists():
       shutil.rmtree(WORK)
   shutil.copytree(ROOT, WORK)
   csproj = WORK / "Tests" / "Tests.csproj"
   csproj.write_text(csproj.read_text()
                     .replace("COVERAGE_VERSION", VERSION)
                     .replace("XUNIT_PKG", XPKG)
                     .replace("XUNIT_VERSION", "3.2.2"))
   
   (WORK / "Tests" / "SlowSuite.cs").write_text(textwrap.dedent("""\
       using BigRepro.LibA;
       using Xunit;
   
       namespace BigRepro.Tests;
   
       public class SlowSuite
       {
           // Keeps instrumented user code executing continuously for ~30s so the
           // coverage buffer is live and being written the whole time.
           [Fact]
           public void ChurnsInstrumentedCodeForThirtySeconds()
           {
               var deadline = DateTime.UtcNow.AddSeconds(30);
               var w1 = new Widget1(1.5, 0.25, 7L, false);
               var w2 = new Widget2(2.5, 0.5, 9L, true);
               long acc = 0;
               while (DateTime.UtcNow < deadline)
               {
                   for (var k = 0; k < 5000; k++)
                   {
                       acc += w1.Touch(k, k + 1, k + 2, k);
                       acc += w2.Evict(k, k + 1, 3);
                       acc += (long)w1.Calc(8);
                       acc += w2.Describe(k % 200).Length;
                       acc += w1.Validate(k, "t") ? 1 : 0;
                   }
               }
               Assert.True(acc != long.MinValue);
           }
       }
       """))
   
   say(f"=== live probe v3: CodeCoverage {VERSION}, {TFM}, {XPKG}, mode={MODE} ===")
   r = sh(["dotnet", "build", "Tests/Tests.csproj", "-c", "Release", "-f", TFM], cwd=WORK, timeout=900)
   say(f"[build] rc={r.returncode}")
   if r.returncode != 0:
       say(r.stdout[-4000:])
       say(r.stderr[-3000:])
       sys.exit(1)
   
   BIN = WORK / "Tests" / "bin" / "Release" / TFM
   litter_before = litter_inventory()
   pre_tmp = set(buffer_candidates()) | set(glob.glob("/tmp/CodeCoverage.*"))
   say(f"[litter before run] {len(litter_before)} /tmp/CodeCoverage* entries")
   
   # ----------------------------------------------------------------- launch
   proc = subprocess.Popen(
       ["dotnet", "test", "--project", "Tests/Tests.csproj", "-c", "Release", "-f", TFM,
        "--coverage", "--coverage-output-format", "cobertura"],
       cwd=WORK, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
       env={**os.environ, "DOTNET_CLI_TELEMETRY_OPTOUT": "1", "DOTNET_NOLOGO": "1"})
   
   observations = {"tmp_bufs": {}, "orig_files": set(), "maps": [], "host_pids": set(),
                   "cmdlines": {}, "modes": {}, "dirs": set(), "hits": {}}
   stop = threading.Event()
   attacked = {"fired": False, "path": None, "size": None, "cycles": 0, "errors": 0,
               "first_at": None, "hits_at_attack": None}
   
   
   def poller():
       t0 = time.time()
       while not stop.is_set() and time.time() - t0 < 150:
           # 1. buffer files (top level or one directory down), plus their modes
           for f in buffer_candidates():
               if f in pre_tmp:
                   continue
               try:
                   st = os.stat(f)
                   observations["tmp_bufs"].setdefault(f, []).append(st.st_size)
                   observations["modes"].setdefault(f, oct(stat.S_IMODE(st.st_mode)))
                   d = os.path.dirname(f)
                   if d != "/tmp" and d not in observations["dirs"]:
                       observations["dirs"].add(d)
                       observations["modes"][d + "/"] = oct(stat.S_IMODE(os.stat(d).st_mode))
               except OSError:
                   pass
           # 2. rewrite-in-place evidence
           for f in BIN.glob("*.orig"):
               observations["orig_files"].add(f.name)
           # 3. does ANYTHING map the buffer file? Observation only in v3 — 18.11.2 is
           #    expected to map nothing on Linux, and that must not gate the attack.
           if not observations["maps"]:
               for pid in os.listdir("/proc"):
                   if not pid.isdigit():
                       continue
                   try:
                       maps = open(f"/proc/{pid}/maps").read()
                   except (OSError, PermissionError):
                       continue
                   if "/tmp/CodeCoverage" not in maps:
                       continue
                   observations["host_pids"].add(pid)
                   try:
                       cl = open(f"/proc/{pid}/cmdline", "rb").read().decode(errors="replace").replace("\x00", " ")
                   except OSError:
                       cl = "?"
                   observations["cmdlines"][pid] = cl.strip()[:200]
                   for line in maps.splitlines():
                       if "/tmp/CodeCoverage" in line:
                           observations["maps"].append(f"pid {pid}: {line}")
           # 4. liveness by CONTENT: are hits landing in the buffer?
           for f, sizes in observations["tmp_bufs"].items():
               if sizes and sizes[-1] > 0:
                   nz = nonzero_bytes(f)
                   if nz > 0:
                       observations["hits"].setdefault(f, []).append((round(time.time() - t0, 2), nz))
           # 5. attack — trigger does NOT depend on a mapping existing
           if MODE in ("truncate", "hammer", "delete") and not attacked["fired"]:
               live = [f for f, sizes in observations["tmp_bufs"].items() if sizes and sizes[-1] > 0]
               if live and time.time() - t0 > 6:
                   target = sorted(live, key=lambda p: os.path.getsize(p))[-1]
                   attacked["path"] = target
                   attacked["first_at"] = round(time.time() - t0, 2)
                   h = observations["hits"].get(target)
                   attacked["hits_at_attack"] = h[-1][1] if h else 0
                   try:
                       size = os.path.getsize(target)
                       attacked["size"] = size
                       if MODE == "truncate":
                           os.truncate(target, 0)
                           attacked["cycles"] = 1
                       elif MODE == "delete":
                           os.unlink(target)
                           attacked["cycles"] = 1
                       else:
                           t1 = time.time()
                           while time.time() - t1 < HAMMER_SECS and proc.poll() is None:
                               try:
                                   os.truncate(target, 0)
                                   os.truncate(target, size)
                                   attacked["cycles"] += 1
                               except OSError:
                                   attacked["errors"] += 1
                                   if attacked["errors"] > 1000:
                                       break
                           # leave it truncated so the exit-time flush sees a short file
                           try:
                               os.truncate(target, 0)
                           except OSError:
                               pass
                       attacked["fired"] = True
                   except OSError as e:
                       attacked["fired"] = False
                       attacked["size"] = f"attack failed: {e}"
           time.sleep(0.02)
   
   
   th = threading.Thread(target=poller, daemon=True)
   th.start()
   try:
       out, _ = proc.communicate(timeout=300)
   except subprocess.TimeoutExpired:
       proc.kill()
       out, _ = proc.communicate()
   stop.set()
   th.join(timeout=5)
   
   # ----------------------------------------------------------------- report
   say(f"\n[dotnet test] rc={proc.returncode}")
   for line in out.strip().splitlines()[-30:]:
       say("  " + line)
   
   say(f"\n[buffer files seen during the run] {len(observations['tmp_bufs'])}")
   for f, sizes in sorted(observations["tmp_bufs"].items())[:12]:
       uniq = sorted(set(sizes))
       say(f"  {f}  mode={observations['modes'].get(f)}  sizes seen: {uniq[:8]}{' ...' if len(uniq) > 8 else ''}  samples={len(sizes)}")
   for d in sorted(observations["dirs"]):
       say(f"  [dir] {d}  mode={observations['modes'].get(d + '/')}")
   
   say(f"\n[hits landing in the buffer (nonzero byte count over time)]")
   for f, series in sorted(observations["hits"].items())[:4]:
       pts = series[::max(1, len(series) // 6)][:7]
       say(f"  {f}: {pts}  (max {max(n for _, n in series)})")
   if not observations["hits"]:
       say("  NONE — no buffer ever carried a nonzero byte")
   
   say(f"\n[mmap of a buffer into any process] {len(observations['maps'])} lines"
       f"  <- 18.11.2 on Linux is expected to map NOTHING; v2's attack gate required this")
   for m in observations["maps"][:10]:
       say("  " + m)
   
   say(f"\n[*.orig in the output dir during the run] {sorted(observations['orig_files'])}")
   say(f"[pids with a CodeCoverage mapping] {sorted(observations['host_pids'])}")
   for pid, cl in observations["cmdlines"].items():
       say(f"  pid {pid}: {cl}")
   
   if MODE != "observe":
       say(f"\n[attack] mode={MODE} FIRED={attacked['fired']} path={attacked['path']} "
           f"size_before={attacked['size']} first_at={attacked['first_at']}s "
           f"hits_at_attack={attacked['hits_at_attack']} cycles={attacked['cycles']} errors={attacked['errors']}")
       if not attacked["fired"]:
           say("  *** ATTACK DID NOT FIRE — this row proves NOTHING about the product ***")
   
   # cobertura: did coverage data survive?
   rates = []
   for f in WORK.rglob("*.cobertura.xml"):
       txt = f.read_text(errors="replace")
       m = re.search(r'line-rate="([0-9.]+)"', txt[:2000])
       rates.append((str(f.relative_to(WORK)), m.group(1) if m else "?", f.stat().st_size))
   say(f"\n[cobertura] {rates if rates else 'NONE produced'}")
   
   # litter: what did the run leave behind?
   time.sleep(2)
   litter_after = litter_inventory()
   new_litter = {k: v for k, v in litter_after.items() if k not in litter_before}
   sizes = {}
   for v in new_litter.values():
       sizes[v] = sizes.get(v, 0) + 1
   say(f"\n[litter left behind] {len(new_litter)} new /tmp/CodeCoverage* entries "
       f"(before {len(litter_before)}, after {len(litter_after)}); size histogram: {sizes}")
   for k in sorted(new_litter)[:5]:
       say(f"  {k}  {new_litter[k]}")
   
   say(f"\n[AccessViolation in output?] {'YES' if 'AccessViolation' in out else 'no'}")
   fm = re.search(r"Fatal error\. ([^\n]+)", out)
   say(f"[fatal error line] {fm.group(1) if fm else 'none'}")
   say(f"[exit] rc={proc.returncode}")
   
   pathlib.Path(f"/data/tmp/cov-repro/log-live3-{VERSION}-{TFM}-{MODE}.txt").write_text(
       "\n".join(LOG) + "\n\n--- full output ---\n" + out)
   say(f"\n[log] /data/tmp/cov-repro/log-live3-{VERSION}-{TFM}-{MODE}.txt")

   python3 gen_repro.py
   python3 probe_live3.py 18.11.2 net8.0 xunit.v3.mtp-v2 observe    # control
   python3 probe_live3.py 18.11.2 net8.0 xunit.v3.mtp-v2 truncate   # green, empty report
   python3 probe_live3.py 18.11.2 net8.0 xunit.v3.mtp-v2 hammer     # no fault, about 8M cycles
   python3 probe_live3.py 18.11.2 net8.0 xunit.v3.mtp-v2 delete     # no fault
   python3 probe_live3.py 18.11.0 net8.0 xunit.v3.mtp-v2 hammer     # positive control, AV
   python3 probe_live3.py 18.10.0 net8.0 xunit.v3.mtp-v2 truncate   # positive control, AV

   Environment: .NET SDK 10.0.301, runtimes 8.0.26 / 10.0.12, Ubuntu 26.04 x64, xunit.v3 3.2.2 (xunit.v3.mtp-v2). /tmp here is tmpfs, so the buffer is fully page-backed and the sparse-file/ENOSPC path is still not exercised by these runs — though with no mapping left, that path should now surface as a caught IOException rather than a fault.

   Thanks for sticking with this one.
9. pcshrosbree commented on Oct 1, 2026

   

   Peter Shrosbree (pcshrosbree)

   on Oct 1, 2026

   Author

   More actions

   Closing this: 18.11.2 fixes the reported fault on Linux (results in the comment above), and it is still the latest release. Thanks again for the fix.

   I've filed the two remaining items separately so they aren't lost when this closes:

   • Coverage buffer that loses its initialization byte is reported as a passing run with an empty report (line-rate="1") #251: a buffer that loses its initialization byte is reported as a passing run with an empty report.
   • Each coverage run leaves ~50 six-byte /tmp/CodeCoverage.* files behind on Linux (18.11.x) #252: each run leaves about 50 small files in /tmp.
10. Peter Shrosbree (pcshrosbree)

    closed this as completedon Oct 1, 2026
11. fhnaseer

    added this to the 18.11 milestone on Oct 2, 2026