Hosted and vendored NuGet reject a packages.lock.json with a UTF-8 BOM that dotnet restores fine: hosted skips the redirect and exits 0 success, vendored fails apply_failed #623

[agent] Found by the scheduled NuGet / dotnet bug-hunt routine (ledger #320).

Summary

If packages.lock.json starts with a UTF-8 BOM (EF BB BF), dotnet restore and dotnet restore --locked-mode read it without complaint. socket-patch's two NuGet lock rewriters parse it with a bare serde_json::from_str, which rejects the BOM:

• Hosted (scan --mode hosted): the whole NuGet redirect is skipped with redirect_nuget_lock_unparseable ("packages.lock.json is not valid JSON"). The run still exits 0 with status: success and redirected: 0. Nothing is wired and the project stays unpatched.
• Vendored (scan --mode vendored): apply_failed "unparseable packages.lock.json: expected value at line 1 column 1", exit 1 partial_failure. This one is loud, but it refuses a lock the toolchain accepts.

The rest of the codebase already treats a BOM as encoding, not content:

• VEX discovery reads the same lock through parse_json → vendor::common::parse_json_manifest (vex/discover/mod.rs:1391), which strips the BOM. Its comment says "the rewriters keep one, so a wired BOM lock must stay discoverable".
• utils::serde::strip_bom, the manifest reader and the npm crawler all strip it.
• The npm twin of this bug ("BOM locks are refused") was npm lock rewrites drop CRLF (and hosted drops tab indent), so rollback and vendor --revert are not byte-exact; BOM locks are refused #324, now fixed and closed.

Realism: NuGet writes the lock without a BOM, but a lock that has been saved by a Windows editor or rewritten by Windows PowerShell 5.1 (Set-Content -Encoding UTF8 / Out-File) carries one, and dotnet keeps restoring it, so the user never notices.

Impact

• Hosted: a scan that patched nothing reports success with exit 0. The only signal is a warning that calls a valid lock "not valid JSON". CI pipelines that gate on the exit code ship the unpatched package.
• Vendored: patching is impossible until the user finds and strips an invisible byte.

Repro (Linux, .NET SDK 8.0.131, main 045d7ec)

This uses the Backend stand-in from crates/socket-patch-cli/tests/e2e_nuget_dotnet_build.rs (patch API plus the hosted NuGet v3 feed for a patched Newtonsoft.Json 13.0.3) and the same fixture: app.csproj with RestorePackagesWithLockFile, a nuget.org-only nuget.config, and a real dotnet restore.

# fixture restored from nuget.org -> packages.lock.json (no BOM), then:
printf '\xef\xbb\xbf' > l; cat packages.lock.json >> l; mv l packages.lock.json
dotnet restore --locked-mode            # rc 0: dotnet accepts the BOM lock

socket-patch scan --mode hosted --json --yes --api-url $URI --org test-org \
  --api-token fake-token --patch-server-url $URI
# rc 0, status "success", redirect: {"redirected": 0, "rewrittenFiles": [],
#   "warnings": [{"code": "redirect_nuget_lock_unparseable",
#                 "detail": "packages.lock.json is not valid JSON; nuget redirect skipped"}]}
# nuget.config untouched, lock contentHash still upstream; a fresh-checkout
# `dotnet restore --locked-mode` installs PRISTINE LICENSE.md.

socket-patch scan --mode vendored --vendor-source service --json --yes ...   # same flags
# rc 1, status "partial_failure":
#   {"action":"failed","errorCode":"apply_failed",
#    "error":"unparseable packages.lock.json: expected value at line 1 column 1"}

Control: the same project without the BOM passes in both modes (lock re-pinned to the patched contentHash, and the fresh-checkout locked restore installs the PATCHED bytes).

Expected vs actual

• Expected: docs/ecosystems.md (NuGet row) says hosted does "nuget.config source + source-mapping, packages.lock.json contentHash rewrite" and vendored does a "packages.lock.json contentHash pin". A lock the toolchain reads should be rewritten, with the BOM preserved so a revert stays byte-exact, as for npm after npm lock rewrites drop CRLF (and hosted drops tab indent), so rollback and vendor --revert are not byte-exact; BOM locks are refused #324. At minimum, a hosted run that redirects nothing because of a refusal shouldn't report success / exit 0.
• Actual: hosted skips silently with exit 0, and vendored refuses.

Matrix

First bad version

This isn't a regression. On release v4.0.0 (npm), hosted fails in a different way: it wires nuget.config (redirected: 1, rewrittenFiles: ["nuget.config"]) but can't parse the lock, so the upstream contentHash stays, and the fresh-checkout locked restore fails NU1403: Package content hash validation failed for Newtonsoft.Json.13.0.3. Vendored on v4.0.0 is the same apply_failed.

Suspect code

• crates/socket-patch-core/src/patch/redirect/mod.rs:5015: serde_json::from_str::<Value>(text) on the raw lock text. Compare line 3460 in the same file, which already does t.strip_prefix('\u{feff}').
• crates/socket-patch-core/src/vendor/nuget_feed.rs:1261-1262: lock_value → serde_json::from_str, used by edit_lock (1279) and lock_pinned (1318).
• The helpers already exist: vendor::common::parse_json_text / parse_json_manifest. The contentHash string surgery keeps all the other bytes, so the BOM would survive.

No probe branch was run: the parse is platform-independent.