Agent-mode apply reports gem native-extension patches as applied without rebuilding #1996

[agent]

Summary

Some gem security fixes change only a C extension's sources: ext/**/*.c, *.h, extconf.rb and similar. Whether such a patch actually takes effect depends on the install mode:

Example: every patch for json 2.10.x–2.17.x under GHSA-x2f5-4prf-w687 changes only ext/json/ext/parser/parser.c and/or ext/json/ext/generator/generator.c. In hosted mode these patches work. Agent-mode apply reports them applied, but the installed json/ext/parser.so is unchanged.

Requested changes

1. Agent mode: never report a native-extension-only gem patch as applied (required)

When a gem patch changes any native-extension build input, apply must not report success while the compiled extension is unchanged. Choose one:

• Refuse (simplest; matches vendored mode): fail that purl with native_extensions_unsupported. Write nothing, and point the user at scan --mode hosted.
• Rebuild (optional, later): after writing the patched sources, rebuild each affected extension against the active Ruby (the steps gem pristine --extensions runs: extconf.rb + make in the extension dir, then install the built artifact where RubyGems placed the original, under extensions/<arch>/<ruby-abi>/<gem>-<version>/ and the gem's lib/). If no compiler toolchain or Ruby headers are available, fall back to the refusal.

A patch that changes Ruby files AND extension sources needs the same treatment. Applying only the Ruby half leaves the gem half-patched.

2. VEX must agree

vex must not attest a purl as patched when agent-mode apply was refused, or when the installed compiled extension does not come from patched sources.

3. Rollback (only if the rebuild option is built)

Rollback must also restore the original compiled extension, not only the source files.

Detection

Treat a changed file as a native-extension build input when any of these holds:

• Its extension is .c, .h, .cc, .cpp, .cxx, .hh, .hpp, .hxx, .m, .y or .rl.
• It is named extconf.rb, or it is depend under ext/.
• It is a file the gemspec's extensions entries build. The vendored backend already parses this in gemspec_declares_extensions.

Out of scope / already correct

• Hosted mode needs no change for these patches. Two existing behaviors still apply: a precompiled platform gem (e.g. x86_64-linux) already fails closed with redirect_gem_platform_unsupported, and a frozen install must be unfrozen to pick up the patched gem (redirect_gem_frozen_install).
• A gem Ruby loads as a default gem without a Gemfile.lock entry (json ships with Ruby) cannot be redirected in hosted mode. That is a separate limitation, not covered here.