Repository navigation
fix: keep RMSNorm offsets out of Gemma3 vision LayerNorm - #1897
Conversation
jlarson4
left a comment
There was a problem hiding this comment.
Thanks for putting this fix together! The offline Gemma3 image-path repro is a really useful test to have. Just a couple comments below, let me know if you have any questions
| weight = ( | ||
| (1.0 + self.weight) | ||
| if getattr(self.config, "rmsnorm_uses_offset", False) | ||
| if self.uses_rms_norm and getattr(self.config, "rmsnorm_uses_offset", False) |
There was a problem hiding this comment.
uses_rms_norm is False for every nn.LayerNorm subclass, including Nemotron's LayerNorm1P, which applies weight + 1 and sets rmsnorm_uses_offset precisely so these paths keep the offset. With this gate, identity hooks or backward hooks on Nemotron norms change the logits substantially, and the LN-rule gradient loses the offset too (line 381).
tests/integration/model_bridge/test_nemotron_adapter.py now fails. Can you look for a gate that excludes the vision tower's plain LayerNorm without dropping Nemotron's offset?
There was a problem hiding this comment.
Thank you, you're right: offset affine weights are independent of RMS vs centered LayerNorm. I've replaced the RMS-only gate with a shared uses_weight_offset property and an explicit component-level override. Nemotron's ln1, ln2, and ln_final enable the override while remaining LayerNorm; ordinary vision LayerNorm does not inherit the language RMS offset. Both affine reconstruction and the LN-rule VJP use this property.
Added real HF LayerNorm1P output/input/weight/bias gradient and relevance-rule regressions, plus tiny full-model tests for all three Nemotron norm slots. The expanded local surface passes 206 tests, including the unchanged original Nemotron integration file and both failing CI cases. Mypy and formatting checks pass; the full suite was not rerun locally.
| model = Gemma3ForConditionalGeneration(config).eval() | ||
| with torch.no_grad(): | ||
| # HF initializes this projection to zero, hiding vision-path regressions. | ||
| model.model.multi_modal_projector.mm_input_projection_weight.normal_(0, 0.2) |
There was a problem hiding this comment.
The post_layernorm cases pass even without the fix. With HF's default LayerNorm init, the offset is a uniform 2× scale, and the projector's mm_soft_emb_norm normalizes it away. Would it be possible to perturb the vision tower's LayerNorm weights here too, so all three norms guard the fix?
There was a problem hiding this comment.
Good catch. I've initialized the vision tower's LayerNorm weights to a deterministic, nonuniform range (0.7 to 1.3) while retaining the nonzero image projection. The incorrect affine offset now changes channel ratios, so the projector's RMSNorm cannot cancel it as a uniform scaling.
I verified the red control by restoring the pre-fix unconditional-offset behavior in memory: all 12 image-path cases fail, including all four post_layernorm combinations, with no setup/teardown errors. They all pass with the fix, as part of the expanded 206-test local regression run. No tolerance changes were needed.
|
Looks good @emerardd! Merging |
Description
Make normalization affine offsets component-specific instead of allowing a shared language configuration to affect plain vision LayerNorm. Gemma3's multimodal adapter shares its language configuration with the SigLIP vision tower. Although
NormalizationBridge.uses_rms_normcorrectly identifies the tower'snn.LayerNormmodules, affine reconstruction and the native relevance-rule VJP previously applied the sharedrmsnorm_uses_offsetflag unconditionally.This caused a content-preserving clone hook or a backward-only normalization hook to change image-path logits. A tiny random Gemma3 with a nonzero image projection reproduced a maximum logit difference of approximately 0.144. The relevance-rule path kept the native forward unchanged but incorrectly applied the offset to the input VJP.
Both paths now use a shared
uses_weight_offsetproperty, independent of the normalization algorithm. By default, only RMSNorm reads the RMS offset configuration; an optional component-level override preserves legitimate offset LayerNorms. Nemotron's three LayerNorm1P slots explicitly enable it while remaining centered LayerNorms. The shared configuration and language RMSNorm offset remain unchanged. Hook edits, backward hooks, and the existing native-fallback warnings are preserved. The constructor option is additive; no model-facing API or dependency changes are required.Regression coverage
hook_scaleandhook_normalized, backward-only hooks, input/weight/bias gradient parity, Python normalization, and a non-identity forward edit.ln1,ln2, andpost_layernorm, both internal hooks, and forward/backward hook directions. Vision LayerNorm weights are nonuniform across channels so the projector's RMSNorm cannot cancel a uniform offset error. The image projection is initialized to nonzero synthetic weights because HF's zero initialization would hide vision-path regressions.Validation
post_layernormcases, with no setup/teardown errors.uv run --no-sync mypy . --cache-dir .adapter-workspace/mypy-cache: no issues in 392 source files.git diff --checkpassed.Type of change
Checklist
Documentation is unchanged because this restores the existing normalization/hook contract without introducing an API. The full-unit-suite checkbox remains unchecked because only the affected test surface was run.