Skip to content

fix: keep RMSNorm offsets out of Gemma3 vision LayerNorm - #1897

Merged
jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
emerardd:fix/layernorm-rms-offset
Oct 14, 2026
Merged

jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
emerardd:fix/layernorm-rms-offset

Conversation

@emerardd

@emerardd emerardd commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Make normalization affine offsets component-specific instead of allowing a shared language configuration to affect plain vision LayerNorm. Gemma3's multimodal adapter shares its language configuration with the SigLIP vision tower. Although NormalizationBridge.uses_rms_norm correctly identifies the tower's nn.LayerNorm modules, affine reconstruction and the native relevance-rule VJP previously applied the shared rmsnorm_uses_offset flag unconditionally.

This caused a content-preserving clone hook or a backward-only normalization hook to change image-path logits. A tiny random Gemma3 with a nonzero image projection reproduced a maximum logit difference of approximately 0.144. The relevance-rule path kept the native forward unchanged but incorrectly applied the offset to the input VJP.

Both paths now use a shared uses_weight_offset property, independent of the normalization algorithm. By default, only RMSNorm reads the RMS offset configuration; an optional component-level override preserves legitimate offset LayerNorms. Nemotron's three LayerNorm1P slots explicitly enable it while remaining centered LayerNorms. The shared configuration and language RMSNorm offset remain unchanged. Hook edits, backward hooks, and the existing native-fallback warnings are preserved. The constructor option is additive; no model-facing API or dependency changes are required.

Regression coverage

  • Standard LayerNorm with a shared RMS/offset configuration: identity-clone hooks on hook_scale and hook_normalized, backward-only hooks, input/weight/bias gradient parity, Python normalization, and a non-identity forward edit.
  • Relevance-rule LayerNorm VJP against a detached-denominator oracle with a nonuniform upstream gradient. A summed output alone hides the offset error through LayerNorm centering.
  • Offline tiny Gemma3 image-path parity against the original HF model across SigLIP ln1, ln2, and post_layernorm, both internal hooks, and forward/backward hook directions. Vision LayerNorm weights are nonuniform across channels so the projector's RMSNorm cannot cancel a uniform offset error. The image projection is initialized to nonzero synthetic weights because HF's zero initialization would hide vision-path regressions.
  • Real HF Nemotron LayerNorm1P component output/input/weight/bias gradient parity, backward-only and forward hooks, Python normalization, and the relevance-rule VJP; tiny Nemotron adapter wiring and full-model hook parity across all three norm slots.
  • Existing Gemma offset RMSNorm, adapter, vision, and relevance-rule tests.

Validation

  • Before the fix: nine forward/hook/image-path cases failed; the additional shared-offset relevance-rule VJP regression also failed.
  • Restoring the pre-fix unconditional-offset semantics in memory makes all 12 strengthened Gemma3 image-path cases fail, including all four post_layernorm cases, with no setup/teardown errors.
  • After the review follow-up: 206 passed across the affected normalization, vision, Gemma/Nemotron adapter, relevance-rule, and integration test surface. This includes the unchanged original Nemotron integration file and both previously failing CI cases.
  • uv run --no-sync mypy . --cache-dir .adapter-workspace/mypy-cache: no issues in 392 source files.
  • Makefile-equivalent pycln, isort, and Black checks passed for the changed files; git diff --check passed.
  • The complete unit-test and PR-review suites were not run. No GPU validation is claimed. The run emitted existing SWIG dependency deprecation warnings; fallback warnings are explicitly tested.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Documentation is unchanged because this restores the existing normalization/hook contract without introducing an API. The full-unit-suite checkbox remains unchecked because only the affected test surface was run.

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this fix together! The offline Gemma3 image-path repro is a really useful test to have. Just a couple comments below, let me know if you have any questions

weight = (
(1.0 + self.weight)
if getattr(self.config, "rmsnorm_uses_offset", False)
if self.uses_rms_norm and getattr(self.config, "rmsnorm_uses_offset", False)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uses_rms_norm is False for every nn.LayerNorm subclass, including Nemotron's LayerNorm1P, which applies weight + 1 and sets rmsnorm_uses_offset precisely so these paths keep the offset. With this gate, identity hooks or backward hooks on Nemotron norms change the logits substantially, and the LN-rule gradient loses the offset too (line 381).

tests/integration/model_bridge/test_nemotron_adapter.py now fails. Can you look for a gate that excludes the vision tower's plain LayerNorm without dropping Nemotron's offset?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, you're right: offset affine weights are independent of RMS vs centered LayerNorm. I've replaced the RMS-only gate with a shared uses_weight_offset property and an explicit component-level override. Nemotron's ln1, ln2, and ln_final enable the override while remaining LayerNorm; ordinary vision LayerNorm does not inherit the language RMS offset. Both affine reconstruction and the LN-rule VJP use this property.

Added real HF LayerNorm1P output/input/weight/bias gradient and relevance-rule regressions, plus tiny full-model tests for all three Nemotron norm slots. The expanded local surface passes 206 tests, including the unchanged original Nemotron integration file and both failing CI cases. Mypy and formatting checks pass; the full suite was not rerun locally.

model = Gemma3ForConditionalGeneration(config).eval()
with torch.no_grad():
# HF initializes this projection to zero, hiding vision-path regressions.
model.model.multi_modal_projector.mm_input_projection_weight.normal_(0, 0.2)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The post_layernorm cases pass even without the fix. With HF's default LayerNorm init, the offset is a uniform 2× scale, and the projector's mm_soft_emb_norm normalizes it away. Would it be possible to perturb the vision tower's LayerNorm weights here too, so all three norms guard the fix?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I've initialized the vision tower's LayerNorm weights to a deterministic, nonuniform range (0.7 to 1.3) while retaining the nonzero image projection. The incorrect affine offset now changes channel ratios, so the projector's RMSNorm cannot cancel it as a uniform scaling.

I verified the red control by restoring the pre-fix unconditional-offset behavior in memory: all 12 image-path cases fail, including all four post_layernorm combinations, with no setup/teardown errors. They all pass with the fix, as part of the expanded 206-test local regression run. No tolerance changes were needed.

@jlarson4
jlarson4 merged commit c80c38e into TransformerLensOrg:dev Oct 14, 2026
28 checks passed
@jlarson4

Copy link
Copy Markdown
Collaborator

Looks good @emerardd! Merging

@emerardd
emerardd deleted the fix/layernorm-rms-offset branch October 17, 2026 12:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants