Repository navigation
fix(bridge): set rmsnorm_uses_offset on the Qwen3.5, Qwen3Next and Nemotron adapters - #1869
Conversation
HF's Qwen3.5, Qwen3.5-MoE and Qwen3Next RMSNorms scale by (1 + weight). Without the flag, NormalizationBridge dropped the 1 + whenever it computed a norm itself: on hook_scale/hook_normalized edits, with a backward hook attached, and in the LN-rule backward pass. Fixes TransformerLensOrg#1868. Worked through with a hand from Claude.
jlarson4
left a comment
There was a problem hiding this comment.
Thank you for putting this fix together @willgosnold, it looks great! Just one comment below
| # split [query|gate] per head at forward time; slicing the gate out changes outputs. | ||
| setattr(cfg, "gated_q_proj", True) | ||
| # HF's Qwen3NextRMSNorm scales by (1 + weight), not weight (cf. Gemma). | ||
| setattr(cfg, "rmsnorm_uses_offset", True) |
There was a problem hiding this comment.
Nemotron's LayerNorm1P also scales by (weight + 1), and our Nemotron architecture adapter doesn't declare this flag. Would you be willing to add this same fix for Nemotron as part of your PR?
There was a problem hiding this comment.
Thanks! Done in a4f9e8d: the Nemotron adapter now sets rmsnorm_uses_offset, with an identity-edit test on the tiny Nemotron checkpoint. Without the flag the edit changed logits by 0.28 to 0.5; with it they match. It also fixes the backward-hook and LN-rule paths, and DLA's identity-norm guard now correctly rejects Nemotron, which it previously accepted. NemotronH uses a plain weight * x norm, so it doesn't need the flag.
NemotronLayerNorm1P applies gamma as (weight + 1). The adapter already disabled fold_ln for this, but never set the flag, so the bridge's own norm paths (hook_scale/hook_normalized edits, backward hooks, LN-rule) scaled by weight alone.
Jais2 subclasses the Nemotron adapter but uses plain nn.LayerNorm and re-enables fold_ln, so inheriting the offset made folding use 1 + w and broke compat-mode parity.
|
Thanks for the quick updates! Approved and merged |
Description
Fixes #1868.
HF's Qwen3.5, Qwen3.5-MoE and Qwen3Next RMSNorms scale by
(1 + weight), the same convention as Gemma. Their adapters never setcfg.rmsnorm_uses_offset, soNormalizationBridgescales byweightalone whenever it computes a norm's output itself. That happens when a forward hook editshook_scaleorhook_normalized, when a backward hook is attached to either, and in the LN-rule backward pass. Unhooked forward passes andrun_with_cachecall HF's module and were already correct.This PR sets the flag in each adapter's
__init__, beforesuper().__init__()builds the component mapping. It has to be set there, becausesetup_blocks_bridgedeep-copies the config into every block. Setting it on a booted bridge only reachesln_final.qwen3_5.pycovers Qwen3.5, and Qwen3.5-MoE through inheritance.qwen3_5_multimodal.pycovers the Qwen3.5 multimodal adapter, and the Qwen3.5-MoE multimodal adapter through inheritance.qwen3_next.pycovers Qwen3Next.Nemotron, added at review:
NemotronLayerNorm1Papplies gamma as(weight + 1)too, and its adapter had the same gap.nemotron.pynow sets the flag as well. Its existing integration tests gain the identity-edit test onblocks.0.ln1andln_final. Without the flag the edit changed logits by 0.28 to 0.5; with it they match. The backward-hook and LN-rule paths are fixed too, and DLA's identity-norm guard now correctly rejects Nemotron, which it previously accepted. NemotronH uses a plainweight * xnorm, so it doesn't need the flag.User-facing change: on these five adapters, editing or backward-hooking a norm's
hook_scaleorhook_normalizednow gives the same output as HF's forward, and LN-rule gradients through their norms use(1 + weight). Unhooked and observe-only use is unchanged.Tests (behaviour only, in the existing test files):
TestQwen3_5RMSNormOffset(unit) builds a tinyQwen3_5ForCausalLMand moves its norm weights off their zero init. It checks that an identityhook_scaleedit onblocks.0.ln1,blocks.3.attn.q_normandln_finalleaves the logits unchanged. It checks that a backward hook onln_final.hook_scaleleaves the forward output unchanged. It checks that the LN-rule gradient throughblocks.0.ln1equals(1 + weight) / denom.TestQwen3NextRMSNormOffset(unit) runs the same identity-edit test on a tinyQwen3NextForCausalLM.blocks.0.ln1andln_final, added to their existing integration tests, which load the tinytrl-internal-testingcheckpoints.helpers.pygets two small helpers for the unit tests.The Qwen3.5-MoE text adapter inherits the fix from Qwen3.5, so it has no test of its own.
Validation:
devwithout the fix, with relative logit errors from 6.1e-3 to 1.0. With the fix they all pass, with errors around 1e-7.make unit-test: 7144 passed, 61 skipped, 3 xfailed, 0 failed.make check-formatanduv run mypy .are clean.fold_lnpaths inweight_processing.py, the batched-expert fold intransformer_bridge.pyand the weight-processing benchmark skip these adapters, because they setsupports_fold_ln = False. On tiny models,enable_compatibility_mode()gives identical logits with and without the flag, and log-probs match HF to 1e-6.Not in this PR: the question in the issue about whether these adapters should also get Gemma's
ArithmeticTensorConversion(ADDITION, 1.0)for their norm weights.Type of change
Checklist:
Claude helped me track this down and write the fix; I've reviewed the change and run the tests myself.