Skip to content

fix: keep synthetic output biases out of the optimizer - #1871

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
kiteretsu903:fix-preserve-output-bias-training
Oct 9, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
kiteretsu903:fix-preserve-output-bias-training

Conversation

@kiteretsu903

Copy link
Copy Markdown
Contributor

Description

as I mentioned in #1854, I ran into an optimizer-parity failure during validation and wanted to look into it separately. I found that wrapping a model with no output bias still adds one as a trainable parameter. For DistilGPT2, that gives the optimizer 50,257 extra values to update.

I changed that synthetic bias to a buffer, so users can use bridge.parameters() without having to filter it out themselves. Wrapping a layer with a real bias still preserves its parameter. Bias folding, saving/loading, and CPU/disk offloading are covered too, including removing and reattaching the offload hooks.

In a small float64 reproduction, three AdamW steps caused the wrapped layer's logits to drift by about 0.003 from the original layer. With the fix, they matched exactly.

I kept the float64 optimizer test from #1867 and removed its manual bias filter. I also added FP32 training, cross-entropy training, and focused bias/offload regression checks. There's still some FP32 AdamW drift to investigate separately.

Validation

I ran these locally with Python 3.12 and the existing locked CPU dependencies:

Check Result
Full unit suite 7343 passed, 55 skipped, 3 xfailed
Doctests 13 passed, 24 skipped
Acceptance suite 204 passed, 28 skipped
Full integration suite 1489 passed, 6 skipped
make format and make check-format Passed
Full-project mypy Passed, 391 source files

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation: no documentation changes needed for this fix.
  • My changes generate no new warnings: existing SWIG and hook warnings still appear in the test output.
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes: the full CPU PR suite passed as listed above.
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@jlarson4

jlarson4 commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Looks good, thanks @kiteretsu903!

@jlarson4
jlarson4 merged commit 5eca4aa into TransformerLensOrg:dev Oct 9, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants