Skip to content

Fix flaky direct path patching correctness test - #1783

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/direct-path-patching-flaky-test
Sep 14, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/direct-path-patching-flaky-test

Conversation

@emerardd

Copy link
Copy Markdown
Contributor

Description

tests/unit/test_direct_path_patching.py::TestCorrectness::test_correctness_against_actual_ln_forward fails intermittently in coverage-test, e.g.:

AssertionError: Linear-LN approx -0.323756 disagrees with actual-LN ref -0.322587 (diff=1.17e-03). process_weights_() should make these exact.

Root cause

  • The tiny_model fixture builds HookedTransformer(cfg) without seeding. The only seed is the one-time global seed in tests/conftest.py::pytest_configure, so under -n auto --dist loadscope the weights depend on which tests a worker ran before this module. (The torch.manual_seed(42) in tokens_and_caches runs after the model is built.)
  • get_act_patch_direct_path reuses the corrupted run's ln1.hook_scale. Folding LN does not make that exact: LayerNormPre still divides by an input-dependent norm. The gap to the true-LN reference depends on the weights.

I re-ran the test's computation over 40 weight seeds: errors ranged from 5.6e-06 to 2.7e-03, and 7/40 exceeded the 1e-3 tolerance. Module-scoped fixtures are reused on rerun, so --reruns doesn't help.

Changes

  • Seed tiny_model locally (torch.manual_seed(0), error 1.85e-04 on that model).
  • Remove the unused torch.manual_seed(0) inside the test.
  • Correct the test docstring, the assertion message, and the module docstring, which all claimed the approximation is exact once LN is folded.

No behaviour change to direct_path_patching.py; docstring only.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Local: pytest tests/unit/test_direct_path_patching.py (16 passed), same file + tests/unit/tools under -n 4 --dist loadscope (738 passed), mypy . clean, black/isort/pycln clean.

tiny_model was built from the unseeded global RNG, so its weights varied with
xdist scheduling. The linear-LN approximation error is weight-dependent
(~17% of random inits exceed the 1e-3 tolerance), making
test_correctness_against_actual_ln_forward flaky. Seed the fixture and correct
the docstrings that claimed folding LN makes the approximation exact.
@jlarson4
jlarson4 merged commit 0106853 into TransformerLensOrg:dev Sep 14, 2026
26 checks passed
@jlarson4 jlarson4 mentioned this pull request Sep 14, 2026
7 tasks done
@emerardd
emerardd deleted the fix/direct-path-patching-flaky-test branch October 17, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants