Repository navigation
Conversation
tiny_model was built from the unseeded global RNG, so its weights varied with xdist scheduling. The linear-LN approximation error is weight-dependent (~17% of random inits exceed the 1e-3 tolerance), making test_correctness_against_actual_ln_forward flaky. Seed the fixture and correct the docstrings that claimed folding LN makes the approximation exact.
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
tests/unit/test_direct_path_patching.py::TestCorrectness::test_correctness_against_actual_ln_forwardfails intermittently incoverage-test, e.g.:Root cause
tiny_modelfixture buildsHookedTransformer(cfg)without seeding. The only seed is the one-time global seed intests/conftest.py::pytest_configure, so under-n auto --dist loadscopethe weights depend on which tests a worker ran before this module. (Thetorch.manual_seed(42)intokens_and_cachesruns after the model is built.)get_act_patch_direct_pathreuses the corrupted run'sln1.hook_scale. Folding LN does not make that exact:LayerNormPrestill divides by an input-dependent norm. The gap to the true-LN reference depends on the weights.I re-ran the test's computation over 40 weight seeds: errors ranged from 5.6e-06 to 2.7e-03, and 7/40 exceeded the 1e-3 tolerance. Module-scoped fixtures are reused on rerun, so
--rerunsdoesn't help.Changes
tiny_modellocally (torch.manual_seed(0), error 1.85e-04 on that model).torch.manual_seed(0)inside the test.No behaviour change to
direct_path_patching.py; docstring only.Type of change
Checklist:
Local:
pytest tests/unit/test_direct_path_patching.py(16 passed), same file +tests/unit/toolsunder-n 4 --dist loadscope(738 passed),mypy .clean, black/isort/pycln clean.