Skip to content

fix(ActivationCache): build layer-0 stacks in the cache's dtype and d… - #1925

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
avgvi:fix/activation-cache-layer0-dtype-device
Oct 21, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
avgvi:fix/activation-cache-layer0-dtype-device

Conversation

@avgvi

@avgvi avgvi commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes #1924 .

At layer=0, stack_head_results and stack_neuron_results return an empty tensor shaped like hook_embed. It was created with torch.zeros(..., device=self.model.cfg.device) and no dtype (1, 2, 3), so on a bf16 model, or a cache upcast to float64, layer 0 came back float32 while later layers followed the cache.

The empty tensor now takes its dtype and device from the tensor it was already using for its shape (pos_slice.apply(self["hook_embed"], dim=-2), projected when project_output_onto is set). In _stack_neuron_results_apply_ln_projected, the dtype is that tensor's dtype promoted with the cached LN scale and the projection, because the layer >= 1 result is computed from all three (float32 on a bf16 model, whose LN scales are float32).

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Testing

I added test_layer_zero_stacks_follow_the_cache_dtype to tests/unit/test_activation_cache.py.

  • stack_head_results(0) and stack_neuron_results(0) are empty and match hook_embed's dtype and device
  • stack_neuron_results(apply_ln=True, project_output_onto=d) has the same dtype at layers 0 and 1

Before (dev):

$ uv run pytest tests/unit/test_activation_cache.py -q -k layer_zero_stacks_follow
E           assert torch.float32 == torch.bfloat16
E           assert torch.float32 == torch.float64
4 failed

After:

$ uv run pytest tests/unit/test_activation_cache.py -q
261 passed
dev, layer 0 / layer 1 this PR, layer 0 / layer 1
stack_head_results float32 / bfloat16 bfloat16 / bfloat16
stack_neuron_results float32 / bfloat16 bfloat16 / bfloat16
stack_neuron_results(project_output_onto=d) float32 / bfloat16 bfloat16 / bfloat16
stack_neuron_results(apply_ln=True, project_output_onto=d) float32 / float32 float32 / float32

make check-format and uv run mypy . are clean.

Checklist:

  • I have commented my code, particularly in hard-to-understand areas (not needed for this change)
  • I have made corresponding changes to the documentation (no docs change needed)
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@jlarson4

Copy link
Copy Markdown
Collaborator

Looks good, merging

@jlarson4
jlarson4 merged commit 873056f into TransformerLensOrg:dev Oct 21, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants