Skip to content

fix: preserve prompt attention masks in generation caches - #1899

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/generate-cache-attention-mask
Oct 14, 2026
Merged

jlarson4 merged 1 commit into
TransformerLensOrg:devfrom
emerardd:fix/generate-cache-attention-mask

Conversation

@emerardd

Copy link
Copy Markdown
Contributor

Description

Preserve the effective prompt attention mask when generate(return_cache=True) recomputes the activation cache. The generation loop already uses and extends this mask, but the extra full-sequence run_with_cache call dropped it. As a result, sampled logits were correct while the returned cache used a different attention context and derived positions, including attention to excluded prompt tokens.

The cache recomputation now extends initial_attention_mask by the number of tokens actually generated and forwards it to run_with_cache. This handles explicit masks, padding-derived masks, and EOS-shortened output while preserving mask dtype/device. The unmasked path is unchanged, as are generation sampling, return formats, cache filtering/offload, and existing unsupported-path guards.

The generate API docstring now states that cache equivalence requires the effective prompt mask extended over the generated tokens, rather than an unconditional unmasked run_with_cache(output) call. No dependency changes are needed.

Regression coverage

New offline tests use real tiny GPT-2 weights from config, without Hub downloads. They cover left-padding and interior-gap masks, bool/int masks, all-ones and absent-mask controls, both KV-cache modes, EOS-shortened sequences, output logits, return-input-token tuples, and an inferred left-padding mask with filtered CPU cache output. Assertions compare every cached tensor with an independently masked recomputation, compare cache logits with each generation step's logits, check excluded-key attention is zero, and verify the caller's mask is not mutated.

Validation

  • Before the fix: 17 failed, 16 passed in the new test file; masked cases failed while all-ones/unmasked controls passed.
  • After the fix: 68 passed across test_generate_cache_attention_mask.py, the existing return-cache and generation attention-mask integration files, and the generation KV-cache unit tests.
  • Mypy: no issues in 393 source files.
  • Makefile-equivalent pycln/isort/Black checks for changed files and git diff --check passed.
  • Complete unit-test and PR-review suites were not run. Existing SWIG deprecations and shared-config warnings appeared in the regression run; this change adds no warning sites. No GPU validation is claimed.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Documentation changes are in the generate API docstring. The full-unit-suite checkbox remains unchecked because only the affected test surface was run.

@jlarson4

Copy link
Copy Markdown
Collaborator

Looks good! Merging

@jlarson4
jlarson4 merged commit 3ed294e into TransformerLensOrg:dev Oct 14, 2026
55 of 56 checks passed
@emerardd
emerardd deleted the fix/generate-cache-attention-mask branch October 17, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants