Repository navigation
fix: preserve prompt attention masks in generation caches - #1899
Merged
jlarson4 merged 1 commit intoOct 14, 2026
Merged
Conversation
Collaborator
|
Looks good! Merging |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Preserve the effective prompt attention mask when
generate(return_cache=True)recomputes the activation cache. The generation loop already uses and extends this mask, but the extra full-sequencerun_with_cachecall dropped it. As a result, sampled logits were correct while the returned cache used a different attention context and derived positions, including attention to excluded prompt tokens.The cache recomputation now extends
initial_attention_maskby the number of tokens actually generated and forwards it torun_with_cache. This handles explicit masks, padding-derived masks, and EOS-shortened output while preserving mask dtype/device. The unmasked path is unchanged, as are generation sampling, return formats, cache filtering/offload, and existing unsupported-path guards.The
generateAPI docstring now states that cache equivalence requires the effective prompt mask extended over the generated tokens, rather than an unconditional unmaskedrun_with_cache(output)call. No dependency changes are needed.Regression coverage
New offline tests use real tiny GPT-2 weights from config, without Hub downloads. They cover left-padding and interior-gap masks, bool/int masks, all-ones and absent-mask controls, both KV-cache modes, EOS-shortened sequences, output logits, return-input-token tuples, and an inferred left-padding mask with filtered CPU cache output. Assertions compare every cached tensor with an independently masked recomputation, compare cache logits with each generation step's logits, check excluded-key attention is zero, and verify the caller's mask is not mutated.
Validation
test_generate_cache_attention_mask.py, the existing return-cache and generation attention-mask integration files, and the generation KV-cache unit tests.git diff --checkpassed.Type of change
Checklist
Documentation changes are in the
generateAPI docstring. The full-unit-suite checkbox remains unchecked because only the affected test surface was run.