You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: kv_cache_cpu patch reads retired DynamicCache attributes, AttributeError on current transformers #217
minference_patch_kv_cache_cpu (patch.py) monkeypatches transformers.cache_utils.DynamicCache.update/.get with cpu_cache_update/cpu_cache_get. Both read and write self.key_cache, self.value_cache and self._seen_tokens/self.seen_tokens directly. Current transformers moved DynamicCache's per-layer state onto self.layers and no longer defines those attributes, so the first cache update after kv_cache_cpu=True raises AttributeError.
Same root cause as #207 (fixed for DynamicCacheWithRepeat in #216), but a different call site that fix didn't touch. kv_cache_cpu is a documented MInferenceConfig option used across the example/benchmark scripts, not a corner case.
Steps to reproduce
No vllm or model load needed, this extracts the two patched functions and runs them against a real DynamicCache:
Activity