You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Default flash attention retains redundant BF16 KV projection history #1029
Default flash attention retains two BF16 representations of KV history.
The sequence-major projection buffer remains allocated and populated after
its keys and values have been transposed into the buffers attention reads.
This increases resident memory as context length grows.
Affected path
Backend: default --attention_impl flash.
Confirmed with Gemma 3 270M, 1B, and 4B text inference.
Run default-flash Gemma 3 270M with a 32,736-token text prompt,
sequence capacity 32,768, prefill batch 4,096, and 16 decode tokens.
Inspect the allocated cache extents and peak process RSS during inference.
The projection buffer occupies 578 MiB including row padding.
The transposed K/V buffers occupy another 576 MiB.
Measured peak process RSS is 1,778.59 MiB on this workload.
This is active retained data, not merely unused virtual address space.
Environment: Linux, Intel i5-12400F, six pinned threads,
Release AVX2/Haswell build, no oneDNN, and approximately 15.5 GiB RAM.
Expected behavior
Projection intermediates should not retain a second complete KV history
after attention's persistent representation has been produced.
Required context, existing BF16 rounding, and model behavior must be preserved.
Activity