You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On dev (1658f88), the default flash attention path reserves full-context BF16 KV/K/V storage for local and global layers alike. Local layers only attend to their sliding window, so retaining capacity for the entire context wastes memory as the configured context grows.
The runtime-aware cache constructor also allocates compact tiled buffers that the default Flash path does not use. The tiled attention backends already have compact local rings; the default Flash path still reads the legacy transposed K/V buffers.
Fix direction
Allocate separate BF16 buffers per owning layer for the default Flash path.
Size local rings for the attention window plus the full prefill batch and trailing SIMD padding, capped by the configured context. The extra rows prevent batch writes and padding from overwriting history needed by early queries.
Keep global layers at the configured context capacity, with the logical context limit separate from physical alignment padding.
Reuse the source layer's buffers for shared-KV layers and retain the largest window required by their consumers.
Preserve live history when runtime batches require larger rings, keep snapshots independent, and clear buffers safely when reusing a cache.
Allocate only the buffers used by the selected attention backend and preserve the existing BF16 attention arithmetic and model-specific cache layouts.
Thanks for pointing to this - indeed useful to reduce KV size.
Further to the comments in #1020, let's also consider whether tiled attention can replace/supersede flash. We're doing some experiments in this direction. tiled_attention already uses a ring buffer, hence there is less room for improvement there.
Activity
[-]Reduce KV memory for local-attention layers[/-][+]Reduce KV memory for local-attention layers on dev[/+][-]Reduce KV memory for local-attention layers on dev[/-][+]Reduce KV memory for local-attention layers[/+]jan-wassenberg commentedon Sep 8, 2026
Thanks for pointing to this - indeed useful to reduce KV size.
Further to the comments in #1020, let's also consider whether tiled attention can replace/supersede flash. We're doing some experiments in this direction. tiled_attention already uses a ring buffer, hence there is less room for improvement there.