← All notes

Short answer

If you keep only a sliding window of recent key/value (KV) cache entries, model quality collapses once the text grows past the cache. Keeping the KV entries of the first few tokens (the attention sinks) as well restores it. StreamingLLM uses this to let models stream very long inputs with no finetuning. [S1]

Key points

• Two problems in long chats: the KV cache grows with every token, and models do not generalize past their training length. [S1]
• Attention sink: models put strong attention on the initial tokens even when those tokens carry little meaning. Dropping them breaks window attention. [S1]
• StreamingLLM: keep the sink tokens plus a recent window. The authors report stable modeling up to 4 million tokens on Llama-2, MPT, Falcon and Pythia. [S1]
• Speed: up to 22.2x faster than re-computing a sliding window. [S1]
• Pre-training tip: adding a dedicated placeholder sink token during pre-training helps streaming further. [S1]

Caveat

This does not extend what the model can remember. Tokens that fall out of the window are gone. It keeps generation stable over long streams.

Claims

• id: C1 claim: Caching previous tokens' KV states consumes extensive memory during decoding. source: S1 evidence: "caching previous tokens' Key and Value states (KV) consumes extensive memory"
• id: C2 claim: Window attention fails when text length surpasses the cache size. source: S1 evidence: "it fails when the text length surpasses the cache size"
• id: C3 claim: Keeping the KV of initial tokens largely recovers window-attention performance. source: S1 evidence: "keeping the KV of initial tokens will largely recover the performance"
• id: C4 claim: StreamingLLM outperforms sliding window recomputation by up to 22.2x speedup. source: S1 evidence: "up to 22.2x speedup"

Sources

• [S1] Efficient Streaming Language Models with Attention Sinks. arXiv. https://arxiv.org/abs/2309.17453 . License: CC BY 4.0. Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_9mw68fqs,
  author = {{iinventedlight}},
  title = {{Efficient Streaming Language Models with Attention Sinks (StreamingLLM)}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/2309-17453-streamingllm-attention-sinks/},
  note = {Permanent ID iil:note:9mw68fqs. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/2309-17453-streamingllm-attention-sinks.md.