---
source: https://iinventedlight.com/library/notes/2309-17453-streamingllm-attention-sinks.md
title: "Efficient Streaming Language Models with Attention Sinks (StreamingLLM)"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, Efficient Streaming Language Models with Attention Sinks, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-05
as_of: 2026-11-05
confidence: medium
untested: "Results were not reproduced; only the arXiv abstract page was read."
category: ai-research
topic_area: "ai-research: long-context inference"
tags: llm, attention, kv-cache, streaming, arxiv
sources:
  - id: S1
    title: "Efficient Streaming Language Models with Attention Sinks"
    publisher: arXiv
    url: https://arxiv.org/abs/2309.17453
    accessed: 2026-11-05
    license: "CC BY 4.0"
---
# Efficient Streaming Language Models with Attention Sinks (StreamingLLM)

## Short answer
If you keep only a sliding window of recent key/value (KV) cache entries, model quality collapses once the text grows past the cache. Keeping the KV entries of the first few tokens (the attention sinks) as well restores it. StreamingLLM uses this to let models stream very long inputs with no finetuning. [S1]

## Key points
- **Two problems in long chats:** the KV cache grows with every token, and models do not generalize past their training length. [S1]
- **Attention sink:** models put strong attention on the initial tokens even when those tokens carry little meaning. Dropping them breaks window attention. [S1]
- **StreamingLLM:** keep the sink tokens plus a recent window. The authors report stable modeling up to 4 million tokens on Llama-2, MPT, Falcon and Pythia. [S1]
- **Speed:** up to 22.2x faster than re-computing a sliding window. [S1]
- **Pre-training tip:** adding a dedicated placeholder sink token during pre-training helps streaming further. [S1]

## Caveat
This does not extend what the model can remember. Tokens that fall out of the window are gone. It keeps generation stable over long streams.

## Claims
- id: C1
  claim: Caching previous tokens' KV states consumes extensive memory during decoding.
  source: S1
  evidence: "caching previous tokens' Key and Value states (KV) consumes extensive memory"
- id: C2
  claim: Window attention fails when text length surpasses the cache size.
  source: S1
  evidence: "it fails when the text length surpasses the cache size"
- id: C3
  claim: Keeping the KV of initial tokens largely recovers window-attention performance.
  source: S1
  evidence: "keeping the KV of initial tokens will largely recover the performance"
- id: C4
  claim: StreamingLLM outperforms sliding window recomputation by up to 22.2x speedup.
  source: S1
  evidence: "up to 22.2x speedup"

## Sources
- [S1] Efficient Streaming Language Models with Attention Sinks. arXiv. https://arxiv.org/abs/2309.17453 . License: CC BY 4.0. Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).