← All notes

Answer: A key-value (KV) cache stores kv pairs from the attention layers of previously processed tokens so autoregressive generation reuses them instead of recomputing.

Why

Autoregressive generation predicts one token at a time, and each prediction depends on all the previous context. Once a token is processed, its keys and values never change with respect to future tokens, so they can be cached. At each step the new key and value vectors are appended to the past keys and values; caching is done per layer.

Caching should only be used for inference; enabled during training it may cause unexpected errors.

Cache classes

| Cache | torch.compile | Memory | |---|---|---| | DynamicCache (default) | No | Medium | | Static Cache | Yes | High | | Quantized Cache | No | Low |

DynamicCache grows as generation progresses. A fixed-size StaticCache pre-allocates a maximum cache size, which enables JIT optimizations like torch.compile at the expense of memory. Disable the cache with use_cache=False; choose a strategy with cache_implementation.

Artifacts

• Hugging Face model load choices (table)
• Hugging Face model load choices (markdown table)
• Topic packet: huggingface

See also

• hf-text-generation-strategies
• hf-transformers-generate-pitfalls

Claims

• id: C1 claim: The KV cache stores kv pairs of processed tokens. source: S1 evidence: "storing kv pairs derived from the attention layers of previously processed tokens."

• id: C2 claim: Caching is for inference only. source: S1 evidence: "Caching should only be used for inference."

• id: C3 claim: DynamicCache is the default cache class. source: S2 evidence: "is the default cache class for all models."

• id: C4 claim: A fixed-size cache pre-allocates its maximum size. source: S2 evidence: "pre-allocates a specific maximum cache size for the kv pairs."

Sources

• [S1] How caching works. Hugging Face docs. https://huggingface.co/docs/transformers/cache_explanation . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] Cache strategies. Hugging Face docs. https://huggingface.co/docs/transformers/kv_cache . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_ac63ndsk,
  author = {{iinventedlight}},
  title = {{KV cache in Transformers: why it exists and which Cache class}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-transformers-kv-cache/},
  note = {Permanent ID iil:note:ac63ndsk. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-transformers-kv-cache.md.