---
source: https://iinventedlight.com/library/notes/hf-transformers-kv-cache.md
title: "KV cache in Transformers: why it exists and which Cache class"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 (huggingface/transformers, huggingface_hub, diffusers docs; LICENSE verified in each repo); prose restates cited source lines in own words"
license_family: hf-apache
attribution: "Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs, How caching works, Apache-2.0 (huggingface docs repos); Hugging Face docs, Cache strategies, Apache-2.0 (huggingface docs repos)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No code was run for this note; only the fetched documentation pages were read."
category: coding-references
topic_area: "code-ai-eng: Hugging Face"
tags: [hugging-face, transformers, kv-cache, generation]
client: Librarian
sources:
  - id: S1
    title: "How caching works"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/cache_explanation
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
  - id: S2
    title: "Cache strategies"
    publisher: Hugging Face docs
    url: https://huggingface.co/docs/transformers/kv_cache
    accessed: 2026-11-09
    license: "Apache-2.0 (huggingface docs repos)"
---
# KV cache in Transformers: why it exists and which Cache class

**Answer:** A key-value (KV) cache stores kv pairs from the attention layers of previously processed tokens so autoregressive generation reuses them instead of recomputing.

## Why

Autoregressive generation predicts one token at a time, and each prediction depends on all the previous context. Once a token is processed, its keys and values never change with respect to future tokens, so they can be cached. At each step the new key and value vectors are appended to the past keys and values; caching is done per layer.

Caching should only be used for inference; enabled during training it may cause unexpected errors.

## Cache classes

| Cache | torch.compile | Memory |
|---|---|---|
| DynamicCache (default) | No | Medium |
| Static Cache | Yes | High |
| Quantized Cache | No | Low |

`DynamicCache` grows as generation progresses. A fixed-size `StaticCache` pre-allocates a maximum cache size, which enables JIT optimizations like torch.compile at the expense of memory. Disable the cache with `use_cache=False`; choose a strategy with `cache_implementation`.

## Artifacts

- [Hugging Face model load choices (table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.json)
- [Hugging Face model load choices (markdown table)](https://iinventedlight.com/library/artifacts/hf-load-choices.table.md)
- [Topic packet: huggingface](https://iinventedlight.com/library/packets/huggingface.json)

## See also

- [hf-text-generation-strategies](https://iinventedlight.com/library/notes/hf-text-generation-strategies/)
- [hf-transformers-generate-pitfalls](https://iinventedlight.com/library/notes/hf-transformers-generate-pitfalls/)

## Claims

- id: C1
  claim: The KV cache stores kv pairs of processed tokens.
  source: S1
  evidence: "storing kv pairs derived from the attention layers of previously processed tokens."

- id: C2
  claim: Caching is for inference only.
  source: S1
  evidence: "Caching should only be used for **inference**."

- id: C3
  claim: DynamicCache is the default cache class.
  source: S2
  evidence: "is the default cache class for all models."

- id: C4
  claim: A fixed-size cache pre-allocates its maximum size.
  source: S2
  evidence: "pre-allocates a specific maximum cache size for the kv pairs."

## Sources

- [S1] How caching works. Hugging Face docs. https://huggingface.co/docs/transformers/cache_explanation . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

- [S2] Cache strategies. Hugging Face docs. https://huggingface.co/docs/transformers/kv_cache . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).