← All notes

Answer: All attention implementations perform the same computation; backends differ in how they do it. Use the attn_implementation argument in from_pretrained to set an attention function.

Why backends matter

Basic attention materializes the full attention matrix in memory, which slows down inference. Optimized implementations rearrange the math to reduce memory traffic.

Options

• sdpa: the built-in PyTorch implementation of scaled dot product attention; the default when no backend is named.
• flash_attention_2: tiles computations into smaller blocks and uses fast on-chip memory.
• flash_attention_3: improves FlashAttention-2 by overlapping operations.
• flex_attention: custom patterns (sparse, block-local, sliding window) without writing low-level kernels.
• paged|sdpa, paged|eager and paged FlashAttention: paged versions of SDPA, eager, and FlashAttention.

Switching and kernels

Switch backends at runtime without reloading the model using set_attn_implementation. The Kernels library downloads compiled kernels from the Hub at runtime, avoiding mismatched PyTorch or CUDA versions.

Multimodal models can map each backbone to a different attention function with a dict keyed by sub-config names.

Artifacts

• Hugging Face model load choices (table)
• Hugging Face model load choices (markdown table)
• Topic packet: huggingface

See also

• hf-transformers-kv-cache

Claims

• id: C1 claim: All attention implementations compute the same thing. source: S1 evidence: "All attention implementations perform the same computation."

• id: C2 claim: Backends can be switched at runtime without reloading. source: S1 evidence: "Switch between attention backends at runtime without reloading the model"

• id: C3 claim: Kernels avoid mismatched PyTorch or CUDA versions. source: S1 evidence: "This avoids packaging issues from mismatched PyTorch or CUDA versions."

• id: C4 claim: Attention is computed per layer. source: S2 evidence: "Attention is calculated independently in each layer of the model"

Sources

• [S1] Attention backends. Hugging Face docs. https://huggingface.co/docs/transformers/attention_interface . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

• [S2] How caching works. Hugging Face docs. https://huggingface.co/docs/transformers/cache_explanation . License: Apache-2.0 (huggingface docs repos). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: Hugging Face docs (huggingface/transformers, huggingface/huggingface_hub, huggingface/diffusers on GitHub), Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). Text was condensed and changed from the originals.

@misc{iil_note_v4cdvdhd,
  author = {{iinventedlight}},
  title = {{Attention backends: sdpa, FlashAttention, flex, paged}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/hf-transformers-attention-backends/},
  note = {Permanent ID iil:note:v4cdvdhd. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/hf-transformers-attention-backends.md.