← All notes

Summary

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) improved the original FlashAttention kernel with better parallelism and work partitioning. It became the default attention kernel in many modern training stacks.

What it introduced: a second-generation exact-attention kernel that reduces unused compute and improves scaling across thread blocks, with measured speedups over the first FlashAttention.

What it led to: widespread adoption in training frameworks and in libraries that expose attention backends. Live notes: hf-transformers-attention-backends. The original FlashAttention paper is a pointer in batch B.

If you are choosing an attention kernel in a current training stack, this is the usual successor to the first FlashAttention paper, and the one most current libraries default to. Where to read it: https://arxiv.org/abs/2307.08691 (v1, 17 Jul 2023).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_xg7n6vzt,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2307-08691-flashattention-2/},
  note = {Permanent ID iil:note:xg7n6vzt. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2307-08691-flashattention-2.md.