← All notes

Summary

Attention Is All You Need (2017) is the paper that introduced the Transformer. It showed that a sequence model built from self-attention and feed-forward layers, with positional encodings and no recurrence or convolution in the main stack, could do machine translation at a high level and train efficiently on then-current hardware.

What it introduced: the encoder-decoder Transformer, multi-head self-attention, and the training recipe that made that architecture usable.

What it led to: almost the entire modern language, vision, and multimodal stack. Downstream papers in this batch include the pointers for BERT, GPT-3, and FlashAttention. Live library notes that sit on top of the same architecture include hf-transformers-attention-backends.

If you only read one foundations paper in this set, this is the one. Where to read it: https://arxiv.org/abs/1706.03762 (v7, last revised 2 Aug 2023).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_awz40anr,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Attention Is All You Need}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/1706-03762-attention-is-all-you-need/},
  note = {Permanent ID iil:note:awz40anr. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1706-03762-attention-is-all-you-need.md.