← All notes

Summary

RETRO (Retrieval-Enhanced Transformer) conditions a language model on document chunks retrieved from a large corpus by local similarity with preceding tokens. With a 2 trillion token database, the authors report performance comparable to GPT-3 and Jurassic-1 on the Pile, despite using 25 times fewer parameters. RETRO combines a frozen Bert retriever, a differentiable encoder, and chunked cross-attention, and can also RETROfit pretrained transformers. [S1]

Claims

• id: C1 claim: 'RETRO retrieves document chunks from a large corpus based on local similarity with preceding tokens.' source: S1 evidence: "document chunks retrieved from a large corpus"
• id: C2 claim: 'With a 2 trillion token database, they report performance comparable to GPT-3 and Jurassic-1 on the Pile, using 25 times fewer parameters.' source: S1 evidence: "comparable performance to GPT-3 and Jurassic-1 on the Pile"
• id: C3 claim: 'RETRO combines a frozen Bert retriever, a differentiable encoder, and chunked cross-attention.' source: S1 evidence: "RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism"

Verification

• Code available: No official RETRO code URL was named as this paper's release on the abs page or HTML. Cited third-party tools such as FiD were not treated as this paper's code.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes 2310-11511-self-rag and genai-rag-embeddings-vector-search. This is the paper-verification note for RETRO itself.
• Limits: Comments note a Table 14 number fix; this note uses the abstract and key description. The retrieval database is not public here.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 8 Dec 2021 (v1); last revised 7 Feb 2022 (v3). Sources: [S1] https://arxiv.org/abs/2112.04426, v3 (7 Feb 2022). checked_on: 2026-11-11.

When to apply

Use when comparing retrieval at training and inference scale with later RAG methods, especially when retrieval is meant to substitute for some model size.

Facts

• id: F1 claim: 'RETRO conditions generation on document chunks retrieved from a large corpus.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors report that with a 2 trillion token database, RETRO matches GPT-3 and Jurassic-1 on the Pile with 25 times fewer parameters.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They describe both training RETRO from scratch and RETROfitting pretrained transformers with retrieval.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2112.04426, https://arxiv.org/abs/2112.04426. License: CC BY 4.0. v3 (7 Feb 2022). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_qc5e0pfx,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Improving language models by retrieving from trillions of tokens}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2112-04426-retro/},
  note = {Permanent ID iil:note:qc5e0pfx. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2112-04426-retro.md.