Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Summary
Scaling Language Models: Methods, Analysis & Insights from Training Gopher (2021) is DeepMind's study of Transformer language models trained across a wide range of scales, up to a 280 billion parameter model named Gopher. It became a main reference for later scale-and-data work at the same lab.
What it introduced: a carefully documented 280B dense language model, plus analysis of how quality and task results change as scale grows from tens of millions of parameters up to Gopher.
What it led to: the compute-optimal training arguments in Chinchilla, and later Gemini-family reports. Live notes: 2203-15556-training-compute-optimal-llms. This week's batch also has 2312-11805-gemini.
Use it when you want the original Gopher scaling study, not only the later compute-optimal follow-up. Where to read it: https://arxiv.org/abs/2112.11446 (v2 (21 Jan 2022)).
The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Jack W. Rae and 79 coauthors (listed on the arXiv abs page), Scaling Language Models: Methods, Analysis & Insights from Training Gopher (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "Scaling Language Models: Methods, Analysis & Insights from Training Gopher." iinventedlight Library, iil:note:zw5fax67, version of 2026-11-12. https://iinventedlight.com/library/notes/2112-11446-gopher/. License: CC-BY-4.0.
@misc{iil_note_zw5fax67,
author = {{'iinventedlight (AI-assisted)'}},
title = {{Scaling Language Models: Methods, Analysis \& Insights from Training Gopher}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2112-11446-gopher/},
note = {Permanent ID iil:note:zw5fax67. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2112-11446-gopher.md.