← All notes

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

Summary

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019) introduced ZeRO, a set of memory optimizations that partition optimizer state, gradients, and parameters across data-parallel devices so that much larger models can be trained without a full copy of every state on every GPU.

What it introduced: a staged redundancy-elimination design for data-parallel training. Early stages shard optimizer state and gradients; later stages can also shard parameters, trading communication for a much smaller per-device memory footprint.

What it led to: DeepSpeed ZeRO and a default large-model training stack, often used together with model-parallel methods such as Megatron. See also the Megatron-LM pointer in this batch, and 2305-13245-gqa for a later inference-side memory theme.

Use it when you want the original sharded-optimizer design. Where to read it: https://arxiv.org/abs/1910.02054 (v3 (13 May 2020)).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He, ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:hs2mq0h0

'iinventedlight (AI-assisted)'. 2026. "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models." iinventedlight Library, iil:note:hs2mq0h0, version of 2026-11-12. https://iinventedlight.com/library/notes/1910-02054-zero/. License: CC-BY-4.0.

@misc{iil_note_hs2mq0h0,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{ZeRO: Memory Optimizations Toward Training Trillion Parameter Models}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/1910-02054-zero/},
  note = {Permanent ID iil:note:hs2mq0h0. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1910-02054-zero.md.