← All notes

Decoupled Weight Decay Regularization

Summary

Decoupled Weight Decay Regularization (2017) showed that L2 regularization and weight decay are not the same thing for adaptive methods such as Adam, even though they coincide for ordinary SGD once the learning rate is taken into account. The authors propose decoupling weight decay from the adaptive update. The resulting optimizer, AdamW, is now the default for almost all large-scale network training.

What it introduced: a corrected implementation of weight decay for Adam-style optimizers, in which decay is applied directly to the parameters instead of being folded into the gradient as an L2 term.

What it led to: AdamW as the usual training default in language, vision, and multimodal models. If a later paper says it trained with AdamW, this is the reference for what that change was meant to fix.

Use it when you want the paper that defined AdamW. Where to read it: https://arxiv.org/abs/1711.05101 (v3 (4 Jan 2019)).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Ilya Loshchilov, Frank Hutter, Decoupled Weight Decay Regularization (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:zey5ysyh

'iinventedlight (AI-assisted)'. 2026. "Decoupled Weight Decay Regularization." iinventedlight Library, iil:note:zey5ysyh, version of 2026-11-12. https://iinventedlight.com/library/notes/1711-05101-adamw/. License: CC-BY-4.0.

@misc{iil_note_zey5ysyh,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Decoupled Weight Decay Regularization}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/1711-05101-adamw/},
  note = {Permanent ID iil:note:zey5ysyh. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1711-05101-adamw.md.