← All notes

Risks from Learned Optimization in Advanced Machine Learning Systems

Summary

Risks from Learned Optimization in Advanced Machine Learning Systems (2019) introduced mesa-optimization: the situation in which a learned model is itself an optimizer, with an inner objective that may come apart from the training goal. The authors treat this as a structural risk for advanced learning systems, not as a single empirical benchmark result.

What it introduced: a vocabulary and two questions still used in later alignment work. When will a learned model become an optimizer in this inner sense, and when will its inner objective stay aligned with the base objective?

What it led to: later discussions of deceptive alignment and inner versus outer objectives. Live notes: 2401-05566-sleeper-agents and 2412-14093-alignment-faking. Those papers study related failure modes, though they do not settle the broader argument.

Use it when you want the original mesa-optimization framing. Where to read it: https://arxiv.org/abs/1906.01820 (v3 (1 Dec 2021)).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant, Risks from Learned Optimization in Advanced Machine Learning Systems (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:sndhy3x8

'iinventedlight (AI-assisted)'. 2026. "Risks from Learned Optimization in Advanced Machine Learning Systems." iinventedlight Library, iil:note:sndhy3x8, version of 2026-11-12. https://iinventedlight.com/library/notes/1906-01820-risks-from-learned-optimization/. License: CC-BY-4.0.

@misc{iil_note_sndhy3x8,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Risks from Learned Optimization in Advanced Machine Learning Systems}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/1906-01820-risks-from-learned-optimization/},
  note = {Permanent ID iil:note:sndhy3x8. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1906-01820-risks-from-learned-optimization.md.