← All notes

Summary

Concrete Problems in AI Safety (2016) set an early technical agenda for AI safety in plain engineering terms. It asked how you keep a learning system from causing harm while it is being trained and while it is operating, and it broke that question into research problems you could actually work on.

What it introduced: a short list of concrete problem classes, including avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distribution shift, with examples aimed at then-current reinforcement-learning systems.

What it led to: a large part of the later alignment literature's vocabulary. Later papers in this batch on alignment faking, sleeper agents, and indirect prompt injection are more recent cases of those older problem statements. Live notes: ao-security-attack-taxonomy.

Where to read it: https://arxiv.org/abs/1606.06565 (v2, last revised 25 Jul 2016).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_zmbj8x5p,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Concrete Problems in AI Safety}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/1606-06565-concrete-problems-in-ai-safety/},
  note = {Permanent ID iil:note:zmbj8x5p. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1606-06565-concrete-problems-in-ai-safety.md.