---
title: 'Concrete Problems in AI Safety'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is arXiv non-exclusive distribution license (verified on arXiv abs page 2026-11-11); this pointer note is our own words and is CC BY 4.0; we link to the paper rather than reproduce it'
source: https://iinventedlight.com/library/notes/1606-06565-concrete-problems-in-ai-safety.md
source_license: 'arXiv non-exclusive distribution license'
attribution: 'Pointer by iinventedlight (AI-assisted). Source: Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, Concrete Problems in AI Safety (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Paper not reproduced. Abs page and license checked 2026-11-11.'
category: arxiv
topic_area: 'ai-research: alignment-and-safety'
tags: arxiv, alignment, safety, foundations
arxiv_id: 1606.06565
alias: 1606.06565-concrete-problems-in-ai-safety
type: pointer
format: pointer
sources:
  - id: S1
    title: 'Concrete Problems in AI Safety'
    publisher: arXiv
    authors: 'Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané'
    url: https://arxiv.org/abs/1606.06565
    version_read: 'v2 (25 Jul 2016)'
    accessed: 2026-11-11
    license: 'arXiv non-exclusive distribution license'
---
# Concrete Problems in AI Safety

## Summary
[Concrete Problems in AI Safety](https://arxiv.org/abs/1606.06565) (2016) set an early technical agenda for AI safety in plain engineering terms. It asked how you keep a learning system from causing harm while it is being trained and while it is operating, and it broke that question into research problems you could actually work on.

What it introduced: a short list of concrete problem classes, including avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distribution shift, with examples aimed at then-current reinforcement-learning systems.

What it led to: a large part of the later alignment literature's vocabulary. Later papers in this batch on alignment faking, sleeper agents, and indirect prompt injection are more recent cases of those older problem statements. Live notes: [ao-security-attack-taxonomy](https://iinventedlight.com/library/notes/ao-security-attack-taxonomy/).

Where to read it: https://arxiv.org/abs/1606.06565 (v2, last revised 25 Jul 2016).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).