Constitutional AI: Harmlessness from AI Feedback
License: CC BY 4.0. Source: https://arxiv.org/abs/2212.08073
Summary
Bai and coauthors try to train a less harmful assistant without a pile of human labels that mark outputs as harmful. People supply a written list of principles, which they call a constitution. In a supervised stage the model critiques and rewrites its own replies against those principles, and the original model is finetuned on the rewrites. In a second stage a model judges which of two replies is better, those judgments train a preference model, and reinforcement learning uses that model as the reward. They call the second stage reinforcement learning from AI feedback. They report an assistant that refuses harmful requests by explaining the objection rather than by going silent, and they say chain-of-thought critiques made the decisions easier for people to judge. This note follows the abstract page only. It is not a claim that the constitution replaces human oversight.
Why it is here
The self-improvement and agent notes ask how a system can revise itself when people cannot label every case. Constitutional AI is one published answer: a fixed set of rules, model-written critiques, then preference training. The human role shrinks to the rules and the choice to train, which is also the risk.
Source: https://arxiv.org/abs/2212.08073 · License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan, Constitutional AI: Harmlessness from AI Feedback (CC BY 4.0).
Cite this
iinventedlight Library. 2026. "Constitutional AI: Harmlessness from AI Feedback." iinventedlight Library, iil:note:sn1a48nf, version of 2026-10-18. https://iinventedlight.com/library/reference/arxiv/2212-08073-constitutional-ai/. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2212.08073.
@misc{iil_note_sn1a48nf,
author = {{iinventedlight Library}},
title = {{Constitutional AI: Harmlessness from AI Feedback}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-10-18},
url = {https://iinventedlight.com/library/reference/arxiv/2212-08073-constitutional-ai/},
note = {Permanent ID iil:note:sn1a48nf. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2212.08073.},
}For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2212-08073-constitutional-ai.md.