Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Summary

Bai and coauthors finetune language models into helpful and harmless assistants with preference models and reinforcement learning from human feedback. The abstract says this alignment training improved performance on almost all of the NLP evaluations they ran, and that it could sit alongside specialized training such as Python coding and summarization. They also describe an online loop that refreshes the preference models and the RL policies on a weekly cadence with new human feedback, which they say improves the datasets and the models. On robustness, the abstract reports a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its starting point. The same page mentions side analyses of calibration, competing objectives, out-of-distribution detection, a comparison with human writers, and sample outputs on prompts from related work. The abs page lists no journal. This note follows that abstract page only. It is one published training setup. It is not a claim that a deployed assistant is safe.

When to apply

The question is how a published assistant was trained from human preference labels, including a repeating loop of fresh feedback.
The question is which written rules to put in a prompt today. This paper is a training method, not a prompt sheet.

Costs and limits

"Almost all NLP evaluations" and the weekly cadence are the authors' report of their own runs. The abstract gives no table of scores. This note does not invent those scores, and the PDF was not re-read.
Comments say data are available at a URL. That URL's license was not checked, so it is not a source here.

Evidence (detail)

The abs page (v1, 12 Apr 2022) says preference modeling and RLHF are used to finetune language models as helpful and harmless assistants (Abstract).
Reported fit on the abstract: the alignment training improved almost all NLP evaluations they considered, and it was compatible with specialized skills such as Python coding and summarization (Abstract).
Online mode named on the abstract: preference models and RL policies updated on a weekly cadence with fresh human feedback (Abstract).
Robustness claim on the abstract: a roughly linear relation between the RL reward and the square root of the KL divergence from the initialization (Abstract).
Comments line: "Data available at this https URL". No journal reference on the page.

Facts

id: F1 claim: The paper finetunes language models as helpful and harmless assistants with preference models and reinforcement learning from human feedback, and describes a loop that updates those models on a weekly cadence with new human feedback. confidence: medium date: 2026-10-18 sources: [S1 Abstract] license: CC BY 4.0
id: F2 claim: The abstract reports that this alignment training improved almost all of their NLP evaluations, stayed compatible with specialized training such as Python coding and summarization, and that RL reward was roughly linear in the square root of the KL divergence from the initial policy. confidence: medium date: 2026-10-18 sources: [S1 Abstract] license: CC BY 4.0
id: F3 claim: The arXiv abs page shows CC BY 4.0 on v1 (12 Apr 2022) and lists no journal or conference. confidence: high date: 2026-10-18 sources: [S1 abs page] license: CC BY 4.0

2212.08073 Constitutional AI trains with model-written critiques against a written list of principles. This paper uses human preference labels.
The library miss is how to try making an assistant. This note covers one published human-feedback training setup. It does not say an assistant is ready to deploy.

Review notes

Staging draft. Paraphrase of the arXiv abstract page checked 2026-10-18. The PDF was not re-read.
License on the abs page: icon by-4.0.png, link text "view license", http://creativecommons.org/licenses/by/4.0/.

Source: https://arxiv.org/abs/2204.05862 · License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan, \"Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback\" (CC BY 4.0).

Cite this

Permanent ID: iil:note:fzwrv2bj

Also known as: 2204.05862-helpful-harmless-rlhf

iinventedlight Library. 2026. "Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback." iinventedlight Library, iil:note:fzwrv2bj, version of 2026-10-18. https://iinventedlight.com/library/reference/arxiv/2204-05862-helpful-harmless-rlhf/. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2204.05862.

@misc{iil_note_fzwrv2bj,
  author = {{iinventedlight Library}},
  title = {{Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-10-18},
  url = {https://iinventedlight.com/library/reference/arxiv/2204-05862-helpful-harmless-rlhf/},
  note = {Permanent ID iil:note:fzwrv2bj. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2204.05862.},
}

For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2204-05862-helpful-harmless-rlhf.md.