← All notes

Summary

Training language models to follow instructions with human feedback (2022) is the InstructGPT paper. It described how to turn a raw pretrained language model into an assistant that follows user instructions, using human preference data and reinforcement learning from those preferences.

What it introduced: a three-stage recipe of supervised fine-tuning, a reward model trained on human comparisons, and a reinforcement-learning update (PPO) of the policy. The resulting InstructGPT models were smaller than GPT-3 but preferred by labelers on instruction-following.

What it led to: the default post-training path for chat models, later variants such as helpful-and-harmless RLHF and direct preference optimization, and today's reasoning-model training. Live notes: 2204-05862-helpful-harmless-rlhf and 2305-18290-direct-preference-optimization. This week's batch also has pointers for PPO and the earlier deep reinforcement learning from human preferences paper.

Where to read it: https://arxiv.org/abs/2203.02155 (v1, 4 Mar 2022).

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_b9cbwdvh,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Training language models to follow instructions with human feedback}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2203-02155-instructgpt/},
  note = {Permanent ID iil:note:b9cbwdvh. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2203-02155-instructgpt.md.