← Reference notes

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Summary

DPO aligns a language model to human preferences without fitting a separate reward model and running RL. It reparameterizes the RLHF objective so the best policy can be found in closed form, which reduces training to a simple classification loss on preference pairs. The authors report that it matches or beats PPO-based RLHF while being simpler and more stable.

Draft. License: CC BY 4.0 (this summary). Sources: [S1] arXiv, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (CC BY 4.0).

When to apply

• Fine-tuning on pairwise preference data (chosen vs rejected responses) when full RLHF is too costly.

Paper

• Title: Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Authors: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn
• arXiv: https://arxiv.org/abs/2305.18290 (first posted 2023-05-29)
• License: CC BY 4.0 (checked on the arXiv abstract page 2026-11-02)
• Read for this note: arXiv abstract page; figures restate the abstract.

Related

• 2204.05862 Helpful and Harmless RLHF (published)

Facts

• id: F1 claim: Standard RLHF first fits a reward model to human preference labels, then fine-tunes the LM with RL while keeping it close to the original model. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
• id: F2 claim: DPO uses a new reward-model parameterization so the optimal policy can be extracted in closed form, solving the RLHF problem with a simple classification loss. confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
• id: F3 claim: DPO avoids sampling from the LM during fine-tuning and needs little hyperparameter tuning (author-reported). confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02
• id: F4 claim: DPO exceeded PPO-based RLHF at controlling sentiment and matched or improved response quality on summarization and single-turn dialogue (author-reported). confidence: medium date: 2026-11-02 sources: [S1] license: CC BY 4.0 checked_on: 2026-11-02

Caveats

• Results are on 2023-scale models and tasks.
• Quality depends on the preference data.

Pointers

• id: P1 answers: Full method, experiments and appendices where: https://arxiv.org/abs/2305.18290 access: free, public web page

@misc{iil_note_p5bhvyhx,
  author = {{iinventedlight Library}},
  title = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-02},
  url = {https://iinventedlight.com/library/reference/arxiv/2305-18290-direct-preference-optimization/},
  note = {Permanent ID iil:note:p5bhvyhx. License: CC-BY-4.0. Summary of https://arxiv.org/abs/2305.18290.},
}

For language models and agents: start at /llms.txt. This page in Markdown: /library/reference/arxiv/2305-18290-direct-preference-optimization.md.