---
title: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://arxiv.org/abs/2305.18290
source_license: "CC BY 4.0"
attribution: "Summary by iinventedlight (AI-assisted). Source: arXiv, \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (CC BY 4.0)."
condensed_by: "iinventedlight (Grok Bot, AI-assisted)"
fact_checked: 2026-11-02
date: 2026-11-02
status: reviewed
reviewed_on: 2026-11-02
category: arxiv
arxiv_id: 2305-18290-direct-preference-optimization
alias: 2305-18290-direct-preference-optimization
checked_on: 2026-11-02
license_url: https://creativecommons.org/licenses/by/4.0/
source: https://arxiv.org/abs/2305.18290
source_license: "CC BY 4.0 (arXiv paper)"
arxiv_license_url: https://creativecommons.org/licenses/by/4.0/
arxiv_license_checked: 2026-11-02
arxiv_license_check_note: "License link read from the paper\'s own arXiv abstract page on 2026-11-02."
summary_policy: "Short original summary with attribution; no passages copied, at most brief phrases quoted."
attribution: "Summary by iinventedlight (AI-assisted), condensed and paraphrased; sources: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\", arXiv:2305.18290 (CC BY 4.0)"
condensed_by: "iinventedlight (Grok Bot, AI-assisted)"
fact_checked: 2026-11-02
date: 2026-11-02
category: arxiv
tags: arxiv, llm, paper-summary
checked_on: 2026-11-02
sources:
  - id: S1
    title: "Direct Preference Optimization: Your Language Model is Secretly a Reward Model"
    authors: "Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn"
    publisher: "arXiv"
    url: https://arxiv.org/abs/2305.18290
    accessed: 2026-11-02
    license: "CC BY 4.0"
    attribution: "Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn, \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\", arXiv:2305.18290 (CC BY 4.0)"
---
# Direct Preference Optimization: Your Language Model is Secretly a Reward Model

# Direct Preference Optimization: Your Language Model is Secretly a Reward Model

## Summary
DPO aligns a language model to human preferences without fitting a separate reward model and running RL. It reparameterizes the RLHF objective so the best policy can be found in closed form, which reduces training to a simple classification loss on preference pairs. The authors report that it matches or beats PPO-based RLHF while being simpler and more stable.

*Draft. License: CC BY 4.0 (this summary). Sources: [S1] arXiv, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (CC BY 4.0).*

## When to apply
- Fine-tuning on pairwise preference data (chosen vs rejected responses) when full RLHF is too costly.

## Paper
- Title: Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Authors: Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn
- arXiv: https://arxiv.org/abs/2305.18290 (first posted 2023-05-29)
- License: CC BY 4.0 (checked on the arXiv abstract page 2026-11-02)
- Read for this note: arXiv abstract page; figures restate the abstract.

## Related
- 2204.05862 Helpful and Harmless RLHF (published)

## Facts
- id: F1
  claim: Standard RLHF first fits a reward model to human preference labels, then fine-tunes the LM with RL while keeping it close to the original model.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F2
  claim: DPO uses a new reward-model parameterization so the optimal policy can be extracted in closed form, solving the RLHF problem with a simple classification loss.
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F3
  claim: DPO avoids sampling from the LM during fine-tuning and needs little hyperparameter tuning (author-reported).
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02
- id: F4
  claim: DPO exceeded PPO-based RLHF at controlling sentiment and matched or improved response quality on summarization and single-turn dialogue (author-reported).
  confidence: medium
  date: 2026-11-02
  sources: [S1]
  license: CC BY 4.0
  checked_on: 2026-11-02

## Caveats
- Results are on 2023-scale models and tasks.
- Quality depends on the preference data.

## Pointers
- id: P1
  answers: Full method, experiments and appendices
  where: https://arxiv.org/abs/2305.18290
  access: free, public web page