---
title: 'Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-11; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2307-15217-rlhf-open-problems.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, et al. (32 authors), Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Not reproduced. Abs page and HTML key sections read 2026-11-11. Named code URLs checked; code not run.'
category: arxiv
topic_area: 'ai-research: alignment-and-safety'
tags: arxiv, alignment, rlhf
arxiv_id: 2307.15217
alias: 2307.15217-rlhf-open-problems
sources:
  - id: S1
    title: 'Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback'
    publisher: arXiv
    authors: 'Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, et al. (32 authors)'
    url: https://arxiv.org/abs/2307.15217
    version_read: 'v2 (11 Sep 2023)'
    accessed: 2026-11-11
    license: 'CC BY 4.0'
---
# Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

## Summary
This paper surveys open problems and fundamental limitations of reinforcement learning from human feedback (RLHF), the method widely used to finetune large language models. The authors also overview techniques that try to understand, improve, or complement RLHF, and they propose auditing and disclosure standards for societal oversight. The work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems. [S1]

## Claims
- id: C1
  claim: 'The paper surveys open problems and fundamental limitations of RLHF.'
  source: S1
  evidence: "survey open problems and fundamental limitations of RLHF and related methods"
- id: C2
  claim: 'RLHF is described as the central method used to finetune state-of-the-art LLMs.'
  source: S1
  evidence: "the central method used to finetune state-of-the-art large language models"
- id: C3
  claim: 'The authors argue for a multi-faceted approach to safer AI systems.'
  source: S1
  evidence: "highlights the importance of a multi-faceted approach to the development of safer AI systems"

## Verification
- Code available: No official code URL was named on the abs page or HTML.
- Reproduced: not reproduced (2026-11-11). Code not run.
- Related work: Live notes 2204-05862-helpful-harmless-rlhf and 2305-18290-direct-preference-optimization. This is the paper-verification note for the RLHF limitations survey itself.
- Limits: This is a survey of problems and proposals, not a new training method with its own scores. It does not by itself measure how often each listed failure occurs in current systems.
- License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 27 Jul 2023 (v1); last revised 11 Sep 2023 (v2). Sources: [S1] https://arxiv.org/abs/2307.15217, v2 (11 Sep 2023). checked_on: 2026-11-11.

## When to apply
Use before treating RLHF as a complete alignment solution, or when mapping known structural limits and proposed complements.

## Facts
- id: F1
  claim: 'The paper surveys open problems and fundamental limitations of RLHF and related methods.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: 'The authors describe RLHF as the central method used to finetune state-of-the-art large language models.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: 'They propose auditing and disclosure standards to improve societal oversight of RLHF systems.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2307.15217, https://arxiv.org/abs/2307.15217. License: CC BY 4.0. v2 (11 Sep 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).