← All notes

Summary

This paper surveys open problems and fundamental limitations of reinforcement learning from human feedback (RLHF), the method widely used to finetune large language models. The authors also overview techniques that try to understand, improve, or complement RLHF, and they propose auditing and disclosure standards for societal oversight. The work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems. [S1]

Claims

• id: C1 claim: 'The paper surveys open problems and fundamental limitations of RLHF.' source: S1 evidence: "survey open problems and fundamental limitations of RLHF and related methods"
• id: C2 claim: 'RLHF is described as the central method used to finetune state-of-the-art LLMs.' source: S1 evidence: "the central method used to finetune state-of-the-art large language models"
• id: C3 claim: 'The authors argue for a multi-faceted approach to safer AI systems.' source: S1 evidence: "highlights the importance of a multi-faceted approach to the development of safer AI systems"

Verification

• Code available: No official code URL was named on the abs page or HTML.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes 2204-05862-helpful-harmless-rlhf and 2305-18290-direct-preference-optimization. This is the paper-verification note for the RLHF limitations survey itself.
• Limits: This is a survey of problems and proposals, not a new training method with its own scores. It does not by itself measure how often each listed failure occurs in current systems.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 27 Jul 2023 (v1); last revised 11 Sep 2023 (v2). Sources: [S1] https://arxiv.org/abs/2307.15217, v2 (11 Sep 2023). checked_on: 2026-11-11.

When to apply

Use before treating RLHF as a complete alignment solution, or when mapping known structural limits and proposed complements.

Facts

• id: F1 claim: 'The paper surveys open problems and fundamental limitations of RLHF and related methods.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors describe RLHF as the central method used to finetune state-of-the-art large language models.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They propose auditing and disclosure standards to improve societal oversight of RLHF systems.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2307.15217, https://arxiv.org/abs/2307.15217. License: CC BY 4.0. v2 (11 Sep 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_45vcchw5,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2307-15217-rlhf-open-problems/},
  note = {Permanent ID iil:note:45vcchw5. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2307-15217-rlhf-open-problems.md.