← All notes

Summary

Judging LLM as a Judge with MT Bench and Chatbot Arena (2023), by Lianmin Zheng, Wei Lin Chiang, Ying Sheng, and colleagues, is at https://arxiv.org/abs/2306.05685. The authors examine limits of model judges, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and they propose ways to reduce some of those problems. They then compare model judges with human preference using MT-bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform. They report that strong judges such as GPT-4 can match both controlled and crowdsourced human preferences well, reaching over 80% agreement, comparable to agreement between humans. They also show that these benchmarks complement traditional tests, and they release MT-bench questions, 3K expert votes, and 30K conversations at the FastChat llm_judge path. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v4 (24 Dec 2023), submitted 9 Jun 2023.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it tested whether a strong language model can stand in for human preference judgments, and it introduced two evaluation resources that are still widely used: MT-Bench and Chatbot Arena.
• Reproduced: not reproduced (2026-11-11).
• Related work: A related note already in the library is RAGAS.
• Limits: This is a pointer note only. The over 80% agreement figure is the authors' reported result from the v4 paper (24 Dec 2023).
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 9 Jun 2023 (v1); last revised 24 Dec 2023 (v4). Sources: [S1] https://arxiv.org/abs/2306.05685, v4 (24 Dec 2023). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original LLM-as-a-judge paper, including MT-Bench and Chatbot Arena as human-preference references.

Facts

• id: F1 claim: 'The paper introduces MT-bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform, to study LLM-as-a-judge against human preference.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors report that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They say MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available in the FastChat repository.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2306.05685, https://arxiv.org/abs/2306.05685. Paper license: arXiv default (non-exclusive distribution license). v4 (24 Dec 2023). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_n4fcvp7s,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2306-05685-mt-bench/},
  note = {Permanent ID iil:note:n4fcvp7s. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2306-05685-mt-bench.md.