---
title: 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2306-05685-mt-bench.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. (13 authors), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, pointer, evals
arxiv_id: 2306.05685
alias: 2306.05685-mt-bench
format: pointer
type: pointer
sources:
  - id: S1
    title: 'Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena'
    publisher: arXiv
    authors: 'Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, et al. (13 authors)'
    url: https://arxiv.org/abs/2306.05685
    version_read: 'v4 (24 Dec 2023)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

## Summary
Judging LLM as a Judge with MT Bench and Chatbot Arena (2023), by Lianmin Zheng, Wei Lin Chiang, Ying Sheng, and colleagues, is at https://arxiv.org/abs/2306.05685. The authors examine limits of model judges, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and they propose ways to reduce some of those problems. They then compare model judges with human preference using MT-bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform. They report that strong judges such as GPT-4 can match both controlled and crowdsourced human preferences well, reaching over 80% agreement, comparable to agreement between humans. They also show that these benchmarks complement traditional tests, and they release MT-bench questions, 3K expert votes, and 30K conversations at the FastChat llm_judge path. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v4 (24 Dec 2023), submitted 9 Jun 2023.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters because it tested whether a strong language model can stand in for human preference judgments, and it introduced two evaluation resources that are still widely used: MT-Bench and Chatbot Arena.
- Reproduced: not reproduced (2026-11-11).
- Related work: A related note already in the library is [RAGAS](https://iinventedlight.com/library/notes/2309-15217-ragas-rag-evaluation/).
- Limits: This is a pointer note only. The over 80% agreement figure is the authors' reported result from the v4 paper (24 Dec 2023).
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 9 Jun 2023 (v1); last revised 24 Dec 2023 (v4). Sources: [S1] https://arxiv.org/abs/2306.05685, v4 (24 Dec 2023). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when you need the original LLM-as-a-judge paper, including MT-Bench and Chatbot Arena as human-preference references.

## Facts
- id: F1
  claim: 'The paper introduces MT-bench, a multi-turn question set, and Chatbot Arena, a crowdsourced battle platform, to study LLM-as-a-judge against human preference.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors report that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'They say MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available in the FastChat repository.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2306.05685, https://arxiv.org/abs/2306.05685. Paper license: arXiv default (non-exclusive distribution license). v4 (24 Dec 2023). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).