← All notes

Answer: Prometheus, an open 13B evaluator given a reference answer and score rubric, scores a Pearson correlation of 0.897 with human evaluators (S1). A separate study of thirteen judge models found that only the best (and largest) models achieve reasonable alignment with humans (S2).

Prometheus (S1)

• Using a powerful proprietary LLM such as GPT-4 as an evaluator for long-form responses has become the de facto standard.
• For large-scale evaluation tasks and custom criteria, proprietary evaluators are unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs.
• The Feedback Collection dataset consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4.
• With 45 customized score rubrics, Prometheus's 0.897 correlation is on par with GPT-4 (0.882) and greatly outperforms ChatGPT (0.392).

Judging the judges (S2)

• The study focuses on a clean scenario in which inter-human agreement is high, with thirteen judge models judging answers of nine exam-taker models.
• Even the best judges are still quite far behind inter-human agreement, and their assigned scores may still differ with up to 5 points from human-assigned scores.
• For ranking the nine exam-taker models, smaller models and even a lexical metric may provide a reasonable signal.
• Judge models show sensitivity to prompt complexity and length, and a tendency toward leniency.
• Judges with high percent agreement can still assign vastly different scores, so alignment metrics beyond simple percent alignment matter.

Practical reading

The S2 authors suggest that caution may be wise when using judges in more complex setups.

Claims

• id: C1 claim: Prometheus is a 13B evaluator LLM. source: S1 evidence: "Using the Feedback Collection, we train Prometheus"
• id: C2 claim: Prometheus greatly outperforms ChatGPT as an evaluator. source: S1 evidence: "and greatly outperforms ChatGPT (0.392)."
• id: C3 claim: Judge models tend toward leniency. source: S2 evidence: "and a tendency toward leniency."

Sources

• [S1] [2310.08491] Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. arXiv. https://arxiv.org/abs/2310.08491 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2406.12624] Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges. arXiv. https://arxiv.org/abs/2406.12624 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_6nm7qxe5,
  author = {{iinventedlight}},
  title = {{LLM judges: an open evaluator model and the limits of judging}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-evals-llm-judges/},
  note = {Permanent ID iil:note:6nm7qxe5. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-evals-llm-judges.md.