← All notes

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

Summary

This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark. It adds more reasoning-focused questions, expands the choice set from four to ten options, and drops trivial or noisy MMLU items. Experimental results show a significant drop in accuracy by 16% to 33% compared to MMLU. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, in contrast to findings on the original MMLU, indicating more complex reasoning questions. Please visit our leaderboard at https://huggingface.co/spaces/TIGER-Lab/MMLU-Pro. The v6 paper is the NeurIPS 2024 Datasets and Benchmarks version. [S1]

Claims

Verification

When to apply

Use when comparing models on a harder, less prompt-fragile successor to MMLU, or when asking why CoT helps more here than on the original set.

Facts

Sources

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Yubo Wang, Xueguang Ma, Ge Zhang, and 14 coauthors, MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (CC BY 4.0).

Cite this

Permanent ID: iil:note:tzxdexvz

'iinventedlight (AI-assisted)'. 2026. "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark." iinventedlight Library, iil:note:tzxdexvz, version of 2026-11-12. https://iinventedlight.com/library/notes/2406-01574-mmlu-pro/. License: CC-BY-4.0.

@misc{iil_note_tzxdexvz,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2406-01574-mmlu-pro/},
  note = {Permanent ID iil:note:tzxdexvz. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2406-01574-mmlu-pro.md.