← All notes

Summary

Measuring Massive Multitask Language Understanding (2020), by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, and coauthors, is at https://arxiv.org/abs/2009.03300. The authors propose a test covering 57 tasks that, in their view, requires both extensive world knowledge and problem solving ability. They report that most models of the time were near random chance, while the very largest GPT-3 model improved over random chance by almost 20 percentage points on average. They also report that even the best models still needed substantial gains before expert-level accuracy, that performance was lopsided, and that models often did not know when they were wrong, including on some socially important subjects. The test and code are pointed to https://github.com/hendrycks/test. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. They find that while most recent models have near random-chance accuracy, the very largest GPT 3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v3 (12 Jan 2021), submitted 7 Sep 2020.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it introduced MMLU, still one of the most cited broad knowledge benchmarks for language models.
• Reproduced: not reproduced (2026-11-11).
• Related work: A related note already in the library is Test set contamination in generative evals.
• Limits: This is a pointer note only. The nearly 20 point gain over random chance is the authors' reported GPT-3 result from the v3 paper (12 Jan 2021).
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 7 Sep 2020 (v1); last revised 12 Jan 2021 (v3). Sources: [S1] https://arxiv.org/abs/2009.03300, v3 (12 Jan 2021). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original MMLU paper, still the most cited broad academic-knowledge benchmark for language models.

Facts

• id: F1 claim: 'The paper proposes a test covering 57 tasks that measures a model''s multitask accuracy, requiring extensive world knowledge and problem solving ability.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors report that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They say the test and code are available at https://github.com/hendrycks/test.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2009.03300, https://arxiv.org/abs/2009.03300. Paper license: arXiv default (non-exclusive distribution license). v3 (12 Jan 2021). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_6wsp2bke,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Measuring Massive Multitask Language Understanding}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2009-03300-mmlu/},
  note = {Permanent ID iil:note:6wsp2bke. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2009-03300-mmlu.md.