---
title: 'Measuring Massive Multitask Language Understanding'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2009-03300-mmlu.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt, Measuring Massive Multitask Language Understanding (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, pointer, evals
arxiv_id: 2009.03300
alias: 2009.03300-mmlu
format: pointer
type: pointer
sources:
  - id: S1
    title: 'Measuring Massive Multitask Language Understanding'
    publisher: arXiv
    authors: 'Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt'
    url: https://arxiv.org/abs/2009.03300
    version_read: 'v3 (12 Jan 2021)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# Measuring Massive Multitask Language Understanding

## Summary
Measuring Massive Multitask Language Understanding (2020), by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, and coauthors, is at https://arxiv.org/abs/2009.03300. The authors propose a test covering 57 tasks that, in their view, requires both extensive world knowledge and problem solving ability. They report that most models of the time were near random chance, while the very largest GPT-3 model improved over random chance by almost 20 percentage points on average. They also report that even the best models still needed substantial gains before expert-level accuracy, that performance was lopsided, and that models often did not know when they were wrong, including on some socially important subjects. The test and code are pointed to https://github.com/hendrycks/test. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. They find that while most recent models have near random-chance accuracy, the very largest GPT 3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v3 (12 Jan 2021), submitted 7 Sep 2020.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters because it introduced MMLU, still one of the most cited broad knowledge benchmarks for language models.
- Reproduced: not reproduced (2026-11-11).
- Related work: A related note already in the library is [Test set contamination in generative evals](https://iinventedlight.com/library/notes/2601-04301-test-set-contamination-generative-evals/).
- Limits: This is a pointer note only. The nearly 20 point gain over random chance is the authors' reported GPT-3 result from the v3 paper (12 Jan 2021).
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 7 Sep 2020 (v1); last revised 12 Jan 2021 (v3). Sources: [S1] https://arxiv.org/abs/2009.03300, v3 (12 Jan 2021). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when you need the original MMLU paper, still the most cited broad academic-knowledge benchmark for language models.

## Facts
- id: F1
  claim: 'The paper proposes a test covering 57 tasks that measures a model''s multitask accuracy, requiring extensive world knowledge and problem solving ability.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors report that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'They say the test and code are available at https://github.com/hendrycks/test.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2009.03300, https://arxiv.org/abs/2009.03300. Paper license: arXiv default (non-exclusive distribution license). v3 (12 Jan 2021). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).