---
title: "Measuring short-form factuality in large language models"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is arXiv non-exclusive distribution license (verified on arXiv abs page 2026-11-12); this pointer note is our own words and is CC BY 4.0; we link to the paper rather than reproduce it'
source: https://iinventedlight.com/library/notes/2411-04368-simpleqa.md
source_license: "arXiv non-exclusive distribution license"
attribution: 'Pointer by iinventedlight (AI-assisted). Source: Jason Wei, Nguyen Karina, Hyung Won Chung, and 5 coauthors, Measuring short-form factuality in large language models (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: 'Pointer note only. Paper not reproduced. Abs page and license checked 2026-11-12.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, evals, factuality, simpleqa, short-form-qa
arxiv_id: 2411.04368
alias: 2411-04368-simpleqa
type: pointer
format: pointer
sources:
  - id: S1
    title: "Measuring short-form factuality in large language models"
    publisher: arXiv
    authors: "Jason Wei and 7 coauthors (listed on the arXiv abs page)"
    url: https://arxiv.org/abs/2411.04368
    version_read: "v1 (7 Nov 2024)"
    accessed: 2026-11-12
    license: "arXiv non-exclusive distribution license"
---
# Measuring short-form factuality in large language models

## Summary
[Measuring short-form factuality in large language models](https://arxiv.org/abs/2411.04368) (2024) introduces SimpleQA, OpenAI's short-answer factuality eval. Questions are built so each one has a single, undisputed answer, and the set was collected adversarially against GPT-4 responses so that it would stay hard.

What it introduced: a grading scheme that marks each answer correct, incorrect, or not attempted. The intended behavior is to get as many items right as possible while skipping questions the model is not confident about, as a test of whether a model knows what it knows. The authors hope the set will stay useful for several further generations of frontier models.

What it led to: a common short-form factuality check that is less saturated than older QA sets. Live notes include [2109-07958-truthfulqa](https://iinventedlight.com/library/notes/2109-07958-truthfulqa/). BrowseComp, a later browsing benchmark released through the same simple-evals repository, is a pointer note in this batch.

Where to read it: https://arxiv.org/abs/2411.04368 (v1, 7 Nov 2024). A blog post is noted at https://openai.com/index/introducing-simpleqa/, and the data live at https://github.com/openai/simple-evals.

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
