← All notes

Measuring short-form factuality in large language models

Summary

Measuring short-form factuality in large language models (2024) introduces SimpleQA, OpenAI's short-answer factuality eval. Questions are built so each one has a single, undisputed answer, and the set was collected adversarially against GPT-4 responses so that it would stay hard.

What it introduced: a grading scheme that marks each answer correct, incorrect, or not attempted. The intended behavior is to get as many items right as possible while skipping questions the model is not confident about, as a test of whether a model knows what it knows. The authors hope the set will stay useful for several further generations of frontier models.

What it led to: a common short-form factuality check that is less saturated than older QA sets. Live notes include 2109-07958-truthfulqa. BrowseComp, a later browsing benchmark released through the same simple-evals repository, is a pointer note in this batch.

Where to read it: https://arxiv.org/abs/2411.04368 (v1, 7 Nov 2024). A blog post is noted at https://openai.com/index/introducing-simpleqa/, and the data live at https://github.com/openai/simple-evals.

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Jason Wei, Nguyen Karina, Hyung Won Chung, and 5 coauthors, Measuring short-form factuality in large language models (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:a0rjr828

'iinventedlight (AI-assisted)'. 2026. "Measuring short-form factuality in large language models." iinventedlight Library, iil:note:a0rjr828, version of 2026-11-12. https://iinventedlight.com/library/notes/2411-04368-simpleqa/. License: CC-BY-4.0.

@misc{iil_note_a0rjr828,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Measuring short-form factuality in large language models}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2411-04368-simpleqa/},
  note = {Permanent ID iil:note:a0rjr828. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2411-04368-simpleqa.md.