← All notes

Summary

GPQA is a dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry, designed to be difficult and hard to solve by casual web search. The authors report that PhD-level experts reach 65% accuracy (74% after discounting mistakes the experts later identified), while skilled non-expert validators reach only 34% despite spending over 30 minutes on average with unrestricted web access. Their strongest GPT-4 based baseline reached 39% accuracy. They hope the difficulty of GPQA can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities. [S1]

Claims

• id: C1 claim: 'GPQA has 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.' source: S1 evidence: "a challenging dataset of 448 multiple-choice questions"
• id: C2 claim: 'They report that experts reach 65% accuracy, and non-expert validators 34%.' source: S1 evidence: "experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy"
• id: C3 claim: 'Their strongest GPT-4 based baseline achieved 39% accuracy.' source: S1 evidence: "our strongest GPT-4 based baseline achieving 39% accuracy"

Verification

• Code available: Paper: https://github.com/idavidrein/gpqa, which returned HTTP 200 on 2026-11-11. Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live note 2601-04301-test-set-contamination-generative-evals. This is the paper-verification note for GPQA itself.
• Limits: The 448-question figure is the main set described in the abstract; the HTML also describes a larger GPQA Extended set of 546 questions, which this note does not treat as the headline number. Results are from the original v1 paper (20 Nov 2023) and do not reflect later models.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 20 Nov 2023 (v1). Sources: [S1] https://arxiv.org/abs/2311.12022, v1 (20 Nov 2023). checked_on: 2026-11-11.

When to apply

Use when you need a graduate-level, web-resistant science QA set, or when thinking about oversight of systems that may outpace non-expert checkers.

Facts

• id: F1 claim: 'GPQA contains 448 expert-written multiple-choice questions in biology, physics, and chemistry.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors report 65% expert accuracy and 34% accuracy for skilled non-expert validators with web access.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They report that their strongest GPT-4 based baseline achieved 39% accuracy.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2311.12022, https://arxiv.org/abs/2311.12022. License: CC BY 4.0. v1 (20 Nov 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_xyevjhbz,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{GPQA: A Graduate-Level Google-Proof Q\&A Benchmark}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2311-12022-gpqa/},
  note = {Permanent ID iil:note:xyevjhbz. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2311-12022-gpqa.md.