---
title: 'GPQA: A Graduate-Level Google-Proof Q&A Benchmark'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-11; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2311-12022-gpqa.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman, GPQA: A Graduate-Level Google-Proof Q&A Benchmark (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Not reproduced. Abs page and HTML key sections read 2026-11-11. Named code URLs checked; code not run.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, evals, benchmarks
arxiv_id: 2311.12022
alias: 2311.12022-gpqa
sources:
  - id: S1
    title: 'GPQA: A Graduate-Level Google-Proof Q&A Benchmark'
    publisher: arXiv
    authors: 'David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman'
    url: https://arxiv.org/abs/2311.12022
    version_read: 'v1 (20 Nov 2023)'
    accessed: 2026-11-11
    license: 'CC BY 4.0'
---
# GPQA: A Graduate-Level Google-Proof Q&A Benchmark

## Summary
GPQA is a dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry, designed to be difficult and hard to solve by casual web search. The authors report that PhD-level experts reach 65% accuracy (74% after discounting mistakes the experts later identified), while skilled non-expert validators reach only 34% despite spending over 30 minutes on average with unrestricted web access. Their strongest GPT-4 based baseline reached 39% accuracy. They hope the difficulty of GPQA can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities. [S1]

## Claims
- id: C1
  claim: 'GPQA has 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.'
  source: S1
  evidence: "a challenging dataset of 448 multiple-choice questions"
- id: C2
  claim: 'They report that experts reach 65% accuracy, and non-expert validators 34%.'
  source: S1
  evidence: "experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy"
- id: C3
  claim: 'Their strongest GPT-4 based baseline achieved 39% accuracy.'
  source: S1
  evidence: "our strongest GPT-4 based baseline achieving 39% accuracy"

## Verification
- Code available: Paper: https://github.com/idavidrein/gpqa, which returned HTTP 200 on 2026-11-11. Not run.
- Reproduced: not reproduced (2026-11-11). Code not run.
- Related work: Live note 2601-04301-test-set-contamination-generative-evals. This is the paper-verification note for GPQA itself.
- Limits: The 448-question figure is the main set described in the abstract; the HTML also describes a larger GPQA Extended set of 546 questions, which this note does not treat as the headline number. Results are from the original v1 paper (20 Nov 2023) and do not reflect later models.
- License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 20 Nov 2023 (v1). Sources: [S1] https://arxiv.org/abs/2311.12022, v1 (20 Nov 2023). checked_on: 2026-11-11.

## When to apply
Use when you need a graduate-level, web-resistant science QA set, or when thinking about oversight of systems that may outpace non-expert checkers.

## Facts
- id: F1
  claim: 'GPQA contains 448 expert-written multiple-choice questions in biology, physics, and chemistry.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: 'The authors report 65% expert accuracy and 34% accuracy for skilled non-expert validators with web access.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: 'They report that their strongest GPT-4 based baseline achieved 39% accuracy.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2311.12022, https://arxiv.org/abs/2311.12022. License: CC BY 4.0. v1 (20 Nov 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).