← All notes

Summary

GAIA: a benchmark for General AI Assistants (2023), by Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom, is at https://arxiv.org/abs/2311.12983. The authors present questions that are conceptually simple for people but hard for advanced models, requiring reasoning, multi-modality handling, web browsing, and tool use. They report that human respondents obtain 92% versus 15% for GPT-4 equipped with plugins. They devise 466 questions and their answers, and they posit that a system should exhibit similar robustness as the average human does on such questions. They release the questions while holding back answers to 300 of them to power a leaderboard at https://huggingface.co/gaia-benchmark. They show that human respondents obtain 92% versus 15% for GPT 4 equipped with plugins. They release their questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v1 (21 Nov 2023), submitted 21 Nov 2023.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it became a standard test of general assistants that must use tools, browse, and handle realistic multi-step questions, rather than only professional exams.
• Reproduced: not reproduced (2026-11-11).
• Related work: A related note already in the library is Evaluating multimodal agents.
• Limits: This is a pointer note only. The 92% versus 15% comparison is the authors' reported human versus GPT-4 with plugins result from the 21 Nov 2023 version.
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 21 Nov 2023 (v1). Sources: [S1] https://arxiv.org/abs/2311.12983, v1 (21 Nov 2023). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original GAIA paper for general assistant evaluation involving tools, browsing, and multi-step questions.

Facts

• id: F1 claim: 'GAIA is a benchmark of 466 real-world questions that require reasoning, multi-modality handling, web browsing, and tool use.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors report that human respondents obtain 92% versus 15% for GPT-4 equipped with plugins.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They release questions while retaining answers to 300 of them for a leaderboard at https://huggingface.co/gaia-benchmark.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2311.12983, https://arxiv.org/abs/2311.12983. Paper license: arXiv default (non-exclusive distribution license). v1 (21 Nov 2023). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_f1820ekc,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{GAIA: a benchmark for General AI Assistants}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2311-12983-gaia/},
  note = {Permanent ID iil:note:f1820ekc. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2311-12983-gaia.md.