BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Summary
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (2025) is a short, hard test of whether an agent can persist on the live web to find facts that are difficult to locate. It has 1,266 questions whose answers are short and easy to check, even though getting to those answers may take a long, entangled search.
What it introduced: a browsing analogue of a programming contest: incomplete as a picture of real user queries, but useful as a core test of persistence and creativity in finding information. The benchmark deliberately skips long-form answers and ambiguous questions in order to keep grading simple.
What it led to: a compact eval used alongside other web-agent suites when people ask whether a browsing agent can actually find buried facts. Live notes include 2307-13854-webarena. Related pointer notes in this batch cover Mind2Web, VisualWebArena, and WebVoyager. SimpleQA, a short-form factuality set from the same group, is also in this batch.
Where to read it: https://arxiv.org/abs/2504.12516 (v1, 16 Apr 2025). The authors point to https://github.com/openai/simple-evals.
The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Jason Wei, Zhiqing Sun, Spencer Papay, and 7 coauthors, BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents." iinventedlight Library, iil:note:q22c379v, version of 2026-11-12. https://iinventedlight.com/library/notes/2504-12516-browsecomp/. License: CC-BY-4.0.
@misc{iil_note_q22c379v,
author = {{'iinventedlight (AI-assisted)'}},
title = {{BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2504-12516-browsecomp/},
note = {Permanent ID iil:note:q22c379v. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2504-12516-browsecomp.md.