TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
Summary
TheAgentCompany is a benchmark of work-like tasks for language-model agents inside a simulated software company. Agents browse the web, write and run code, and talk with coworkers in a self-contained environment with internal sites and data. The authors test baseline agents that use both closed APIs and open-weight models, and they report that the most competitive agent can complete 30% of tasks on its own. A good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. The authors release code, data, environment, and experiments on https://the-agent-company.com. [S1]
Claims
- id: C1 claim: "TheAgentCompany evaluates agents in a simulated software company where they browse, code, run programs, and communicate." source: S1 evidence: "mimics a small software company environment"
- id: C2 claim: "The most competitive agent can complete 30% of the tasks autonomously." source: S1 evidence: "the most competitive agent can complete 30% of tasks autonomously"
- id: C3 claim: "Code, data, environment, and experiments are released on the project site." source: S1 evidence: "https://the-agent-company.com"
Verification
- Code available: Paper: https://the-agent-company.com and https://github.com/TheAgentCompany/TheAgentCompany, both HTTP 200 on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Related agent-work evals include 2310-06770-swe-bench, 2405-15793-swe-agent, and 2406-12045-tau-bench.
- Limits: The 30% figure is for the best baseline in the paper's environment, not a claim about real workplaces. Task definitions and model choices are as of v3 (10 Sep 2025).
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 18 Dec 2024 (v1); last revised v3 (10 Sep 2025). Sources: [S1] https://arxiv.org/abs/2412.14161, v3 (10 Sep 2025). checked_on: 2026-11-12.
When to apply
Use when asking how well current agents handle multi-step work tasks in a simulated company setting.
Facts
- id: F1 claim: "TheAgentCompany is a self-contained simulated software-company environment for work-like agent tasks." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F2 claim: "The authors report that their most competitive agent completes 30% of tasks autonomously." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F3 claim: "Code and data are released at https://the-agent-company.com and on GitHub." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
Sources
- [S1] arXiv 2412.14161, https://arxiv.org/abs/2412.14161. License: CC BY 4.0. v3 (10 Sep 2025). Accessed 2026-11-12.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.
License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Frank F. Xu, Yufan Song, Boxuan Li, and 18 coauthors, TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (CC BY 4.0).
Cite this
'iinventedlight (AI-assisted)'. 2026. "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks." iinventedlight Library, iil:note:m06tdxbh, version of 2026-11-12. https://iinventedlight.com/library/notes/2412-14161-theagentcompany/. License: CC-BY-4.0.
@misc{iil_note_m06tdxbh,
author = {{'iinventedlight (AI-assisted)'}},
title = {{TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2412-14161-theagentcompany/},
note = {Permanent ID iil:note:m06tdxbh. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2412-14161-theagentcompany.md.