OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

iinventedlight (AI-assisted)

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Summary

OSWorld is a real computer environment for multimodal agents, with task setup, execution-based evaluation, and interactive learning across Ubuntu, Windows, and macOS. The authors build a benchmark of 369 computer tasks involving real web and desktop apps and multi-app workflows. They report human success over 72.36% and best-model success of only 12.24%. [S1]

Claims

id: C1 claim: 'OSWorld is a real computer environment for multimodal agents across Ubuntu, Windows, and macOS.' source: S1 evidence: "we introduce OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents"
id: C2 claim: 'The benchmark has 369 computer tasks involving real web and desktop apps.' source: S1 evidence: "a benchmark of 369 computer tasks involving real web and desktop apps"
id: C3 claim: 'They report human success over 72.36% and best-model success of 12.24%.' source: S1 evidence: "humans can accomplish over 72.36% of the tasks, the best model achieves only 12.24% success"

Verification

Code available: The paper points to https://os-world.github.io, which returned HTTP 500 from this fetcher on 2026-11-11, so availability of that project page was not verified here. Not run.
Reproduced: not reproduced (2026-11-11). Code not run.
Related work: Live notes ao-evals-agentbench-osworld and ao-evals-multimodal-agents. This is the paper-verification note for OSWorld itself.
Limits: The 12.24% best-model figure is from the v2 evaluation (revised 30 May 2024) and does not reflect later computer-use models. The official project URL named in the paper did not return HTTP 200 to this fetcher.
License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 11 Apr 2024 (v1); last revised 30 May 2024 (v2). Sources: [S1] https://arxiv.org/abs/2404.07972, v2 (30 May 2024). checked_on: 2026-11-11.

When to apply

Use when evaluating computer-use or desktop agents on real applications, not on a toy mock UI.

Facts

id: F1 claim: 'OSWorld is a real computer environment for multimodal agents, covering Ubuntu, Windows, and macOS.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
id: F2 claim: 'The authors build a benchmark of 369 computer tasks involving real web and desktop apps and multi-app workflows.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
id: F3 claim: 'They report human success over 72.36% and best-model success of 12.24% in their evaluation.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

[S1] arXiv 2404.07972, https://arxiv.org/abs/2404.07972. License: CC BY 4.0. v2 (30 May 2024). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, et al. (17 authors), OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (CC BY 4.0).

Cite this

Permanent ID: iil:note:jtzefge2

'iinventedlight (AI-assisted)'. 2026. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." iinventedlight Library, iil:note:jtzefge2, version of 2026-11-11. https://iinventedlight.com/library/notes/2404-07972-osworld/. License: CC-BY-4.0.

@misc{iil_note_jtzefge2,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2404-07972-osworld/},
  note = {Permanent ID iil:note:jtzefge2. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2404-07972-osworld.md.