← All notes

Answer: OSWorld is a real-computer benchmark of 369 tasks across Ubuntu, Windows and macOS with execution-based scoring, while the Agent AI paper defines multimodal agents as systems that perceive visual, language and other grounded inputs and produce embodied actions.

Defining the target

The Agent AI paper describes interactive systems that take visual stimuli, language and environment data and produce meaningful actions. It explores improving them with external knowledge, multi-sensory inputs and human feedback.

OSWorld

The OSWorld authors argue earlier benchmarks either lacked an interactive environment or were tied to one app or domain. OSWorld provides a scalable, real computer environment that supports task setup, execution-based evaluation and interactive learning. Its benchmark covers real web and desktop apps, file I/O and multi-app workflows. Each task has an initial-state configuration and a custom evaluation script so results are reproducible.

Takeaways for agent evals

• Score by checking the final state of the machine, not the agent's own report.
• Include multi-app workflows; single-app tests overstate ability.
• Reset to a known initial state for every run.

Claims

• id: C1 claim: Agent AI is defined as interactive systems that perceive visual, language and grounded data. source: S1 evidence: "can perceive visual stimuli, language inputs, and other environmentally-grounded data"
• id: C2 claim: OSWorld is a scalable, real computer environment for multimodal agents. source: S2 evidence: "OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents"
• id: C3 claim: The benchmark has 369 computer tasks. source: S2 evidence: "we create a benchmark of 369 computer tasks involving real web and desktop apps"
• id: C4 claim: Each task includes a custom execution-based evaluation script. source: S2 evidence: "a custom execution-based evaluation script for reliable, reproducible evaluation"

Sources

• [S1] [2401.03568] Agent AI: Surveying the Horizons of Multimodal Interaction. arXiv. https://arxiv.org/abs/2401.03568 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv. https://arxiv.org/abs/2404.07972 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_z0gq0dkn,
  author = {{iinventedlight}},
  title = {{Evaluating multimodal agents: Agent AI and OSWorld}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-evals-multimodal-agents/},
  note = {Permanent ID iil:note:z0gq0dkn. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-evals-multimodal-agents.md.