---
source: https://iinventedlight.com/library/notes/ao-evals-multimodal-agents.md
title: "Evaluating multimodal agents: Agent AI and OSWorld"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2401.03568] Agent AI: Surveying the Horizons of Multimodal Interaction, CC BY 4.0; arXiv, [2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: evals"
tags: ['agent-operations', 'evals', 'multimodal']
sources:
  - id: S1
    title: "[2401.03568] Agent AI: Surveying the Horizons of Multimodal Interaction"
    publisher: arXiv
    url: https://arxiv.org/abs/2401.03568
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments"
    publisher: arXiv
    url: https://arxiv.org/abs/2404.07972
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Evaluating multimodal agents: Agent AI and OSWorld

**Answer:** OSWorld is a real-computer benchmark of 369 tasks across Ubuntu, Windows and macOS with execution-based scoring, while the Agent AI paper defines multimodal agents as systems that perceive visual, language and other grounded inputs and produce embodied actions.

## Defining the target

The Agent AI paper describes interactive systems that take visual stimuli, language and environment data and produce meaningful actions. It explores improving them with external knowledge, multi-sensory inputs and human feedback.

## OSWorld

The OSWorld authors argue earlier benchmarks either lacked an interactive environment or were tied to one app or domain. OSWorld provides a scalable, real computer environment that supports task setup, execution-based evaluation and interactive learning. Its benchmark covers real web and desktop apps, file I/O and multi-app workflows. Each task has an initial-state configuration and a custom evaluation script so results are reproducible.

## Takeaways for agent evals

- Score by checking the final state of the machine, not the agent's own report.
- Include multi-app workflows; single-app tests overstate ability.
- Reset to a known initial state for every run.

## Claims
- id: C1
  claim: Agent AI is defined as interactive systems that perceive visual, language and grounded data.
  source: S1
  evidence: "can perceive visual stimuli, language inputs, and other environmentally-grounded data"
- id: C2
  claim: OSWorld is a scalable, real computer environment for multimodal agents.
  source: S2
  evidence: "OSWorld, the first-of-its-kind scalable, real computer environment for multimodal agents"
- id: C3
  claim: The benchmark has 369 computer tasks.
  source: S2
  evidence: "we create a benchmark of 369 computer tasks involving real web and desktop apps"
- id: C4
  claim: Each task includes a custom execution-based evaluation script.
  source: S2
  evidence: "a custom execution-based evaluation script for reliable, reproducible evaluation"

## Sources
- [S1] [2401.03568] Agent AI: Surveying the Horizons of Multimodal Interaction. arXiv. https://arxiv.org/abs/2401.03568 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv. https://arxiv.org/abs/2404.07972 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).