---
source: https://iinventedlight.com/library/notes/ao-evals-agentbench-osworld.md
title: "Agent benchmarks: AgentBench and OSWorld"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2308.03688] AgentBench: Evaluating LLMs as Agents, CC BY 4.0; arXiv, [2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: evaluation"
tags: agents, benchmarks, evaluation
sources:
  - id: S1
    title: "[2308.03688] AgentBench: Evaluating LLMs as Agents"
    publisher: arXiv
    url: https://arxiv.org/abs/2308.03688
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments"
    publisher: arXiv
    url: https://arxiv.org/abs/2404.07972
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Agent benchmarks: AgentBench and OSWorld

## Short answer
AgentBench tests LLMs as agents across 8 interactive environments. OSWorld tests multimodal agents on open-ended tasks on real computers. AgentBench found that poor long-term reasoning, decision-making and instruction following are the main reasons agents fail. [S1][S2]

## Claims
- id: C1
  claim: AgentBench consists of 8 distinct environments.
  source: S1
  evidence: "a multi-dimensional benchmark that consists of 8 distinct environments"
- id: C2
  claim: Poor long-term reasoning and instruction following are main obstacles.
  source: S1
  evidence: "poor long-term reasoning, decision-making, and instruction following abilities are the main obstacles"
- id: C3
  claim: Existing benchmarks often lack an interactive environment.
  source: S2
  evidence: "existing benchmarks either lack an interactive environment"

## Sources
- [S1] [2308.03688] AgentBench: Evaluating LLMs as Agents. arXiv. https://arxiv.org/abs/2308.03688 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2404.07972] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv. https://arxiv.org/abs/2404.07972 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).