---
source: https://iinventedlight.com/library/notes/ao-evals-app-world-agents.md
title: "AppWorld and tau-bench: benchmarks for agents that use apps"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2407.18901] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents, CC BY 4.0; arXiv, [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: evaluation"
tags: ['agents', 'benchmarks', 'tool-use', 'evaluation']
sources:
  - id: S1
    title: "[2407.18901] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents"
    publisher: arXiv
    url: https://arxiv.org/abs/2407.18901
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains"
    publisher: arXiv
    url: https://arxiv.org/abs/2406.12045
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# AppWorld and tau-bench: benchmarks for agents that use apps

**Answer:** The state-of-the-art LLM, GPT-4o, solves only about 49% of AppWorld's 'normal' tasks and about 30% of 'challenge' tasks (S1). On tau-bench, even state-of-the-art function calling agents (like gpt-4o) succeed on under 50% of the tasks (S2).

## AppWorld (S1)

- Existing tool-use benchmarks only cover tasks that require a simple sequence of API calls.
- The AppWorld Engine is an execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs, populated with digital activities simulating the lives of about 100 fictitious users.
- The AppWorld Benchmark is a suite of 750 autonomous agent tasks requiring rich and interactive code generation.
- It uses state-based unit tests, which allow different ways of completing a task while also checking for unexpected changes, i.e., collateral damage.

## tau-bench (S2)

- Existing benchmarks do not test agents on interaction with human users or the ability to follow domain-specific rules.
- tau-bench emulates dynamic conversations between a user, simulated by language models, and an agent given domain-specific API tools and policy guidelines.
- It compares the database state at the end of a conversation with the annotated goal state.
- A new metric, pass^k, evaluates the reliability of agent behavior over multiple trials. Agents were quite inconsistent (pass^8 under 25% in retail).

## What to borrow

It supports robust programmatic evaluation with state-based unit tests (S1). The tau-bench evaluation compares the database state at the end of a conversation with the annotated goal state (S2).

## Claims
- id: C1
  claim: Existing tool-use benchmarks cover only simple sequences of API calls.
  source: S1
  evidence: "as they only cover tasks that require a simple sequence of API calls."
- id: C2
  claim: Existing tool-use benchmarks are inadequate.
  source: S1
  evidence: "However, existing benchmarks for tool use are inadequate"
- id: C3
  claim: In tau-bench the user is simulated by language models.
  source: S2
  evidence: "simulated by language models"

## Sources
- [S1] [2407.18901] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. arXiv. https://arxiv.org/abs/2407.18901 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv. https://arxiv.org/abs/2406.12045 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).