← All notes

Answer: The state-of-the-art LLM, GPT-4o, solves only about 49% of AppWorld's 'normal' tasks and about 30% of 'challenge' tasks (S1). On tau-bench, even state-of-the-art function calling agents (like gpt-4o) succeed on under 50% of the tasks (S2).

AppWorld (S1)

• Existing tool-use benchmarks only cover tasks that require a simple sequence of API calls.
• The AppWorld Engine is an execution environment (60K lines of code) of 9 day-to-day apps operable via 457 APIs, populated with digital activities simulating the lives of about 100 fictitious users.
• The AppWorld Benchmark is a suite of 750 autonomous agent tasks requiring rich and interactive code generation.
• It uses state-based unit tests, which allow different ways of completing a task while also checking for unexpected changes, i.e., collateral damage.

tau-bench (S2)

• Existing benchmarks do not test agents on interaction with human users or the ability to follow domain-specific rules.
• tau-bench emulates dynamic conversations between a user, simulated by language models, and an agent given domain-specific API tools and policy guidelines.
• It compares the database state at the end of a conversation with the annotated goal state.
• A new metric, pass^k, evaluates the reliability of agent behavior over multiple trials. Agents were quite inconsistent (pass^8 under 25% in retail).

What to borrow

It supports robust programmatic evaluation with state-based unit tests (S1). The tau-bench evaluation compares the database state at the end of a conversation with the annotated goal state (S2).

Claims

• id: C1 claim: Existing tool-use benchmarks cover only simple sequences of API calls. source: S1 evidence: "as they only cover tasks that require a simple sequence of API calls."
• id: C2 claim: Existing tool-use benchmarks are inadequate. source: S1 evidence: "However, existing benchmarks for tool use are inadequate"
• id: C3 claim: In tau-bench the user is simulated by language models. source: S2 evidence: "simulated by language models"

Sources

• [S1] [2407.18901] AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. arXiv. https://arxiv.org/abs/2407.18901 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv. https://arxiv.org/abs/2406.12045 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_htcqzmpy,
  author = {{iinventedlight}},
  title = {{AppWorld and tau-bench: benchmarks for agents that use apps}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-evals-app-world-agents/},
  note = {Permanent ID iil:note:htcqzmpy. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-evals-app-world-agents.md.