← All notes

Short answer

Score agents on repeated runs, not one, and judge the steps as well as the end state. tau-bench's pass^k shows how consistent an agent is. Agent-as-a-Judge checks intermediate work and matched human raters in its study. [S1][S2]

Numbers (as reported)

Top function-calling agents passed under 50% of tau-bench tasks, with pass^8 under 25% in retail. [S1]

Claims

• id: C1 claim: pass^k measures reliability over trials. source: S1 evidence: "a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials"
• id: C2 claim: Strong agents succeed on under 50% of tasks. source: S1 evidence: "succeed on <50% of the tasks"
• id: C3 claim: Evaluation compares final database state to goal. source: S1 evidence: "compares the database state at the end of a conversation with the annotated goal state"
• id: C4 claim: Agent-as-a-Judge is as reliable as human evaluation. source: S2 evidence: "is as reliable as our human evaluation baseline"

Sources

• [S1] [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv. https://arxiv.org/abs/2406.12045 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2410.10934] Agent-as-a-Judge: Evaluate Agents with Agents. arXiv. https://arxiv.org/abs/2410.10934 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_rjtjypye,
  author = {{iinventedlight}},
  title = {{Evaluating agents: reliability over trials and agent judges}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-evals-agent-benchmarks-and-judges/},
  note = {Permanent ID iil:note:rjtjypye. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-evals-agent-benchmarks-and-judges.md.