---
source: https://iinventedlight.com/library/notes/ao-evals-agent-benchmarks-and-judges.md
title: "Evaluating agents: reliability over trials and agent judges"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, CC BY 4.0; arXiv, [2410.10934] Agent-as-a-Judge: Evaluate Agents with Agents, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was rerun; only the fetched arXiv abstract pages were read. Results are as reported by the authors."
category: agent-operations
topic_area: "agent operations: evaluation"
tags: agents, evaluation, benchmarks, reliability
sources:
  - id: S1
    title: "[2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains"
    publisher: arXiv
    url: https://arxiv.org/abs/2406.12045
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2410.10934] Agent-as-a-Judge: Evaluate Agents with Agents"
    publisher: arXiv
    url: https://arxiv.org/abs/2410.10934
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Evaluating agents: reliability over trials and agent judges

## Short answer
Score agents on repeated runs, not one, and judge the steps as well as the end state. tau-bench's pass^k shows how consistent an agent is. Agent-as-a-Judge checks intermediate work and matched human raters in its study. [S1][S2]

## Numbers (as reported)
Top function-calling agents passed under 50% of tau-bench tasks, with pass^8 under 25% in retail. [S1]

## Claims
- id: C1
  claim: pass^k measures reliability over trials.
  source: S1
  evidence: "a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials"
- id: C2
  claim: Strong agents succeed on under 50% of tasks.
  source: S1
  evidence: "succeed on <50% of the tasks"
- id: C3
  claim: Evaluation compares final database state to goal.
  source: S1
  evidence: "compares the database state at the end of a conversation with the annotated goal state"
- id: C4
  claim: Agent-as-a-Judge is as reliable as human evaluation.
  source: S2
  evidence: "is as reliable as our human evaluation baseline"

## Sources
- [S1] [2406.12045] $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv. https://arxiv.org/abs/2406.12045 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2410.10934] Agent-as-a-Judge: Evaluate Agents with Agents. arXiv. https://arxiv.org/abs/2410.10934 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).