---
title: 'AI Agents That Matter'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2407-01502-ai-agents-that-matter.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan, AI Agents That Matter (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: agents'
tags: arxiv, pointer, agents, evals
arxiv_id: 2407.01502
alias: 2407.01502-ai-agents-that-matter
format: pointer
type: pointer
sources:
  - id: S1
    title: 'AI Agents That Matter'
    publisher: arXiv
    authors: 'Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, Arvind Narayanan'
    url: https://arxiv.org/abs/2407.01502
    version_read: 'v1 (1 Jul 2024)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# AI Agents That Matter

## Summary
AI Agents That Matter (2024), by Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan, is at https://arxiv.org/abs/2407.01502. They report a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. The benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Many agent benchmarks have inadequate holdout sets, and sometimes none at all. There is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. They prescribe a principled framework for avoiding overfitting. They design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v1 (1 Jul 2024), submitted 1 Jul 2024.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters as a critical look at how agent papers are evaluated, and as a warning that some reported progress may not hold up under better controls.
- Reproduced: not reproduced (2026-11-11).
- Related work: Related notes already in the library include [SWE-bench](https://iinventedlight.com/library/notes/2310-06770-swe-bench/) and [Evaluating multimodal agents](https://iinventedlight.com/library/notes/ao-evals-multimodal-agents/).
- Limits: This is a pointer note only. It is a critique of evaluation practice, not a new agent with a headline score.
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 1 Jul 2024 (v1). Sources: [S1] https://arxiv.org/abs/2407.01502, v1 (1 Jul 2024). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when reviewing how an agent paper was evaluated, especially if cost, holdout data, or reproducibility are unclear.

## Facts
- id: F1
  claim: 'The paper argues that current agent benchmarks often focus narrowly on accuracy and neglect cost, which can make state-of-the-art agents needlessly complex and costly.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors say the needs of model developers and downstream developers have been conflated, and that many benchmarks have inadequate holdout sets or none at all.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'They also point to a lack of standardization in evaluation practices and to problems of reproducibility.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2407.01502, https://arxiv.org/abs/2407.01502. Paper license: arXiv default (non-exclusive distribution license). v1 (1 Jul 2024). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).