← All notes

Summary

AI Agents That Matter (2024), by Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan, is at https://arxiv.org/abs/2407.01502. They report a narrow focus on accuracy without attention to other metrics. As a result, SOTA agents are needlessly complex and costly, and the community has reached mistaken conclusions about the sources of accuracy gains. The benchmarking needs of model and downstream developers have been conflated, making it hard to identify which agent would be best suited for a particular application. Many agent benchmarks have inadequate holdout sets, and sometimes none at all. There is a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility. They prescribe a principled framework for avoiding overfitting. They design and implement one such optimization, showing its potential to greatly reduce cost while maintaining accuracy. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v1 (1 Jul 2024), submitted 1 Jul 2024.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters as a critical look at how agent papers are evaluated, and as a warning that some reported progress may not hold up under better controls.
• Reproduced: not reproduced (2026-11-11).
• Related work: Related notes already in the library include SWE-bench and Evaluating multimodal agents.
• Limits: This is a pointer note only. It is a critique of evaluation practice, not a new agent with a headline score.
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 1 Jul 2024 (v1). Sources: [S1] https://arxiv.org/abs/2407.01502, v1 (1 Jul 2024). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when reviewing how an agent paper was evaluated, especially if cost, holdout data, or reproducibility are unclear.

Facts

• id: F1 claim: 'The paper argues that current agent benchmarks often focus narrowly on accuracy and neglect cost, which can make state-of-the-art agents needlessly complex and costly.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors say the needs of model developers and downstream developers have been conflated, and that many benchmarks have inadequate holdout sets or none at all.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They also point to a lack of standardization in evaluation practices and to problems of reproducibility.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2407.01502, https://arxiv.org/abs/2407.01502. Paper license: arXiv default (non-exclusive distribution license). v1 (1 Jul 2024). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_hmrjwyf3,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{AI Agents That Matter}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2407-01502-ai-agents-that-matter/},
  note = {Permanent ID iil:note:hmrjwyf3. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2407-01502-ai-agents-that-matter.md.