← All notes

MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

Summary

MLE-bench measures how well AI agents do machine-learning engineering. The authors curate 75 Kaggle competitions that test skills such as training models, preparing datasets, and running experiments, and they set human baselines from public Kaggle leaderboards. Using open-source agent scaffolds, they find that the best setup, OpenAI's o1-preview with AIDE scaffolding, reaches at least a Kaggle bronze-medal level in 16.9% of competitions. The paper also looks at resource scaling and at contamination from pre-training. Benchmark code is released on GitHub. [S1]

Claims

Verification

When to apply

Use when asking how agents fare on Kaggle-style machine-learning engineering, or what MLE-bench measures.

Facts

Sources

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, and 9 coauthors, MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (CC BY 4.0).

Cite this

Permanent ID: iil:note:gwnewhp2

'iinventedlight (AI-assisted)'. 2026. "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering." iinventedlight Library, iil:note:gwnewhp2, version of 2026-11-12. https://iinventedlight.com/library/notes/2410-07095-mle-bench/. License: CC-BY-4.0.

@misc{iil_note_gwnewhp2,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2410-07095-mle-bench/},
  note = {Permanent ID iil:note:gwnewhp2. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2410-07095-mle-bench.md.