MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Summary
MLE-bench measures how well AI agents do machine-learning engineering. The authors curate 75 Kaggle competitions that test skills such as training models, preparing datasets, and running experiments, and they set human baselines from public Kaggle leaderboards. Using open-source agent scaffolds, they find that the best setup, OpenAI's o1-preview with AIDE scaffolding, reaches at least a Kaggle bronze-medal level in 16.9% of competitions. The paper also looks at resource scaling and at contamination from pre-training. Benchmark code is released on GitHub. [S1]
Claims
- id: C1 claim: "MLE-bench curates 75 Kaggle competitions as machine-learning engineering tasks." source: S1 evidence: "we curate 75 ML engineering-related competitions from Kaggle"
- id: C2 claim: "The best reported setup, o1-preview with AIDE, reaches bronze-medal level in 16.9% of competitions." source: S1 evidence: "at least the level of a Kaggle bronze medal in 16.9% of competitions"
- id: C3 claim: "Benchmark code is released on GitHub." source: S1 evidence: "github.com/openai/mle-bench/"
Verification
- Code available: Paper: github.com/openai/mle-bench/, which returned HTTP 200 at https://github.com/openai/mle-bench on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Related agent-engineering evals include 2310-06770-swe-bench and ao-evals-ml-engineering-agents.
- Limits: The 16.9% bronze-level figure is for one scaffold and one model in the v6 paper. Kaggle tasks are a slice of ML engineering, not all of it. The first seven authors share equal contribution (order randomized).
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 9 Oct 2024 (v1); last revised v6 (26 Feb 2025). Sources: [S1] https://arxiv.org/abs/2410.07095, v6 (26 Feb 2025). checked_on: 2026-11-12.
When to apply
Use when asking how agents fare on Kaggle-style machine-learning engineering, or what MLE-bench measures.
Facts
- id: F1 claim: "MLE-bench is a set of 75 Kaggle-style machine-learning engineering tasks with human leaderboard baselines." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F2 claim: "The authors report that o1-preview with AIDE scaffolding reaches bronze-medal level on 16.9% of competitions." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F3 claim: "Code is at https://github.com/openai/mle-bench/." confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
Sources
- [S1] arXiv 2410.07095, https://arxiv.org/abs/2410.07095. License: CC BY 4.0. v6 (26 Feb 2025). Accessed 2026-11-12.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.
License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, and 9 coauthors, MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (CC BY 4.0).
Cite this
'iinventedlight (AI-assisted)'. 2026. "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering." iinventedlight Library, iil:note:gwnewhp2, version of 2026-11-12. https://iinventedlight.com/library/notes/2410-07095-mle-bench/. License: CC-BY-4.0.
@misc{iil_note_gwnewhp2,
author = {{'iinventedlight (AI-assisted)'}},
title = {{MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2410-07095-mle-bench/},
note = {Permanent ID iil:note:gwnewhp2. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2410-07095-mle-bench.md.