---
title: "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-12; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2410-07095-mle-bench.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, and 9 coauthors, MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: "Not reproduced. Abs page and HTML key sections read 2026-11-12. Named code URLs checked for HTTP 200; code not run."
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, evals, ml-engineering, agents, mle-bench
arxiv_id: 2410.07095
alias: 2410-07095-mle-bench
sources:
  - id: S1
    title: "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering"
    publisher: arXiv
    authors: "Jun Shern Chan and 11 coauthors (listed on the arXiv abs page)"
    url: https://arxiv.org/abs/2410.07095
    version_read: "v6 (26 Feb 2025)"
    accessed: 2026-11-12
    license: 'CC BY 4.0'
---
# MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

## Summary
MLE-bench measures how well AI agents do machine-learning engineering. The authors curate 75 Kaggle competitions that test skills such as training models, preparing datasets, and running experiments, and they set human baselines from public Kaggle leaderboards. Using open-source agent scaffolds, they find that the best setup, OpenAI's o1-preview with AIDE scaffolding, reaches at least a Kaggle bronze-medal level in 16.9% of competitions. The paper also looks at resource scaling and at contamination from pre-training. Benchmark code is released on GitHub. [S1]

## Claims
- id: C1
  claim: "MLE-bench curates 75 Kaggle competitions as machine-learning engineering tasks."
  source: S1
  evidence: "we curate 75 ML engineering-related competitions from Kaggle"
- id: C2
  claim: "The best reported setup, o1-preview with AIDE, reaches bronze-medal level in 16.9% of competitions."
  source: S1
  evidence: "at least the level of a Kaggle bronze medal in 16.9% of competitions"
- id: C3
  claim: "Benchmark code is released on GitHub."
  source: S1
  evidence: "github.com/openai/mle-bench/"

## Verification
- Code available: Paper: github.com/openai/mle-bench/, which returned HTTP 200 at https://github.com/openai/mle-bench on 2026-11-12. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Related agent-engineering evals include [2310-06770-swe-bench](https://iinventedlight.com/library/notes/2310-06770-swe-bench/) and [ao-evals-ml-engineering-agents](https://iinventedlight.com/library/notes/ao-evals-ml-engineering-agents/).
- Limits: The 16.9% bronze-level figure is for one scaffold and one model in the v6 paper. Kaggle tasks are a slice of ML engineering, not all of it. The first seven authors share equal contribution (order randomized).
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 9 Oct 2024 (v1); last revised v6 (26 Feb 2025). Sources: [S1] https://arxiv.org/abs/2410.07095, v6 (26 Feb 2025). checked_on: 2026-11-12.

## When to apply
Use when asking how agents fare on Kaggle-style machine-learning engineering, or what MLE-bench measures.

## Facts
- id: F1
  claim: "MLE-bench is a set of 75 Kaggle-style machine-learning engineering tasks with human leaderboard baselines."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: "The authors report that o1-preview with AIDE scaffolding reaches bronze-medal level on 16.9% of competitions."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: "Code is at https://github.com/openai/mle-bench/."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2410.07095, https://arxiv.org/abs/2410.07095. License: CC BY 4.0. v6 (26 Feb 2025). Accessed 2026-11-12.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
