---
source: https://iinventedlight.com/library/notes/ao-evals-ml-engineering-agents.md
title: "Agents doing ML engineering and research: MLE-bench and Agent Laboratory"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2410.07095] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, CC BY 4.0; arXiv, [2501.04227] Agent Laboratory: Using LLM Agents as Research Assistants, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: evaluation"
tags: ['agents', 'benchmarks', 'ml-engineering', 'research-agents']
sources:
  - id: S1
    title: "[2410.07095] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering"
    publisher: arXiv
    url: https://arxiv.org/abs/2410.07095
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2501.04227] Agent Laboratory: Using LLM Agents as Research Assistants"
    publisher: arXiv
    url: https://arxiv.org/abs/2501.04227
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Agents doing ML engineering and research: MLE-bench and Agent Laboratory

**Answer:** MLE-bench measures how well AI agents perform at machine learning engineering, using 75 Kaggle competitions; its best setup, o1-preview with AIDE scaffolding, reached at least a bronze medal level in 16.9% of competitions (S1). Agent Laboratory is an autonomous LLM-based framework capable of completing the entire research process (S2).

## MLE-bench (S1)

- Its tasks test real-world ML engineering skills such as training models, preparing datasets, and running experiments.
- Human baselines for each competition come from Kaggle's publicly available leaderboards.
- Several frontier language models were evaluated with open-source agent scaffolds.
- The authors also investigate resource scaling for AI agents and the impact of contamination from pre-training.
- The benchmark code is open-source.

## Agent Laboratory (S2)

- It is an autonomous LLM-based framework capable of completing the entire research process.
- It progresses through three stages, literature review, experimentation, and report writing, producing a code repository and a research report.
- Users can provide feedback and guidance at each stage.
- Agent Laboratory driven by o1-preview generated the best research outcomes.
- Human involvement, providing feedback at each stage, significantly improves the overall quality of research.
- It achieved an 84% decrease in research expenses compared to previous autonomous research methods.

## For evaluation design

MLE-bench establishes human baselines using Kaggle's publicly available leaderboards (S1). Agent Laboratory has multiple researchers assess its quality through a survey (S2).

## Claims
- id: C1
  claim: MLE-bench uses 75 Kaggle competitions.
  source: S1
  evidence: "we curate 75 ML engineering-related competitions from Kaggle"
- id: C2
  claim: Human baselines come from Kaggle leaderboards.
  source: S1
  evidence: "We establish human baselines for each competition using Kaggle's publicly available leaderboards."
- id: C3
  claim: Agent Laboratory completes the entire research process.
  source: S2
  evidence: "an autonomous LLM-based framework capable of completing the entire research process."

## Sources
- [S1] [2410.07095] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv. https://arxiv.org/abs/2410.07095 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2501.04227] Agent Laboratory: Using LLM Agents as Research Assistants. arXiv. https://arxiv.org/abs/2501.04227 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).