← All notes

Answer: MLE-bench measures how well AI agents perform at machine learning engineering, using 75 Kaggle competitions; its best setup, o1-preview with AIDE scaffolding, reached at least a bronze medal level in 16.9% of competitions (S1). Agent Laboratory is an autonomous LLM-based framework capable of completing the entire research process (S2).

MLE-bench (S1)

• Its tasks test real-world ML engineering skills such as training models, preparing datasets, and running experiments.
• Human baselines for each competition come from Kaggle's publicly available leaderboards.
• Several frontier language models were evaluated with open-source agent scaffolds.
• The authors also investigate resource scaling for AI agents and the impact of contamination from pre-training.
• The benchmark code is open-source.

Agent Laboratory (S2)

• It is an autonomous LLM-based framework capable of completing the entire research process.
• It progresses through three stages, literature review, experimentation, and report writing, producing a code repository and a research report.
• Users can provide feedback and guidance at each stage.
• Agent Laboratory driven by o1-preview generated the best research outcomes.
• Human involvement, providing feedback at each stage, significantly improves the overall quality of research.
• It achieved an 84% decrease in research expenses compared to previous autonomous research methods.

For evaluation design

MLE-bench establishes human baselines using Kaggle's publicly available leaderboards (S1). Agent Laboratory has multiple researchers assess its quality through a survey (S2).

Claims

• id: C1 claim: MLE-bench uses 75 Kaggle competitions. source: S1 evidence: "we curate 75 ML engineering-related competitions from Kaggle"
• id: C2 claim: Human baselines come from Kaggle leaderboards. source: S1 evidence: "We establish human baselines for each competition using Kaggle's publicly available leaderboards."
• id: C3 claim: Agent Laboratory completes the entire research process. source: S2 evidence: "an autonomous LLM-based framework capable of completing the entire research process."

Sources

• [S1] [2410.07095] MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv. https://arxiv.org/abs/2410.07095 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2501.04227] Agent Laboratory: Using LLM Agents as Research Assistants. arXiv. https://arxiv.org/abs/2501.04227 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_sfa0bys1,
  author = {{iinventedlight}},
  title = {{Agents doing ML engineering and research: MLE-bench and Agent Laboratory}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-evals-ml-engineering-agents/},
  note = {Permanent ID iil:note:sfa0bys1. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-evals-ml-engineering-agents.md.