---
title: 'Evaluating Large Language Models Trained on Code'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2107-03374-humaneval.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. (58 authors), Evaluating Large Language Models Trained on Code (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: evals'
tags: arxiv, pointer, evals, code
arxiv_id: 2107.03374
alias: 2107.03374-humaneval
format: pointer
type: pointer
sources:
  - id: S1
    title: 'Evaluating Large Language Models Trained on Code'
    publisher: arXiv
    authors: 'Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. (58 authors)'
    url: https://arxiv.org/abs/2107.03374
    version_read: 'v2 (14 Jul 2021)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# Evaluating Large Language Models Trained on Code

## Summary
Evaluating Large Language Models Trained on Code (2021), by Mark Chen, Jerry Tworek, Heewoo Jun, and many coauthors, is at https://arxiv.org/abs/2107.03374. Codex is described as a GPT model fine-tuned on publicly available GitHub code; the authors also note that a distinct production version powers GitHub Copilot. HumanEval is a new evaluation set for functional correctness when synthesizing programs from docstrings. They report that their model solves 28.8% of the problems, compared with 0% for GPT 3 and 11.4% for GPT J. They also find that repeated sampling is surprisingly effective, solving 70.2% of problems with 100 samples per problem. They discuss limits such as difficulty with long chains of operations in docstrings, and they discuss broader impacts around safety, security, and economics. Furthermore, they find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, they solve 70.2% of the problems with 100 samples per problem. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v2 (14 Jul 2021), submitted 7 Jul 2021.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters because it introduced both Codex and HumanEval, which became the standard starting point for measuring code generation.
- Reproduced: not reproduced (2026-11-11).
- Related work: A related note already in the library is [SWE-bench](https://iinventedlight.com/library/notes/2310-06770-swe-bench/).
- Limits: This is a pointer note only. The 28.8%, 0%, 11.4%, and 70.2% figures are the authors' reported HumanEval results from the v2 paper (14 Jul 2021).
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 7 Jul 2021 (v1); last revised 14 Jul 2021 (v2). Sources: [S1] https://arxiv.org/abs/2107.03374, v2 (14 Jul 2021). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when you need the original Codex and HumanEval paper, the usual starting point for code-generation evaluation.

## Facts
- id: F1
  claim: 'The paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and HumanEval, an evaluation set for synthesizing programs from docstrings.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors report that their model solves 28.8% of HumanEval problems, compared with 0% for GPT-3 and 11.4% for GPT-J.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'With 100 samples per problem they report solving 70.2% of the problems.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2107.03374, https://arxiv.org/abs/2107.03374. Paper license: arXiv default (non-exclusive distribution license). v2 (14 Jul 2021). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).