← All notes

Summary

Evaluating Large Language Models Trained on Code (2021), by Mark Chen, Jerry Tworek, Heewoo Jun, and many coauthors, is at https://arxiv.org/abs/2107.03374. Codex is described as a GPT model fine-tuned on publicly available GitHub code; the authors also note that a distinct production version powers GitHub Copilot. HumanEval is a new evaluation set for functional correctness when synthesizing programs from docstrings. They report that their model solves 28.8% of the problems, compared with 0% for GPT 3 and 11.4% for GPT J. They also find that repeated sampling is surprisingly effective, solving 70.2% of problems with 100 samples per problem. They discuss limits such as difficulty with long chains of operations in docstrings, and they discuss broader impacts around safety, security, and economics. Furthermore, they find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, they solve 70.2% of the problems with 100 samples per problem. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v2 (14 Jul 2021), submitted 7 Jul 2021.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it introduced both Codex and HumanEval, which became the standard starting point for measuring code generation.
• Reproduced: not reproduced (2026-11-11).
• Related work: A related note already in the library is SWE-bench.
• Limits: This is a pointer note only. The 28.8%, 0%, 11.4%, and 70.2% figures are the authors' reported HumanEval results from the v2 paper (14 Jul 2021).
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 7 Jul 2021 (v1); last revised 14 Jul 2021 (v2). Sources: [S1] https://arxiv.org/abs/2107.03374, v2 (14 Jul 2021). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original Codex and HumanEval paper, the usual starting point for code-generation evaluation.

Facts

• id: F1 claim: 'The paper introduces Codex, a GPT language model fine-tuned on publicly available code from GitHub, and HumanEval, an evaluation set for synthesizing programs from docstrings.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors report that their model solves 28.8% of HumanEval problems, compared with 0% for GPT-3 and 11.4% for GPT-J.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'With 100 samples per problem they report solving 70.2% of the problems.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2107.03374, https://arxiv.org/abs/2107.03374. Paper license: arXiv default (non-exclusive distribution license). v2 (14 Jul 2021). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_b17djbnb,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Evaluating Large Language Models Trained on Code}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2107-03374-humaneval/},
  note = {Permanent ID iil:note:b17djbnb. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2107-03374-humaneval.md.