Extracting Training Data from Large Language Models
Summary
Extracting Training Data from Large Language Models (2020) showed that a large language model can memorize and emit individual training examples when queried, even if those examples are not obviously present in ordinary use. The authors demonstrate the attack on GPT-2 and recover many training sequences.
What it introduced: a concrete training-data extraction attack against a large autoregressive language model, with evidence that memorization is not only a theoretical concern for models trained on broad web scrapes.
What it led to: later measurements of memorization as models scale, and extraction work on production systems. This is a pointer to the original demonstration, not a how-to. Live notes on related questions include 2401-05566-sleeper-agents and 2412-14093-alignment-faking.
Use it when you want the original extraction result on GPT-2. Where to read it: https://arxiv.org/abs/2012.07805 (v2 (15 Jun 2021)).
The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Nicholas Carlini, Florian Tramer, Eric Wallace, and 9 coauthors (listed on the arXiv abs page), Extracting Training Data from Large Language Models (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "Extracting Training Data from Large Language Models." iinventedlight Library, iil:note:m1kjy19y, version of 2026-11-12. https://iinventedlight.com/library/notes/2012-07805-extracting-training-data/. License: CC-BY-4.0.
@misc{iil_note_m1kjy19y,
author = {{'iinventedlight (AI-assisted)'}},
title = {{Extracting Training Data from Large Language Models}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2012-07805-extracting-training-data/},
note = {Permanent ID iil:note:m1kjy19y. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2012-07805-extracting-training-data.md.