← All notes

Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

Summary

Distilling Step-by-Step (2023) shows that a small model can beat a much larger prompted model if it is trained on the large model's rationales, not only on its labels, and if that training uses less data than ordinary finetuning or distillation.

What it introduced: a multi-task setup that extracts rationales from a large model and uses them as extra supervision for a small student. Across four NLP benchmarks, the method beats both finetuning and label distillation with fewer examples, and it beats few-shot prompted large models with a much smaller student. In one result, a 770 million parameter T5 outperforms a few-shot 540 billion parameter PaLM using only 80% of the available data, while standard finetuning of the same T5 does not catch up even with the full set.

What it led to: later rationale-distillation and small-reasoner recipes. Live notes: 2201-11903-chain-of-thought-prompting and 2210-11416-flan-instruction-finetuning.

Where to read it: https://arxiv.org/abs/2305.02301 (v2, last revised 5 Jul 2023). Findings of ACL 2023. Code: https://github.com/google-research/distilling-step-by-step.

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, and 6 coauthors, Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:xbbjrmq7

'iinventedlight (AI-assisted)'. 2026. "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes." iinventedlight Library, iil:note:xbbjrmq7, version of 2026-11-12. https://iinventedlight.com/library/notes/2305-02301-distilling-step-by-step/. License: CC-BY-4.0.

@misc{iil_note_xbbjrmq7,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2305-02301-distilling-step-by-step/},
  note = {Permanent ID iil:note:xbbjrmq7. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2305-02301-distilling-step-by-step.md.