← All notes

Answer: Speculative decoding makes large autoregressive models faster without changing their outputs: a cheaper model proposes tokens and the large model checks several of them in parallel. Speculative sampling achieves a 2-2.5x decoding speedup with Chinchilla.

Why decoding is slow

Inference from large autoregressive models like Transformers is slow: decoding K tokens takes K serial runs of the model (S1).

Speculative decoding (S1)

• Hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models.
• Using speculative execution and a novel sampling method, exact decoding from the large models becomes faster by running them in parallel on the outputs of the approximation models, without changing the distribution.
• It can accelerate existing off-the-shelf models without retraining or architecture changes.
• On T5-XXL it showed a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

Speculative sampling (S2)

• The algorithm enables the generation of multiple tokens from each transformer call.
• It relies on the observation that the latency of parallel scoring of short continuations, generated by a faster draft model, is comparable to that of sampling a single token from the larger target model.
• A novel modified rejection sampling scheme preserves the distribution of the target model within hardware numerics.
• With Chinchilla, a 70 billion parameter language model, it achieved a 2-2.5x decoding speedup in a distributed setup, without compromising sample quality or modifying the model.

Takeaway for operators

S1 makes decoding faster without changing the distribution, and S2 preserves the distribution of the target model within hardware numerics. S1 runs approximation models; S2 uses a faster but less powerful draft model.

Claims

• id: C1 claim: Decoding K tokens takes K serial runs of the model. source: S1 evidence: "decoding K tokens takes K serial runs of the model."
• id: C2 claim: Speculative decoding needs no retraining or architecture changes. source: S1 evidence: "Our method can accelerate existing off-the-shelf models without retraining or architecture changes."
• id: C3 claim: Speculative sampling was benchmarked with Chinchilla. source: S2 evidence: "We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model"

Sources

• [S1] [2211.17192] Fast Inference from Transformers via Speculative Decoding. arXiv. https://arxiv.org/abs/2211.17192 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2302.01318] Accelerating Large Language Model Decoding with Speculative Sampling. arXiv. https://arxiv.org/abs/2302.01318 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_zh0m831r,
  author = {{iinventedlight}},
  title = {{Speculative decoding and speculative sampling}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-cost-speculative-decoding/},
  note = {Permanent ID iil:note:zh0m831r. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-cost-speculative-decoding.md.