---
source: https://iinventedlight.com/library/notes/ao-cost-speculative-decoding.md
title: "Speculative decoding and speculative sampling"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2211.17192] Fast Inference from Transformers via Speculative Decoding, CC BY 4.0; arXiv, [2302.01318] Accelerating Large Language Model Decoding with Speculative Sampling, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: inference cost"
tags: ['llm-inference', 'decoding', 'latency', 'speculative-decoding']
sources:
  - id: S1
    title: "[2211.17192] Fast Inference from Transformers via Speculative Decoding"
    publisher: arXiv
    url: https://arxiv.org/abs/2211.17192
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2302.01318] Accelerating Large Language Model Decoding with Speculative Sampling"
    publisher: arXiv
    url: https://arxiv.org/abs/2302.01318
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Speculative decoding and speculative sampling

**Answer:** Speculative decoding makes large autoregressive models faster without changing their outputs: a cheaper model proposes tokens and the large model checks several of them in parallel. Speculative sampling achieves a 2-2.5x decoding speedup with Chinchilla.

## Why decoding is slow

Inference from large autoregressive models like Transformers is slow: decoding K tokens takes K serial runs of the model (S1).

## Speculative decoding (S1)

- Hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models.
- Using speculative execution and a novel sampling method, exact decoding from the large models becomes faster by running them in parallel on the outputs of the approximation models, without changing the distribution.
- It can accelerate existing off-the-shelf models without retraining or architecture changes.
- On T5-XXL it showed a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

## Speculative sampling (S2)

- The algorithm enables the generation of multiple tokens from each transformer call.
- It relies on the observation that the latency of parallel scoring of short continuations, generated by a faster draft model, is comparable to that of sampling a single token from the larger target model.
- A novel modified rejection sampling scheme preserves the distribution of the target model within hardware numerics.
- With Chinchilla, a 70 billion parameter language model, it achieved a 2-2.5x decoding speedup in a distributed setup, without compromising sample quality or modifying the model.

## Takeaway for operators

S1 makes decoding faster without changing the distribution, and S2 preserves the distribution of the target model within hardware numerics. S1 runs approximation models; S2 uses a faster but less powerful draft model.

## Claims
- id: C1
  claim: Decoding K tokens takes K serial runs of the model.
  source: S1
  evidence: "decoding K tokens takes K serial runs of the model."
- id: C2
  claim: Speculative decoding needs no retraining or architecture changes.
  source: S1
  evidence: "Our method can accelerate existing off-the-shelf models without retraining or architecture changes."
- id: C3
  claim: Speculative sampling was benchmarked with Chinchilla.
  source: S2
  evidence: "We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model"

## Sources
- [S1] [2211.17192] Fast Inference from Transformers via Speculative Decoding. arXiv. https://arxiv.org/abs/2211.17192 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2302.01318] Accelerating Large Language Model Decoding with Speculative Sampling. arXiv. https://arxiv.org/abs/2302.01318 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).