← All notes

Summary

Speculative decoding is an algorithm for faster sampling from autoregressive models without changing the outputs. It computes several tokens in parallel by running the large model on the outputs of an approximation model, potentially generating several tokens concurrently, and without changing the distribution. The method can accelerate existing off-the-shelf models without retraining or architecture changes. On T5-XXL they report a 2X-3X acceleration compared with the standard T5X implementation, with identical outputs. [S1]

Claims

• id: C1 claim: 'Speculative decoding samples from autoregressive models faster without any changes to the outputs.' source: S1 evidence: "an algorithm to sample from autoregressive models faster without any changes to the outputs"
• id: C2 claim: 'The method can accelerate existing off-the-shelf models without retraining or architecture changes.' source: S1 evidence: "accelerate existing off-the-shelf models without retraining or architecture changes"
• id: C3 claim: 'On T5-XXL they report a 2X-3X acceleration with identical outputs.' source: S1 evidence: "a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs"

Verification

• Code available: No official code URL was named on the abs page or HTML.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes ao-cost-speculative-decoding and ao-cost-multi-head-decoding. This is the paper-verification note for this speculative decoding paper itself.
• Limits: The 2X-3X figure is for T5-XXL versus T5X in the authors' study. Speedups depend on how often the draft model is right and on the hardware setup.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 30 Nov 2022 (v1); last revised 18 May 2023 (v2). Sources: [S1] https://arxiv.org/abs/2211.17192, v2 (18 May 2023). checked_on: 2026-11-11.

When to apply

Use when looking at the original speculative-decoding method for faster inference that preserves the large model's distribution.

Facts

• id: F1 claim: 'Speculative decoding speeds up autoregressive sampling without changing the output distribution.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'The authors say it can be applied to existing models without retraining or architecture changes.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They report a 2X-3X acceleration on T5-XXL versus the standard T5X implementation, with identical outputs.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2211.17192, https://arxiv.org/abs/2211.17192. License: CC BY 4.0. v2 (18 May 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_39k7hnpw,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Fast Inference from Transformers via Speculative Decoding}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2211-17192-speculative-decoding/},
  note = {Permanent ID iil:note:39k7hnpw. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2211-17192-speculative-decoding.md.