---
title: 'Fast Inference from Transformers via Speculative Decoding'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-11; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2211-17192-speculative-decoding.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Yaniv Leviathan, Matan Kalman, Yossi Matias, Fast Inference from Transformers via Speculative Decoding (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Not reproduced. Abs page and HTML key sections read 2026-11-11. Named code URLs checked; code not run.'
category: arxiv
topic_area: 'ai-research: efficiency'
tags: arxiv, efficiency, inference
arxiv_id: 2211.17192
alias: 2211.17192-speculative-decoding
sources:
  - id: S1
    title: 'Fast Inference from Transformers via Speculative Decoding'
    publisher: arXiv
    authors: 'Yaniv Leviathan, Matan Kalman, Yossi Matias'
    url: https://arxiv.org/abs/2211.17192
    version_read: 'v2 (18 May 2023)'
    accessed: 2026-11-11
    license: 'CC BY 4.0'
---
# Fast Inference from Transformers via Speculative Decoding

## Summary
Speculative decoding is an algorithm for faster sampling from autoregressive models without changing the outputs. It computes several tokens in parallel by running the large model on the outputs of an approximation model, potentially generating several tokens concurrently, and without changing the distribution. The method can accelerate existing off-the-shelf models without retraining or architecture changes. On T5-XXL they report a 2X-3X acceleration compared with the standard T5X implementation, with identical outputs. [S1]

## Claims
- id: C1
  claim: 'Speculative decoding samples from autoregressive models faster without any changes to the outputs.'
  source: S1
  evidence: "an algorithm to sample from autoregressive models faster without any changes to the outputs"
- id: C2
  claim: 'The method can accelerate existing off-the-shelf models without retraining or architecture changes.'
  source: S1
  evidence: "accelerate existing off-the-shelf models without retraining or architecture changes"
- id: C3
  claim: 'On T5-XXL they report a 2X-3X acceleration with identical outputs.'
  source: S1
  evidence: "a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs"

## Verification
- Code available: No official code URL was named on the abs page or HTML.
- Reproduced: not reproduced (2026-11-11). Code not run.
- Related work: Live notes ao-cost-speculative-decoding and ao-cost-multi-head-decoding. This is the paper-verification note for this speculative decoding paper itself.
- Limits: The 2X-3X figure is for T5-XXL versus T5X in the authors' study. Speedups depend on how often the draft model is right and on the hardware setup.
- License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 30 Nov 2022 (v1); last revised 18 May 2023 (v2). Sources: [S1] https://arxiv.org/abs/2211.17192, v2 (18 May 2023). checked_on: 2026-11-11.

## When to apply
Use when looking at the original speculative-decoding method for faster inference that preserves the large model's distribution.

## Facts
- id: F1
  claim: 'Speculative decoding speeds up autoregressive sampling without changing the output distribution.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: 'The authors say it can be applied to existing models without retraining or architecture changes.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: 'They report a 2X-3X acceleration on T5-XXL versus the standard T5X implementation, with identical outputs.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2211.17192, https://arxiv.org/abs/2211.17192. License: CC BY 4.0. v2 (18 May 2023). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).