← All notes

Answer: Medusa speeds up LLM inference by adding extra decoding heads that predict multiple subsequent tokens in parallel, avoiding the separate draft model that speculative decoding needs. Both methods attack the same bottleneck: auto-regressive decoding requires sequential computation, with each step reliant on the previous one's output.

The bottleneck

Each decoding step necessitates moving the full model parameters from High-Bandwidth Memory (HBM) to the accelerator's cache (S1). Decoding K tokens takes K serial runs of the model (S2).

Speculative decoding (S2)

• Speculative decoding samples from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel.
• Hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models.
• It can accelerate existing off-the-shelf models without retraining or architecture changes.
• On T5-XXL, the authors report a 2X-3X acceleration compared to the standard T5X implementation, with identical outputs.

Medusa (S1)

• Speculative decoding's implementation is impeded by the challenges of acquiring and maintaining a separate draft model.
• Using a tree-based attention mechanism, Medusa constructs multiple candidate continuations and verifies them simultaneously in each decoding step.
• Medusa-1 is fine-tuned on top of a frozen backbone LLM, enabling lossless inference acceleration.
• Medusa-2 is fine-tuned together with the backbone LLM, giving higher speedup but needing a special training recipe that preserves the backbone model's capabilities.
• Extensions include self-distillation for when no training data is available and a typical acceptance scheme to boost the acceptance rate.

Choosing

Speculative decoding can accelerate existing off-the-shelf models without retraining (S2). Medusa avoids acquiring and maintaining a separate draft model, but its heads are fine-tuned (S1).

Claims

• id: C1 claim: Medusa adds extra decoding heads to predict multiple tokens in parallel. source: S1 evidence: "Medusa substantially reduces the number of decoding steps required."
• id: C2 claim: Medusa-1 enables lossless inference acceleration on a frozen backbone. source: S1 evidence: "Medusa is directly fine-tuned on top of a frozen backbone LLM, enabling lossless inference acceleration."
• id: C3 claim: Speculative decoding needs no retraining or architecture changes. source: S2 evidence: "Our method can accelerate existing off-the-shelf models without retraining or architecture changes."

Sources

• [S1] [2401.10774] Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv. https://arxiv.org/abs/2401.10774 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2211.17192] Fast Inference from Transformers via Speculative Decoding. arXiv. https://arxiv.org/abs/2211.17192 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_6r516xne,
  author = {{iinventedlight}},
  title = {{Multi-head decoding (Medusa) vs. speculative decoding}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-cost-multi-head-decoding/},
  note = {Permanent ID iil:note:6r516xne. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-cost-multi-head-decoding.md.