DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Summary
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024) is the report that introduced the efficient DeepSeek architecture later scaled in V3 and R1. The model has 236 billion parameters in total, activates 21 billion per token, and supports a 128K context.
What it introduced: Multi-head Latent Attention, which compresses the key-value cache into a latent vector, and DeepSeekMoE, a sparse expert design aimed at cheaper training. Compared with DeepSeek 67B, the authors report stronger quality, 42.5% lower training cost, a 93.3% smaller KV cache, and up to 5.76 times higher maximum generation throughput. The model was pretrained on 8.1 trillion tokens, then fine-tuned and trained with reinforcement learning.
What it led to: the backbone used in DeepSeek-V3 and in the later DeepSeek-R1 reasoning models. Live notes: 2412-19437-deepseek-v3, 2501-12948-deepseek-r1, and 2402-03300-deepseekmath.
Where to read it: https://arxiv.org/abs/2405.04434 (v5, last revised 19 Jun 2024).
The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: DeepSeek-AI and 156 coauthors (listed on the arXiv abs page), DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model." iinventedlight Library, iil:note:d799gtk3, version of 2026-11-12. https://iinventedlight.com/library/notes/2405-04434-deepseek-v2/. License: CC-BY-4.0.
@misc{iil_note_d799gtk3,
author = {{'iinventedlight (AI-assisted)'}},
title = {{DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2405-04434-deepseek-v2/},
note = {Permanent ID iil:note:d799gtk3. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2405-04434-deepseek-v2.md.