← All notes

Short answer

Mixtral 8x7B is a sparse mixture-of-experts model. Each layer has 8 feedforward experts, and a router sends each token to 2 of them. The model holds 47B parameters but uses only about 13B per token at inference. [S1]

How it works

• Same base architecture as Mistral 7B, except each layer's feedforward block is replaced by 8 experts. [S1]
• For every token at every layer, a router picks two experts and combines their outputs. The chosen pair can change from token to token. [S1]
• Trained with a 32k-token context. [S1]

Results reported

• Matches or beats Llama 2 70B and GPT-3.5 on the benchmarks evaluated, with large leads in math, code and multilingual tasks over Llama 2 70B. [S1]
• An instruction-tuned version (Mixtral 8x7B - Instruct) was also released. [S1]
• Base and instruct weights are under Apache 2.0. [S1]

Why it matters

Sparse routing keeps inference cost close to that of a much smaller dense model while giving the model access to far more parameters.

Claims

• id: C1 claim: Each Mixtral layer is composed of 8 feedforward blocks (experts). source: S1 evidence: "each layer is composed of 8 feedforward blocks (i.e. experts)"
• id: C2 claim: A router network selects two experts per token at each layer. source: S1 evidence: "a router network selects two experts to process the current state"
• id: C3 claim: Each token has access to 47B parameters but uses 13B active parameters. source: S1 evidence: "has access to 47B parameters, but only uses 13B active parameters"
• id: C4 claim: Both base and instruct models are released under Apache 2.0. source: S1 evidence: "released under the Apache 2.0 license"

Sources

• [S1] Mixtral of Experts. arXiv. https://arxiv.org/abs/2401.04088 . License: CC BY 4.0. Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_h51h8v21,
  author = {{iinventedlight}},
  title = {{Mixtral of Experts (Mixtral 8x7B)}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/2401-04088-mixtral-of-experts/},
  note = {Permanent ID iil:note:h51h8v21. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/2401-04088-mixtral-of-experts.md.