---
source: https://iinventedlight.com/library/notes/2401-04088-mixtral-of-experts.md
title: "Mixtral of Experts (Mixtral 8x7B)"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, Mixtral of Experts, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-05
as_of: 2026-11-05
confidence: medium
untested: "Results were not reproduced; only the arXiv abstract page was read."
category: ai-research
topic_area: "ai-research: sparse mixture-of-experts models"
tags: llm, mixture-of-experts, moe, open-weights, arxiv
sources:
  - id: S1
    title: "Mixtral of Experts"
    publisher: arXiv
    url: https://arxiv.org/abs/2401.04088
    accessed: 2026-11-05
    license: "CC BY 4.0"
---
# Mixtral of Experts (Mixtral 8x7B)

## Short answer
Mixtral 8x7B is a sparse mixture-of-experts model. Each layer has 8 feedforward experts, and a router sends each token to 2 of them. The model holds 47B parameters but uses only about 13B per token at inference. [S1]

## How it works
- Same base architecture as Mistral 7B, except each layer's feedforward block is replaced by 8 experts. [S1]
- For every token at every layer, a router picks two experts and combines their outputs. The chosen pair can change from token to token. [S1]
- Trained with a 32k-token context. [S1]

## Results reported
- Matches or beats Llama 2 70B and GPT-3.5 on the benchmarks evaluated, with large leads in math, code and multilingual tasks over Llama 2 70B. [S1]
- An instruction-tuned version (Mixtral 8x7B - Instruct) was also released. [S1]
- Base and instruct weights are under Apache 2.0. [S1]

## Why it matters
Sparse routing keeps inference cost close to that of a much smaller dense model while giving the model access to far more parameters.

## Claims
- id: C1
  claim: Each Mixtral layer is composed of 8 feedforward blocks (experts).
  source: S1
  evidence: "each layer is composed of 8 feedforward blocks (i.e. experts)"
- id: C2
  claim: A router network selects two experts per token at each layer.
  source: S1
  evidence: "a router network selects two experts to process the current state"
- id: C3
  claim: Each token has access to 47B parameters but uses 13B active parameters.
  source: S1
  evidence: "has access to 47B parameters, but only uses 13B active parameters"
- id: C4
  claim: Both base and instruct models are released under Apache 2.0.
  source: S1
  evidence: "released under the Apache 2.0 license"

## Sources
- [S1] Mixtral of Experts. arXiv. https://arxiv.org/abs/2401.04088 . License: CC BY 4.0. Accessed 2026-11-05.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).