← All notes

Jamba: A Hybrid Transformer-Mamba Language Model

Summary

Jamba: A Hybrid Transformer-Mamba Language Model (2024) introduced a large hybrid that interleaves Transformer attention with Mamba state-space layers, and adds mixture-of-experts in some layers so capacity can grow without activating every parameter on every token.

What it introduced: a flexible hybrid stack in which those two layer types, plus optional experts, can be mixed for a given compute budget. The configuration the authors implemented is described as fitting on a single 80GB GPU, with high throughput and a smaller memory footprint than a vanilla Transformer, and with reported strength on standard benchmarks and on long context, including results out to 256K tokens. They also discuss which architectural choices mattered at scale and planned to release ablation checkpoints.

What it led to: later hybrid and state-space language models that try to keep long context cheaper than full attention. Live notes include 2312-00752-mamba and 1706-03762-attention-is-all-you-need. The authors said the Jamba weights would be released under a permissive license; see the paper for the terms that actually apply.

Where to read it: https://arxiv.org/abs/2403.19887 (v2, last revised 3 Jul 2024). A project page is noted at https://www.ai21.com/jamba.

The paper is posted under CC BY-SA 4.0. This note links to it rather than reproducing it, so the note can stay CC BY 4.0. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: CC BY-SA 4.0; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Opher Lieber, Barak Lenz, Hofit Bata, and 19 coauthors, Jamba: A Hybrid Transformer-Mamba Language Model (CC BY-SA 4.0). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:6zaj3hh8

'iinventedlight (AI-assisted)'. 2026. "Jamba: A Hybrid Transformer-Mamba Language Model." iinventedlight Library, iil:note:6zaj3hh8, version of 2026-11-12. https://iinventedlight.com/library/notes/2403-19887-jamba/. License: CC-BY-4.0.

@misc{iil_note_6zaj3hh8,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Jamba: A Hybrid Transformer-Mamba Language Model}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2403-19887-jamba/},
  note = {Permanent ID iil:note:6zaj3hh8. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2403-19887-jamba.md.