Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Summary
This paper argues that Transformers and state-space models (SSMs) such as Mamba are closely related, and it develops a framework of theoretical connections between SSMs and attention variants through decompositions of structured semiseparable matrices. Using this state space duality (SSD) framework, the authors design a new architecture, Mamba-2, whose core layer is a refinement of Mamba's selective SSM that they report is 2-8X faster, while remaining competitive with Transformers on language modeling. [S1]
Claims
- id: C1 claim: 'The authors develop a state space duality (SSD) framework connecting SSMs and attention variants.' source: S1 evidence: "Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2)"
- id: C2 claim: 'Mamba-2''s core layer is a refinement of Mamba''s selective SSM.' source: S1 evidence: "an a refinement of Mamba's selective SSM"
- id: C3 claim: 'They report that this core layer is 2-8X faster, while remaining competitive with Transformers on language modeling.' source: S1 evidence: "that is 2-8X faster, while continuing to be competitive with Transformers on language modeling"
Verification
- Code available: Paper: https://github.com/state-spaces/mamba, which returned HTTP 200 on 2026-11-12. The HTML states that model code and pre-trained checkpoints are open-sourced there. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: Live note 2312-00752-mamba. This is the paper-verification note for the SSD / Mamba-2 paper itself.
- Limits: The 2-8X figure refers to the authors' comparison for the refined core layer, not a blanket claim about every model size or task (v1, 31 May 2024; comments: ICML 2024). Competitiveness with Transformers is author-reported on language modeling.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 31 May 2024 (v1). Sources: [S1] https://arxiv.org/abs/2405.21060, v1 (31 May 2024). checked_on: 2026-11-12.
When to apply
Use when asking how Mamba-2 relates to attention through state space duality, or when comparing SSM and Transformer architectures.
Facts
- id: F1 claim: 'The paper introduces a state space duality (SSD) framework connecting SSMs and variants of attention.' confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F2 claim: 'Mamba-2 is designed as a new architecture whose core layer refines Mamba''s selective SSM.' confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
- id: F3 claim: 'The authors report that this core layer is 2-8X faster and remains competitive with Transformers on language modeling.' confidence: medium date: 2026-11-12 sources: [S1] license: CC BY 4.0
Sources
- [S1] arXiv 2405.21060, https://arxiv.org/abs/2405.21060. License: CC BY 4.0. v1 (31 May 2024). Accessed 2026-11-12.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.
License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Tri Dao, Albert Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (CC BY 4.0).
Cite this
'iinventedlight (AI-assisted)'. 2026. "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." iinventedlight Library, iil:note:wg9f42a8, version of 2026-11-12. https://iinventedlight.com/library/notes/2405-21060-mamba-2/. License: CC-BY-4.0.
@misc{iil_note_wg9f42a8,
author = {{'iinventedlight (AI-assisted)'}},
title = {{Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2405-21060-mamba-2/},
note = {Permanent ID iil:note:wg9f42a8. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2405-21060-mamba-2.md.