---
title: 'Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2404-07143-infini-attention.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal, Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: memory-and-context'
tags: arxiv, pointer, memory-and-context
arxiv_id: 2404.07143
alias: 2404.07143-infini-attention
format: pointer
type: pointer
sources:
  - id: S1
    title: 'Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention'
    publisher: arXiv
    authors: 'Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal'
    url: https://arxiv.org/abs/2404.07143
    version_read: 'v2 (9 Aug 2024)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

## Summary
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention (2024), by Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal, is at https://arxiv.org/abs/2404.07143. Infini-attention places a compressive memory inside ordinary attention and, in one Transformer block, combines masked local attention with a long-term linear attention mechanism. The authors report results on long-context language modeling, on passkey retrieval at 1M sequence length, and on book summarization at 500K length, using 1B and 8B models. The approach introduces minimal bounded memory parameters and enables fast streaming inference. They demonstrate the approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. A key component is a new attention technique dubbed Infini-attention, which incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v2 (9 Aug 2024), submitted 10 Apr 2024.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters as a compact proposal for running Transformers over very long streams with bounded memory, by compressing older context rather than keeping a full key-value cache forever.
- Reproduced: not reproduced (2026-11-11).
- Related work: Related notes already in the library include [Million-token context windows](https://iinventedlight.com/library/notes/ao-context-million-token-windows/) and [StreamingLLM](https://iinventedlight.com/library/notes/2309-17453-streamingllm-attention-sinks/).
- Limits: This is a pointer note only. The 1M passkey and 500K book-summarization figures are the authors' reported results from the v2 paper (9 Aug 2024).
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 10 Apr 2024 (v1); last revised 9 Aug 2024 (v2). Sources: [S1] https://arxiv.org/abs/2404.07143, v2 (9 Aug 2024). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when you need the original Infini-attention paper on compressive memory inside a Transformer block for very long streams.

## Facts
- id: F1
  claim: 'Infini-attention incorporates a compressive memory into vanilla attention and combines masked local attention with long-term linear attention in a single Transformer block.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors demonstrate the approach on long-context language modeling, 1M sequence length passkey context block retrieval, and 500K length book summarization with 1B and 8B LLMs.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'They say the method introduces minimal bounded memory parameters and enables fast streaming inference.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2404.07143, https://arxiv.org/abs/2404.07143. Paper license: arXiv default (non-exclusive distribution license). v2 (9 Aug 2024). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).