--- title: 'Ring Attention with Blockwise Transformers for Near-Infinite Context' license: CC-BY-4.0 license_url: https://creativecommons.org/licenses/by/4.0/ license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).' source: https://iinventedlight.com/library/notes/2310-01889-ring-attention.md source_license: 'arXiv default (non-exclusive distribution license)' attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Hao Liu, Matei Zaharia, Pieter Abbeel, Ring Attention with Blockwise Transformers for Near-Infinite Context (arXiv). This note does not reproduce the paper.' author: 'iinventedlight (AI-assisted)' date: 2026-11-11 status: published approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\"" checked_on: 2026-11-11 as_of: 2026-11-11 confidence: medium untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.' category: arxiv topic_area: 'ai-research: memory-and-context' tags: arxiv, pointer, memory-and-context arxiv_id: 2310.01889 alias: 2310.01889-ring-attention format: pointer type: pointer sources: - id: S1 title: 'Ring Attention with Blockwise Transformers for Near-Infinite Context' publisher: arXiv authors: 'Hao Liu, Matei Zaharia, Pieter Abbeel' url: https://arxiv.org/abs/2310.01889 version_read: 'v4 (27 Nov 2023)' accessed: 2026-11-11 license: 'arXiv default (non-exclusive distribution license)' --- # Ring Attention with Blockwise Transformers for Near-Infinite Context ## Summary Ring Attention with Blockwise Transformers for Near-Infinite Context (2023), by Hao Liu, Matei Zaharia, and Pieter Abbeel, is at https://arxiv.org/abs/2310.01889. The method computes self-attention and feedforward in blocks, distributes long sequences over many devices, and overlaps communication of key-value blocks with the blockwise attention work. The authors say this lets training and inference use sequences up to the device count times longer than earlier memory-efficient Transformers, without approximations and without added communication or compute overhead. They report experiments on language modeling and reinforcement learning, including context sizes of millions of tokens. The comments include a GitHub link for code. The approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of the approach in allowing millions of tokens context size. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v4 (27 Nov 2023), submitted 3 Oct 2023. ## Claims Pointer notes do not quote the paper. See Verification and Sources for the link and license. ## Verification - Format: pointer note only. The paper is not reproduced here. - Why it matters: It matters because it made very long context more feasible by spreading attention across devices rather than shrinking the sequence with approximations. - Reproduced: not reproduced (2026-11-11). - Related work: Related notes already in the library include [Million-token context windows](https://iinventedlight.com/library/notes/ao-context-million-token-windows/) and [RULER](https://iinventedlight.com/library/notes/2404-06654-ruler/). - Limits: This is a pointer note only. Claims about sequence length scaling and millions of tokens are the authors' reported results from the v4 paper (27 Nov 2023). The comments name https://github.com/lhao499/llm_large_context, which redirected to https://github.com/haoliuhl/ringattention on 2026-11-11. - License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 3 Oct 2023 (v1); last revised 27 Nov 2023 (v4). Sources: [S1] https://arxiv.org/abs/2310.01889, v4 (27 Nov 2023). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0. ## When to apply Use when you need the original Ring Attention paper on distributing long-context attention across devices. ## Facts - id: F1 claim: 'Ring Attention uses blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices, overlapping communication of key-value blocks with computation.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0 - id: F2 claim: 'The authors say the approach enables training and inference of sequences up to device count times longer than prior memory-efficient Transformers, without approximations or extra communication and computation overheads.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0 - id: F3 claim: 'They point to code at the GitHub URL named in the paper comments, which later redirected to https://github.com/haoliuhl/ringattention.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0 ## Sources - [S1] arXiv 2310.01889, https://arxiv.org/abs/2310.01889. Paper license: arXiv default (non-exclusive distribution license). v4 (27 Nov 2023). Accessed 2026-11-11. This note links to the paper and does not reproduce it. License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. --- Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).