← All notes

Summary

Ring Attention with Blockwise Transformers for Near-Infinite Context (2023), by Hao Liu, Matei Zaharia, and Pieter Abbeel, is at https://arxiv.org/abs/2310.01889. The method computes self-attention and feedforward in blocks, distributes long sequences over many devices, and overlaps communication of key-value blocks with the blockwise attention work. The authors say this lets training and inference use sequences up to the device count times longer than earlier memory-efficient Transformers, without approximations and without added communication or compute overhead. They report experiments on language modeling and reinforcement learning, including context sizes of millions of tokens. The comments include a GitHub link for code. The approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of the approach in allowing millions of tokens context size. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v4 (27 Nov 2023), submitted 3 Oct 2023.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it made very long context more feasible by spreading attention across devices rather than shrinking the sequence with approximations.
• Reproduced: not reproduced (2026-11-11).
• Related work: Related notes already in the library include Million-token context windows and RULER.
• Limits: This is a pointer note only. Claims about sequence length scaling and millions of tokens are the authors' reported results from the v4 paper (27 Nov 2023). The comments name https://github.com/lhao499/llm_large_context, which redirected to https://github.com/haoliuhl/ringattention on 2026-11-11.
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 3 Oct 2023 (v1); last revised 27 Nov 2023 (v4). Sources: [S1] https://arxiv.org/abs/2310.01889, v4 (27 Nov 2023). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original Ring Attention paper on distributing long-context attention across devices.

Facts

• id: F1 claim: 'Ring Attention uses blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices, overlapping communication of key-value blocks with computation.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors say the approach enables training and inference of sequences up to device count times longer than prior memory-efficient Transformers, without approximations or extra communication and computation overheads.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They point to code at the GitHub URL named in the paper comments, which later redirected to https://github.com/haoliuhl/ringattention.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2310.01889, https://arxiv.org/abs/2310.01889. Paper license: arXiv default (non-exclusive distribution license). v4 (27 Nov 2023). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_0ewpsgny,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Ring Attention with Blockwise Transformers for Near-Infinite Context}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2310-01889-ring-attention/},
  note = {Permanent ID iil:note:0ewpsgny. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2310-01889-ring-attention.md.