← All notes

Landmark Attention: Random-Access Infinite Context Length for Transformers

Summary

Landmark Attention: Random-Access Infinite Context Length for Transformers (2023) is a method for very long context that keeps random-access attention, instead of giving up the ability to jump to any earlier token.

What it introduced: a landmark token for each block of the input. Attention is trained to use those landmarks to select relevant blocks, so retrieval happens inside the attention mechanism rather than through a separate retriever. The authors say this can work with specialized data structures and the memory hierarchy, and they report that a fine-tuned LLaMA 7B can handle context lengths over 32k tokens. They also compare with Transformer-XL while retrieving fewer tokens at each step. Code is public.

What it led to: later block- or landmark-based attention schemes for long context, alongside interpolation methods and sparse patterns. Live notes: 2307-03172-lost-in-the-middle, 2308-14508-longbench, and 2310-01889-ring-attention. Position Interpolation is a full note in this week's batch F.

Where to read it: https://arxiv.org/abs/2305.16300 (v2, last revised 20 Nov 2023). NeurIPS 2023. Code: https://github.com/epfml/landmark-attention/.

The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Amirkeivan Mohtashami, Martin Jaggi, Landmark Attention: Random-Access Infinite Context Length for Transformers (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.

Cite this

Permanent ID: iil:note:kr2ete3f

'iinventedlight (AI-assisted)'. 2026. "Landmark Attention: Random-Access Infinite Context Length for Transformers." iinventedlight Library, iil:note:kr2ete3f, version of 2026-11-12. https://iinventedlight.com/library/notes/2305-16300-landmark-attention/. License: CC-BY-4.0.

@misc{iil_note_kr2ete3f,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Landmark Attention: Random-Access Infinite Context Length for Transformers}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2305-16300-landmark-attention/},
  note = {Permanent ID iil:note:kr2ete3f. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2305-16300-landmark-attention.md.