Landmark Attention: Random-Access Infinite Context Length for Transformers
Summary
Landmark Attention: Random-Access Infinite Context Length for Transformers (2023) is a method for very long context that keeps random-access attention, instead of giving up the ability to jump to any earlier token.
What it introduced: a landmark token for each block of the input. Attention is trained to use those landmarks to select relevant blocks, so retrieval happens inside the attention mechanism rather than through a separate retriever. The authors say this can work with specialized data structures and the memory hierarchy, and they report that a fine-tuned LLaMA 7B can handle context lengths over 32k tokens. They also compare with Transformer-XL while retrieving fewer tokens at each step. Code is public.
What it led to: later block- or landmark-based attention schemes for long context, alongside interpolation methods and sparse patterns. Live notes: 2307-03172-lost-in-the-middle, 2308-14508-longbench, and 2310-01889-ring-attention. Position Interpolation is a full note in this week's batch F.
Where to read it: https://arxiv.org/abs/2305.16300 (v2, last revised 20 Nov 2023). NeurIPS 2023. Code: https://github.com/epfml/landmark-attention/.
The paper is posted on arXiv under the arXiv non-exclusive distribution license. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Amirkeivan Mohtashami, Martin Jaggi, Landmark Attention: Random-Access Infinite Context Length for Transformers (arXiv non-exclusive distribution license). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "Landmark Attention: Random-Access Infinite Context Length for Transformers." iinventedlight Library, iil:note:kr2ete3f, version of 2026-11-12. https://iinventedlight.com/library/notes/2305-16300-landmark-attention/. License: CC-BY-4.0.
@misc{iil_note_kr2ete3f,
author = {{'iinventedlight (AI-assisted)'}},
title = {{Landmark Attention: Random-Access Infinite Context Length for Transformers}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-12},
url = {https://iinventedlight.com/library/notes/2305-16300-landmark-attention/},
note = {Permanent ID iil:note:kr2ete3f. License: CC-BY-4.0.},
}
Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2305-16300-landmark-attention.md.