Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
Summary
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (2019) was an early attempt to let a Transformer model more than one fixed window of tokens. It introduced segment-level recurrence and a relative positional encoding so that information could carry across segment boundaries.
What it introduced: a recurrence mechanism over cached hidden states from previous segments, plus relative encodings that made that recurrence stable, so language models could use a longer effective context than a single training window.
What it led to: later long-context work, from relative and rotary position methods to today's long-context evals. Live notes: ao-context-long-context-limits. See also the RoFormer / RoPE pointer in this batch and the Lost in the Middle pointer in batch B.
Where to read it: https://arxiv.org/abs/1901.02860 (v3, last revised 2 Jun 2019). ACL 2019 long paper.
The paper is posted under CC BY-NC-SA 4.0. This note links to it rather than reproducing it. The note itself is CC BY 4.0.
License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: CC BY-NC-SA 4.0; this note links to it and does not reproduce it.
License: CC-BY-4.0 · Pointer by iinventedlight (AI-assisted). Source: Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov, Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (CC BY-NC-SA 4.0). This note links to the paper rather than reproducing it.
Cite this
'iinventedlight (AI-assisted)'. 2026. "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context." iinventedlight Library, iil:note:eb1nxw61, version of 2026-11-11. https://iinventedlight.com/library/notes/1901-02860-transformer-xl/. License: CC-BY-4.0.
@misc{iil_note_eb1nxw61,
author = {{'iinventedlight (AI-assisted)'}},
title = {{Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context}},
howpublished = {iinventedlight Library},
year = {2026},
date = {2026-11-11},
url = {https://iinventedlight.com/library/notes/1901-02860-transformer-xl/},
note = {Permanent ID iil:note:eb1nxw61. License: CC-BY-4.0.},
}Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.
For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/1901-02860-transformer-xl.md.