← All notes

Summary

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021), by Ofir Press, Noah A. Smith, and Mike Lewis, is at https://arxiv.org/abs/2108.12409. The method, Attention with Linear Biases, or ALiBi, does not add positional embeddings to word embeddings. Instead, it biases query-key attention scores with a penalty that is proportional to their distance. The authors report that a 1.3 billion parameter model trained on length 1024 sequences can extrapolate to length 2048 and match the perplexity of a sinusoidal position embedding model trained on length 2048, while training 11% faster and using 11% less memory. They also report that this inductive bias towards recency leads it to outperform multiple strong position methods on the WikiText 103 benchmark. They first show that extrapolation can be enabled by simply changing the position representation method, though they find that current methods do not allow for efficient extrapolation. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v2 (22 Apr 2022), submitted 27 Aug 2021.

Claims

Pointer notes do not quote the paper. See Verification and Sources for the link and license.

Verification

• Format: pointer note only. The paper is not reproduced here.
• Why it matters: It matters because it showed that a simple linear bias on attention can let a model handle inputs longer than those it saw in training, without heavy position-embedding tricks.
• Reproduced: not reproduced (2026-11-11).
• Related work: Related notes already in the library include StreamingLLM and RULER.
• Limits: This is a pointer note only. The 1024-to-2048 extrapolation result and the 11% training-time and memory figures are from the authors' v2 paper (22 Apr 2022).
• License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 27 Aug 2021 (v1); last revised 22 Apr 2022 (v2). Sources: [S1] https://arxiv.org/abs/2108.12409, v2 (22 Apr 2022). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

When to apply

Use when you need the original ALiBi paper on extrapolating attention beyond the training context length with a simple distance penalty.

Facts

• id: F1 claim: 'ALiBi does not add positional embeddings to word embeddings; instead it biases query-key attention scores with a penalty proportional to distance.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F2 claim: 'The authors report training a 1.3 billion parameter model on length 1024 sequences that extrapolates to length 2048, matching the perplexity of a sinusoidal model trained on length 2048 while training 11% faster and using 11% less memory.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0
• id: F3 claim: 'They also report that ALiBi''s bias toward recency helps it outperform multiple strong position methods on WikiText-103.' confidence: medium date: 2026-11-11 sources: [S1] license: CC-BY-4.0

Sources

• [S1] arXiv 2108.12409, https://arxiv.org/abs/2108.12409. Paper license: arXiv default (non-exclusive distribution license). v2 (22 Apr 2022). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

@misc{iil_note_19gq94fz,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2108-12409-alibi/},
  note = {Permanent ID iil:note:19gq94fz. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2108-12409-alibi.md.