---
title: 'Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'This note is CC BY 4.0. It links to the paper and does not reproduce it. Paper license: arXiv non-exclusive distribution (verified on arXiv abs page 2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/).'
source: https://iinventedlight.com/library/notes/2108-12409-alibi.md
source_license: 'arXiv default (non-exclusive distribution license)'
attribution: 'Pointer note by iinventedlight (AI-assisted). Links to: Ofir Press, Noah A. Smith, Mike Lewis, Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (arXiv). This note does not reproduce the paper.'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:58 PT, Grok Bot chat widget t1717s7407: \"Publish all 100 including the LLMs Encode Clinical Knowledge note\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Pointer note only. Abs page read 2026-11-11. The paper is not reproduced here.'
category: arxiv
topic_area: 'ai-research: memory-and-context'
tags: arxiv, pointer, memory-and-context
arxiv_id: 2108.12409
alias: 2108.12409-alibi
format: pointer
type: pointer
sources:
  - id: S1
    title: 'Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation'
    publisher: arXiv
    authors: 'Ofir Press, Noah A. Smith, Mike Lewis'
    url: https://arxiv.org/abs/2108.12409
    version_read: 'v2 (22 Apr 2022)'
    accessed: 2026-11-11
    license: 'arXiv default (non-exclusive distribution license)'
---
# Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

## Summary
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021), by Ofir Press, Noah A. Smith, and Mike Lewis, is at https://arxiv.org/abs/2108.12409. The method, Attention with Linear Biases, or ALiBi, does not add positional embeddings to word embeddings. Instead, it biases query-key attention scores with a penalty that is proportional to their distance. The authors report that a 1.3 billion parameter model trained on length 1024 sequences can extrapolate to length 2048 and match the perplexity of a sinusoidal position embedding model trained on length 2048, while training 11% faster and using 11% less memory. They also report that this inductive bias towards recency leads it to outperform multiple strong position methods on the WikiText 103 benchmark. They first show that extrapolation can be enabled by simply changing the position representation method, though they find that current methods do not allow for efficient extrapolation. Title, authors, abstract, and comments were taken from the arXiv abs page. The version is v2 (22 Apr 2022), submitted 27 Aug 2021.

## Claims
Pointer notes do not quote the paper. See Verification and Sources for the link and license.

## Verification
- Format: pointer note only. The paper is not reproduced here.
- Why it matters: It matters because it showed that a simple linear bias on attention can let a model handle inputs longer than those it saw in training, without heavy position-embedding tricks.
- Reproduced: not reproduced (2026-11-11).
- Related work: Related notes already in the library include [StreamingLLM](https://iinventedlight.com/library/notes/2309-17453-streamingllm-attention-sinks/) and [RULER](https://iinventedlight.com/library/notes/2404-06654-ruler/).
- Limits: This is a pointer note only. The 1024-to-2048 extrapolation result and the 11% training-time and memory figures are from the authors' v2 paper (22 Apr 2022).
- License: arXiv default (non-exclusive distribution license) on the abs page (2026-11-11; http://arxiv.org/licenses/nonexclusive-distrib/1.0/). Submitted 27 Aug 2021 (v1); last revised 22 Apr 2022 (v2). Sources: [S1] https://arxiv.org/abs/2108.12409, v2 (22 Apr 2022). checked_on: 2026-11-11. Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it. The note itself is CC BY 4.0.

## When to apply
Use when you need the original ALiBi paper on extrapolating attention beyond the training context length with a simple distance penalty.

## Facts
- id: F1
  claim: 'ALiBi does not add positional embeddings to word embeddings; instead it biases query-key attention scores with a penalty proportional to distance.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F2
  claim: 'The authors report training a 1.3 billion parameter model on length 1024 sequences that extrapolates to length 2048, matching the perplexity of a sinusoidal model trained on length 2048 while training 11% faster and using 11% less memory.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0
- id: F3
  claim: 'They also report that ALiBi''s bias toward recency helps it outperform multiple strong position methods on WikiText-103.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC-BY-4.0

## Sources
- [S1] arXiv 2108.12409, https://arxiv.org/abs/2108.12409. Paper license: arXiv default (non-exclusive distribution license). v2 (22 Apr 2022). Accessed 2026-11-11. This note links to the paper and does not reproduce it.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Pointer by iinventedlight (AI-assisted). Paper license: arXiv non-exclusive distribution; this note links to it and does not reproduce it.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).