← All notes

Extending Context Window of Large Language Models via Positional Interpolation

Summary

Position Interpolation (PI) extends the context window of RoPE-based models such as LLaMA to up to 32,768 tokens with light fine-tuning, within 1,000 steps. The method linearly down-scales input position indices so they fit the original trained window, instead of extrapolating past that length, which can produce very large attention scores. The authors report strong results on passkey retrieval, language modeling, and long-document summarization from LLaMA 7B to 65B, and they say quality on the original window is preserved relatively well. The extended models keep the original architecture. [S1]

Claims

Verification

When to apply

Use when asking how RoPE context windows were first extended by interpolation, or how PI differs from plain extrapolation.

Facts

Sources

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

License: CC-BY-4.0 · Summary by iinventedlight (AI-assisted). Source: Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, Extending Context Window of Large Language Models via Positional Interpolation (CC BY 4.0).

Cite this

Permanent ID: iil:note:k1vwhch4

'iinventedlight (AI-assisted)'. 2026. "Extending Context Window of Large Language Models via Positional Interpolation." iinventedlight Library, iil:note:k1vwhch4, version of 2026-11-12. https://iinventedlight.com/library/notes/2306-15595-positional-interpolation/. License: CC-BY-4.0.

@misc{iil_note_k1vwhch4,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{Extending Context Window of Large Language Models via Positional Interpolation}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-12},
  url = {https://iinventedlight.com/library/notes/2306-15595-positional-interpolation/},
  note = {Permanent ID iil:note:k1vwhch4. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2306-15595-positional-interpolation.md.