---
title: "Extending Context Window of Large Language Models via Positional Interpolation"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-12; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2306-15595-positional-interpolation.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian, Extending Context Window of Large Language Models via Positional Interpolation (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-12
status: published
approved_by: "Cole Withington, 2026-11-12 14:44 PT, Librarian chat widget t51s316: \"Yes, publish all 100 second-list arXiv notes\""
checked_on: 2026-11-12
as_of: 2026-11-12
confidence: medium
untested: "Not reproduced. Abs page and HTML key sections read 2026-11-12. Code not run."
category: arxiv
topic_area: 'ai-research: memory-and-context'
tags: arxiv, long-context, rope, position-interpolation, llama
arxiv_id: 2306.15595
alias: 2306-15595-positional-interpolation
sources:
  - id: S1
    title: "Extending Context Window of Large Language Models via Positional Interpolation"
    publisher: arXiv
    authors: "Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian"
    url: https://arxiv.org/abs/2306.15595
    version_read: "v2 (28 Jun 2023)"
    accessed: 2026-11-12
    license: 'CC BY 4.0'
---
# Extending Context Window of Large Language Models via Positional Interpolation

## Summary
Position Interpolation (PI) extends the context window of RoPE-based models such as LLaMA to up to 32,768 tokens with light fine-tuning, within 1,000 steps. The method linearly down-scales input position indices so they fit the original trained window, instead of extrapolating past that length, which can produce very large attention scores. The authors report strong results on passkey retrieval, language modeling, and long-document summarization from LLaMA 7B to 65B, and they say quality on the original window is preserved relatively well. The extended models keep the original architecture. [S1]

## Claims
- id: C1
  claim: "Position Interpolation extends RoPE-based models such as LLaMA to up to 32,768 tokens with fine-tuning within 1,000 steps."
  source: S1
  evidence: "up to 32768 with minimal fine-tuning (within 1000 steps)"
- id: C2
  claim: "It linearly down-scales position indices to the original context window instead of extrapolating beyond it."
  source: S1
  evidence: "linearly down-scales the input position indices to match the original context window size"
- id: C3
  claim: "The authors argue this is more stable than extrapolation and keeps the original architecture."
  source: S1
  evidence: "Models extended via Position Interpolation retain its original architecture"

## Verification
- Code available: No official code URL is named on the abs page. Not run.
- Reproduced: not reproduced (2026-11-12). Code not run.
- Related work: See [2309-00071-yarn](https://iinventedlight.com/library/notes/2309-00071-yarn/) and the effective long-context scaling note in this batch.
- Limits: The 32,768 figure and the 1,000-step fine-tune are as reported for the LLaMA models in v2. Later work (for example YaRN) revisits how positions should be scaled.
- License: CC BY 4.0 on the abs page (2026-11-12; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 27 Jun 2023 (v1); last revised v2 (28 Jun 2023). Sources: [S1] https://arxiv.org/abs/2306.15595, v2 (28 Jun 2023). checked_on: 2026-11-12.

## When to apply
Use when asking how RoPE context windows were first extended by interpolation, or how PI differs from plain extrapolation.

## Facts
- id: F1
  claim: "Position Interpolation extends RoPE-based LLaMA models to up to 32,768 tokens with fine-tuning within 1,000 steps."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: "The method scales position indices down into the original trained window rather than extrapolating past it."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: "The extended models keep the original architecture and can reuse existing infrastructure."
  confidence: medium
  date: 2026-11-12
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2306.15595, https://arxiv.org/abs/2306.15595. License: CC BY 4.0. v2 (28 Jun 2023). Accessed 2026-11-12.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).
