---
source: https://iinventedlight.com/library/notes/ao-context-long-context-limits.md
title: "Long context is shorter than advertised"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?, CC BY 4.0; arXiv, [2505.06120] LLMs Get Lost In Multi-Turn Conversation, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was rerun; only the fetched arXiv abstract pages were read. Results are as reported by the authors."
category: agent-operations
topic_area: "agent operations: context"
tags: agents, context-window, long-context, multi-turn
sources:
  - id: S1
    title: "[2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?"
    publisher: arXiv
    url: https://arxiv.org/abs/2404.06654
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2505.06120] LLMs Get Lost In Multi-Turn Conversation"
    publisher: arXiv
    url: https://arxiv.org/abs/2505.06120
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# Long context is shorter than advertised

## Short answer
Treat claimed context size as an upper bound. In RULER, only half of models claiming 32K or more held up at 32K. Separately, models lost an average 39% in multi-turn chats versus one complete prompt. [S1][S2]

## Practice
- Keep active context lean; put key facts near the request.
- If a chat goes wrong early, restart with one clear, complete prompt. [S2]

## Claims
- id: C1
  claim: Only half of models perform well at 32K.
  source: S1
  evidence: "only half of them can maintain satisfactory performance at the length of 32K"
- id: C2
  claim: Performance drops as context length grows.
  source: S1
  evidence: "almost all models exhibit large performance drops as the context length increases"
- id: C3
  claim: Multi-turn performance drops 39% on average.
  source: S2
  evidence: "with an average drop of 39% across six generation tasks"
- id: C4
  claim: LLMs get lost after a wrong turn.
  source: S2
  evidence: "when LLMs take a wrong turn in a conversation, they get lost"

## Sources
- [S1] [2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?. arXiv. https://arxiv.org/abs/2404.06654 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2505.06120] LLMs Get Lost In Multi-Turn Conversation. arXiv. https://arxiv.org/abs/2505.06120 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).