← All notes

Short answer

Treat claimed context size as an upper bound. In RULER, only half of models claiming 32K or more held up at 32K. Separately, models lost an average 39% in multi-turn chats versus one complete prompt. [S1][S2]

Practice

• Keep active context lean; put key facts near the request.
• If a chat goes wrong early, restart with one clear, complete prompt. [S2]

Claims

• id: C1 claim: Only half of models perform well at 32K. source: S1 evidence: "only half of them can maintain satisfactory performance at the length of 32K"
• id: C2 claim: Performance drops as context length grows. source: S1 evidence: "almost all models exhibit large performance drops as the context length increases"
• id: C3 claim: Multi-turn performance drops 39% on average. source: S2 evidence: "with an average drop of 39% across six generation tasks"
• id: C4 claim: LLMs get lost after a wrong turn. source: S2 evidence: "when LLMs take a wrong turn in a conversation, they get lost"

Sources

• [S1] [2404.06654] RULER: What's the Real Context Size of Your Long-Context Language Models?. arXiv. https://arxiv.org/abs/2404.06654 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2505.06120] LLMs Get Lost In Multi-Turn Conversation. arXiv. https://arxiv.org/abs/2505.06120 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_2yqcdrb4,
  author = {{iinventedlight}},
  title = {{Long context is shorter than advertised}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-context-long-context-limits/},
  note = {Permanent ID iil:note:2yqcdrb4. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-context-long-context-limits.md.