← All notes

Summary

The top open- and closed-weight LLMs tested exhibit significantly lower performance in multi-turn conversations than single-turn, with an average drop of 39% across six generation tasks. The 15 LLMs tested perform much worse in multi-turn settings (-35%), explained by some loss in aptitude, and large losses in reliability. When LLMs take a wrong turn in a conversation, they get lost and do not recover. [S1]

Claims

• id: C1 claim: 'They report an average 39% drop across six generation tasks.' source: S1 evidence: "an average drop of 39% across six generation tasks"
• id: C2 claim: 'They report that 15 models perform much worse in multi-turn settings.' source: S1 evidence: "The 15 LLMs we test perform much worse in multi-turn settings (-35%)"
• id: C3 claim: 'A public Microsoft repository name appears in the HTML.' source: S1 evidence: "Microsoft/lost_in_conversation"

Verification

• Code available: HTML points to https://github.com/Microsoft/lost_in_conversation (HTTP 200 on 2026-11-11) and https://huggingface.co/datasets/Microsoft/lost_in_conversation (HTTP 200). Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live note ao-context-long-context-limits. This paper is about multi-turn conversation, not compaction, but it is a core context-use result for agents that talk over many turns.
• Limits: Results are from simulated sharded conversations, not live users. The paper also reports a -35% multi-turn figure for the 15 models tested; both figures are author-reported and refer to slightly different summaries in the text.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 9 May 2025 (v1). Sources: [S1] https://arxiv.org/abs/2505.06120, v1 (9 May 2025). checked_on: 2026-11-11.

When to apply

Use when asking why an assistant degrades over a back-and-forth chat, even if it can solve the same task when the request is fully specified up front.

Facts

• id: F1 claim: 'The authors report an average 39% drop from single-turn to multi-turn performance across six generation tasks.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'They report that 15 tested models perform much worse in multi-turn settings, described as a 35% drop, with larger losses in reliability than in aptitude.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'In one summary, multi-turn underspecified conversations reached 65% against single-turn 90%.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2505.06120, https://arxiv.org/abs/2505.06120. License: CC BY 4.0. v1 (9 May 2025). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_0m371bz0,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{LLMs Get Lost In Multi-Turn Conversation}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2505-06120-llms-lost-in-multiturn/},
  note = {Permanent ID iil:note:0m371bz0. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2505-06120-llms-lost-in-multiturn.md.