---
source: https://iinventedlight.com/library/notes/ao-context-longbench-v2.md
title: "LongBench v2: long-context reasoning is still hard"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "Apache-2.0 or MIT (repo docs; LICENSE file verified in each GitHub repo); prose restates cited source lines in own words"
license_family: agentops-green
attribution: "Summary by iinventedlight (AI-assisted). Sources: THUDM LongBench (GitHub), LongBench v2 README, MIT (THUDM/LongBench LICENSE); NVIDIA RULER (GitHub), RULER README, Apache-2.0 (NVIDIA/RULER LICENSE)."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-09
status: published
checked_on: 2026-11-09
as_of: 2026-11-09
confidence: medium
untested: "No tool or code was run for this note; only the fetched repository documentation was read."
category: agent-operations
topic_area: "agent operations: context-window management"
tags: [long-context, benchmarks, context-window, evaluation]
client: Librarian
sources:
  - id: S1
    title: "LongBench v2 README"
    publisher: THUDM LongBench (GitHub)
    url: https://github.com/THUDM/LongBench/blob/main/README.md
    accessed: 2026-11-09
    license: "MIT (THUDM/LongBench LICENSE)"
  - id: S2
    title: "RULER README"
    publisher: NVIDIA RULER (GitHub)
    url: https://github.com/NVIDIA/RULER/blob/main/README.md
    accessed: 2026-11-09
    license: "Apache-2.0 (NVIDIA/RULER LICENSE)"
---
# LongBench v2: long-context reasoning is still hard

**Answer:** LongBench v2 assesses whether LLMs can handle long-context problems that require deep understanding and reasoning. It consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words.

## What LongBench v2 measures

The questions span six major task categories, among them multi-document QA, code repo understanding, and long-dialogue history. The majority of contexts are under 128k words.

## Results reported in the README

- Human experts reached only 53.7% accuracy under a 15-minute time constraint.
- The best-performing model answering directly reached 50.1%.
- o1-preview, with longer reasoning, reached 57.7%, surpassing the human baseline by 4%.

The README highlights enhanced reasoning ability and scaling inference-time compute as the way to tackle these long-context challenges.

## Same direction from RULER

RULER generates synthetic examples with configurable sequence length. Almost all models fall below its threshold before reaching their claimed context lengths.

## Artifacts

- [Topic packet: ao-context](https://iinventedlight.com/library/packets/ao-context.json)

## See also

- [ao-context-long-context-limits](https://iinventedlight.com/library/notes/ao-context-long-context-limits/)
- [ao-context-million-token-windows](https://iinventedlight.com/library/notes/ao-context-million-token-windows/)

## Claims

- id: C1
  claim: LongBench v2 has 503 multiple-choice questions.
  source: S1
  evidence: "LongBench v2 consists of 503 challenging multiple-choice questions"

- id: C2
  claim: Experts scored 53.7% in 15 minutes.
  source: S1
  evidence: "human experts achieving only 53.7% accuracy under a 15-minute time constraint."

- id: C3
  claim: The best direct model scored 50.1%.
  source: S1
  evidence: "achieves only 50.1% accuracy."

- id: C4
  claim: Most models fail before their claimed length.
  source: S2
  evidence: "Almost all models fall below the threshold before reaching the claimed context lengths."

## Sources

- [S1] LongBench v2 README. THUDM LongBench (GitHub). https://github.com/THUDM/LongBench/blob/main/README.md . License: MIT (THUDM/LongBench LICENSE). Accessed 2026-11-09.

- [S2] RULER README. NVIDIA RULER (GitHub). https://github.com/NVIDIA/RULER/blob/main/README.md . License: Apache-2.0 (NVIDIA/RULER LICENSE). Accessed 2026-11-09.


License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: GitHub repository docs under Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0) or MIT License (https://opensource.org/licenses/MIT), as listed per source; copyright notices remain with the original authors. Text was condensed and changed from the originals.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).