← All notes

Answer: LongBench v2 assesses whether LLMs can handle long-context problems that require deep understanding and reasoning. It consists of 503 challenging multiple-choice questions, with contexts ranging from 8k to 2M words.

What LongBench v2 measures

The questions span six major task categories, among them multi-document QA, code repo understanding, and long-dialogue history. The majority of contexts are under 128k words.

Results reported in the README

• Human experts reached only 53.7% accuracy under a 15-minute time constraint.
• The best-performing model answering directly reached 50.1%.
• o1-preview, with longer reasoning, reached 57.7%, surpassing the human baseline by 4%.

The README highlights enhanced reasoning ability and scaling inference-time compute as the way to tackle these long-context challenges.

Same direction from RULER

RULER generates synthetic examples with configurable sequence length. Almost all models fall below its threshold before reaching their claimed context lengths.

Artifacts

• Topic packet: ao-context

See also

• ao-context-long-context-limits
• ao-context-million-token-windows

Claims

• id: C1 claim: LongBench v2 has 503 multiple-choice questions. source: S1 evidence: "LongBench v2 consists of 503 challenging multiple-choice questions"

• id: C2 claim: Experts scored 53.7% in 15 minutes. source: S1 evidence: "human experts achieving only 53.7% accuracy under a 15-minute time constraint."

• id: C3 claim: The best direct model scored 50.1%. source: S1 evidence: "achieves only 50.1% accuracy."

• id: C4 claim: Most models fail before their claimed length. source: S2 evidence: "Almost all models fall below the threshold before reaching the claimed context lengths."

Sources

• [S1] LongBench v2 README. THUDM LongBench (GitHub). https://github.com/THUDM/LongBench/blob/main/README.md . License: MIT (THUDM/LongBench LICENSE). Accessed 2026-11-09.

• [S2] RULER README. NVIDIA RULER (GitHub). https://github.com/NVIDIA/RULER/blob/main/README.md . License: Apache-2.0 (NVIDIA/RULER LICENSE). Accessed 2026-11-09.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Sources: GitHub repository docs under Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0) or MIT License (https://opensource.org/licenses/MIT), as listed per source; copyright notices remain with the original authors. Text was condensed and changed from the originals.

@misc{iil_note_8z8t6k79,
  author = {{iinventedlight}},
  title = {{LongBench v2: long-context reasoning is still hard}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-09},
  url = {https://iinventedlight.com/library/notes/ao-context-longbench-v2/},
  note = {Permanent ID iil:note:8z8t6k79. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/ao-context-longbench-v2.md.