← All notes

Summary

RULER is a synthetic benchmark for the usable length of long-context models, beyond a simple needle-in-a-haystack test. The authors evaluate 17 models with 13 tasks and report that, although all claim context sizes of 32K tokens or greater, only half maintain satisfactory performance at 32K, and almost all fall below their threshold before the claimed length. [S1]

Claims

• id: C1 claim: 'They evaluate 17 long-context models with 13 representative tasks.' source: S1 evidence: "We evaluate 17 long-context LMs with 13 representative tasks in Ruler"
• id: C2 claim: 'They report that only half maintain satisfactory performance at 32K.' source: S1 evidence: "only half of them can maintain satisfactory performance at the length of 32K"
• id: C3 claim: 'Code is pointed to a public GitHub repository.' source: S1 evidence: "https://github.com/hsiehjackson/RULER"

Verification

• Code available: Paper: https://github.com/hsiehjackson/RULER, which returned HTTP 200 and redirected to https://github.com/NVIDIA/RULER on 2026-11-11. Not run.
• Reproduced: not reproduced (2026-11-11). Code not run.
• Related work: Live notes ao-context-long-context-limits and ao-context-longbench-v2. This is the paper-verification note for RULER itself.
• Limits: The authors note limited correlation with some realistic long-context tasks, and a lack of position control and short-context evaluation, in the limitations section.
• License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 9 Apr 2024 (v1); last revised 6 Aug 2024 (v3). Comments: COLM 2024; code is available at the paper URL. Sources: [S1] https://arxiv.org/abs/2404.06654, v3 (6 Aug 2024). checked_on: 2026-11-11.

When to apply

Use when asking how much of an advertised context window a model can actually use, not only whether it can retrieve one planted fact.

Facts

• id: F1 claim: 'The authors evaluate 17 long-context models on 13 RULER tasks.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F2 claim: 'They report that only half of the models that claim a 32K or larger window maintain satisfactory performance at 32K.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0
• id: F3 claim: 'They report that almost all evaluated models fall below their qualitative threshold before reaching the claimed context length.' confidence: medium date: 2026-11-11 sources: [S1] license: CC BY 4.0

Sources

• [S1] arXiv 2404.06654, https://arxiv.org/abs/2404.06654. License: CC BY 4.0. v3 (6 Aug 2024). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

@misc{iil_note_ac9vj2ec,
  author = {{'iinventedlight (AI-assisted)'}},
  title = {{RULER: What's the Real Context Size of Your Long-Context Language Models?}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-11},
  url = {https://iinventedlight.com/library/notes/2404-06654-ruler/},
  note = {Permanent ID iil:note:ac9vj2ec. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt | /agents.md | /.well-known/agent-library.json. This page in Markdown: /library/notes/2404-06654-ruler.md.