---
title: 'RULER: What''s the Real Context Size of Your Long-Context Language Models?'
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: 'paper is CC BY 4.0 (verified on arXiv abs page 2026-11-11; icon by-4.0.png, http://creativecommons.org/licenses/by/4.0/); note is our own summary'
source: https://iinventedlight.com/library/notes/2404-06654-ruler.md
source_license: 'CC BY 4.0'
attribution: 'Summary by iinventedlight (AI-assisted). Source: Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg, RULER: What''s the Real Context Size of Your Long-Context Language Models? (CC BY 4.0).'
author: 'iinventedlight (AI-assisted)'
date: 2026-11-11
status: published
approved_by: "Cole Withington, 2026-11-11 20:07 PT, Librarian chat widget t1715u: \"okay you can publish the ones you're comfortable with\""
checked_on: 2026-11-11
as_of: 2026-11-11
confidence: medium
untested: 'Not reproduced. Abs page and HTML key sections read 2026-11-11. Named code URLs checked for HTTP 200; code not run.'
category: arxiv
topic_area: 'ai-research: agent memory and context compaction'
tags: agents, memory, compaction, arxiv
arxiv_id: 2404.06654
alias: 2404.06654-ruler
sources:
  - id: S1
    title: 'RULER: What''s the Real Context Size of Your Long-Context Language Models?'
    publisher: arXiv
    authors: 'Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg'
    url: https://arxiv.org/abs/2404.06654
    version_read: 'v3 (6 Aug 2024)'
    accessed: 2026-11-11
    license: 'CC BY 4.0'
---
# RULER: What's the Real Context Size of Your Long-Context Language Models?

## Summary
RULER is a synthetic benchmark for the usable length of long-context models, beyond a simple needle-in-a-haystack test. The authors evaluate 17 models with 13 tasks and report that, although all claim context sizes of 32K tokens or greater, only half maintain satisfactory performance at 32K, and almost all fall below their threshold before the claimed length. [S1]

## Claims
- id: C1
  claim: 'They evaluate 17 long-context models with 13 representative tasks.'
  source: S1
  evidence: "We evaluate 17 long-context LMs with 13 representative tasks in Ruler"
- id: C2
  claim: 'They report that only half maintain satisfactory performance at 32K.'
  source: S1
  evidence: "only half of them can maintain satisfactory performance at the length of 32K"
- id: C3
  claim: 'Code is pointed to a public GitHub repository.'
  source: S1
  evidence: "https://github.com/hsiehjackson/RULER"

## Verification
- Code available: Paper: https://github.com/hsiehjackson/RULER, which returned HTTP 200 and redirected to https://github.com/NVIDIA/RULER on 2026-11-11. Not run.
- Reproduced: not reproduced (2026-11-11). Code not run.
- Related work: Live notes ao-context-long-context-limits and ao-context-longbench-v2. This is the paper-verification note for RULER itself.
- Limits: The authors note limited correlation with some realistic long-context tasks, and a lack of position control and short-context evaluation, in the limitations section.
- License: CC BY 4.0 on the abs page (2026-11-11; icon by-4.0.png; http://creativecommons.org/licenses/by/4.0/). Submitted 9 Apr 2024 (v1); last revised 6 Aug 2024 (v3). Comments: COLM 2024; code is available at the paper URL. Sources: [S1] https://arxiv.org/abs/2404.06654, v3 (6 Aug 2024). checked_on: 2026-11-11.

## When to apply
Use when asking how much of an advertised context window a model can actually use, not only whether it can retrieve one planted fact.

## Facts
- id: F1
  claim: 'The authors evaluate 17 long-context models on 13 RULER tasks.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F2
  claim: 'They report that only half of the models that claim a 32K or larger window maintain satisfactory performance at 32K.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0
- id: F3
  claim: 'They report that almost all evaluated models fall below their qualitative threshold before reaching the claimed context length.'
  confidence: medium
  date: 2026-11-11
  sources: [S1]
  license: CC BY 4.0

## Sources
- [S1] arXiv 2404.06654, https://arxiv.org/abs/2404.06654. License: CC BY 4.0. v3 (6 Aug 2024). Accessed 2026-11-11.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted). Paper is CC BY 4.0.

---

Agents: [/agents.md](/agents.md) | [/.well-known/agent-library.json](/.well-known/agent-library.json).