---
source: https://iinventedlight.com/library/notes/ao-routing-cost-aware.md
title: "LLM routing: does it pay for itself?"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
source_license: "CC BY 4.0; restated in our own words"
attribution: "Summary by iinventedlight (AI-assisted). Sources: arXiv, [2609.37402] Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing, CC BY 4.0; arXiv, [2601.07206] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing, CC BY 4.0."
author: "iinventedlight (synthesis of cited sources); text drafted with AI assistance"
date: 2026-11-04
status: published
checked_on: 2026-11-04
as_of: 2026-11-04
confidence: medium
untested: "No experiment was run; only the fetched arXiv abstract pages were read, so results are as reported by the authors."
category: agent-operations
topic_area: "agent-ops: model routing"
tags: agents, model-routing, cost
sources:
  - id: S1
    title: "[2609.37402] Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing"
    publisher: arXiv
    url: https://arxiv.org/abs/2609.37402
    accessed: 2026-11-04
    license: "CC BY 4.0"
  - id: S2
    title: "[2601.07206] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing"
    publisher: arXiv
    url: https://arxiv.org/abs/2601.07206
    accessed: 2026-11-04
    license: "CC BY 4.0"
---
# LLM routing: does it pay for itself?

## Short answer
Routing each query to a cheaper or stronger model can cut serving cost, but training the router has its own cost, and many routers do no better than a simple baseline. Count supervision cost as well as serving savings. Sparse feedback (about a third of the labels) was enough to train a competitive router. [S1][S2]

## Practice
- Benchmark against a simple baseline first. [S2]
- Choosing models carefully beats adding more to the ensemble. [S2]

## Claims
- id: C1
  claim: Routing reduces serving cost while preserving quality.
  source: S1
  evidence: "Large language model (LLM) routing reduces serving cost by assigning each query to an appropriate"
- id: C2
  claim: Routing quality often saturates before all feedback is collected.
  source: S1
  evidence: "routing quality often saturates well before all query--model feedback is collected"
- id: C3
  claim: Several routers, including commercial ones, fail to beat a simple baseline.
  source: S2
  evidence: "fail to reliably outperform a simple baseline."
- id: C4
  claim: Larger ensembles show diminishing returns versus careful curation.
  source: S2
  evidence: "larger ensembles exhibit diminishing returns compared to careful model curation"

## Sources
- [S1] [2609.37402] Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing. arXiv. https://arxiv.org/abs/2609.37402 . License: CC BY 4.0. Accessed 2026-11-04.
- [S2] [2601.07206] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. arXiv. https://arxiv.org/abs/2601.07206 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).