← All notes

Short answer

Routing each query to a cheaper or stronger model can cut serving cost, but training the router has its own cost, and many routers do no better than a simple baseline. Count supervision cost as well as serving savings. Sparse feedback (about a third of the labels) was enough to train a competitive router. [S1][S2]

Practice

• Benchmark against a simple baseline first. [S2]
• Choosing models carefully beats adding more to the ensemble. [S2]

Claims

• id: C1 claim: Routing reduces serving cost while preserving quality. source: S1 evidence: "Large language model (LLM) routing reduces serving cost by assigning each query to an appropriate"
• id: C2 claim: Routing quality often saturates before all feedback is collected. source: S1 evidence: "routing quality often saturates well before all query--model feedback is collected"
• id: C3 claim: Several routers, including commercial ones, fail to beat a simple baseline. source: S2 evidence: "fail to reliably outperform a simple baseline."
• id: C4 claim: Larger ensembles show diminishing returns versus careful curation. source: S2 evidence: "larger ensembles exhibit diminishing returns compared to careful model curation"

Sources

• [S1] [2609.37402] Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routing. arXiv. https://arxiv.org/abs/2609.37402 . License: CC BY 4.0. Accessed 2026-11-04.
• [S2] [2601.07206] LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. arXiv. https://arxiv.org/abs/2601.07206 . License: CC BY 4.0. Accessed 2026-11-04.

License: this note is CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Summary by iinventedlight (AI-assisted).

@misc{iil_note_z1gepq4g,
  author = {{iinventedlight}},
  title = {{LLM routing: does it pay for itself?}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-04},
  url = {https://iinventedlight.com/library/notes/ao-routing-cost-aware/},
  note = {Permanent ID iil:note:z1gepq4g. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-routing-cost-aware.md.