← All notes

Summary

A router sends each request to a model that fits it, instead of always calling the most expensive model. The abstracts below say two practical things: collecting the labels to train that router can cost more than people count, and under one large shared test many routers look alike and still miss the best model.

Count the training bill, not only the serving bill

Routing is supposed to cut serving cost while keeping answer quality. Learning the router usually means running several candidate models on old queries so you know which model was good for which query. That work happens before the router is deployed, and it is easy to ignore if you only look at cost per request after launch.

The first abstract says routing quality often levels off before every query-model score has been collected, so buying all of that feedback can be more supervision than the economics need. SaveRouter tries to buy the informative feedback, share what it learns about capability across related queries, and still refine the choice at the level of a single query.

They score a router by adding the money spent on supervision to the money saved later at serving time. In the main setting, across four routing benchmarks, they used only a portion of the available training feedback and still report competitive or better routing quality. Compared with the fastest conventional router they cite, the volume of traffic needed before the router pays for its training dropped by a large factor.

More labels are not automatically the better buy. The amount of supervision that makes serving cheapest can be a different amount from the one that pays the training cost back soonest. If you are choosing how much to label, decide which of those two goals you care about. Do not assume the quality-maximizing label budget is the one that recovers its cost first.

What a shared benchmark shows

The second abstract introduces LLMRouterBench as a large benchmark and one framework for routing. It draws on many datasets and many models, scores both quality-first routing and quality-versus-cost routing, and includes a set of representative baseline routers. The point of the re-evaluation is to compare methods under the same setup.

The central reason to route still holds: models complement each other, so the best model is not the same for every query. Under that unified test, though, many routing methods land near each other. Several newer methods, including commercial routers, do not reliably beat a simple baseline.

A large gap remains to an oracle that already knows the right model. The abstract says that gap is mostly persistent failure to recall the right model, not a small tuning detail. The embedding model used as the backbone has limited effect. Making the pool of models larger brings diminishing returns next to choosing the pool carefully. The same benchmark can also be used to look at latency, not only quality and dollar cost.

What to do with this

Treat router training runs as a cost line, and stop collecting model feedback once quality has flattened if payback is the goal. When you compare routers, use one evaluation rather than each paper's own scoreboard. Prefer a small, deliberately chosen model pool over a long list of similar models, and do not treat a commercial router as better until it beats a simple baseline on that shared test. Expect misses where the router fails to call the model that would have been right.

Claims

• id: C1 claim: Routing lowers serving cost by sending each query to a fitting model while trying to keep response quality. source: S1 evidence: "routing reduces serving cost by assigning each query to an appropriate model while preserving response quality."
• id: C2 claim: Training usually means running several candidate models on past queries, which is a real cost before deployment. source: S1 evidence: "often requires executing multiple candidate models on historical queries to collect query--model quality feedback"
• id: C3 claim: Routing quality often stops improving before every query-model label has been gathered. source: S1 evidence: "routing quality often saturates well before all query--model feedback is collected"
• id: C4 claim: In their main setting a minority of the available training feedback still kept routing quality competitive or better. source: S1 evidence: "the main setting uses only about 33--41% of available training feedback while maintaining competitive or better routing quality"
• id: C5 claim: The supervision budget that makes serving cheapest need not be the one that pays back soonest. source: S1 evidence: "the supervision level that minimizes serving cost can differ from the one that achieves the earliest payback."
• id: C6 claim: Under one shared evaluation many routers score similarly, and some recent ones including commercial routers do not reliably beat a simple baseline. source: S2 evidence: "many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline."
• id: C7 claim: A large gap to the oracle remains, mainly because routers keep failing to recall the right model. source: S2 evidence: "a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures."
• id: C8 claim: Growing the model pool has diminishing returns compared with choosing the models carefully. source: S2 evidence: "larger ensembles exhibit diminishing returns compared to careful model curation"

@misc{iil_note_6y1yy50e,
  author = {{iinventedlight}},
  title = {{Model routing for agents}},
  howpublished = {iinventedlight Library},
  year = {2026},
  date = {2026-11-03},
  url = {https://iinventedlight.com/library/notes/ao-routing/},
  note = {Permanent ID iil:note:6y1yy50e. License: CC-BY-4.0.},
}

Markdown

Spotted a mistake? Use the MCP feedback tool or email hello@iinventedlight.com.

For language models and agents: start at /llms.txt. This page in Markdown: /library/notes/ao-routing.md.