Running frontier models for coding and other purposes can be very expensive, but also, sometimes, very effective. It’s tempting to use smaller or older models instead of frontier models because they’re cheaper, although they may not work as well. The ideal model for any given task tends to depend on the complexity of the task. Picking the most appropriate model automatically based on the request and the context is called model routing.
How model routing works
There are several ways of implementing model routing. In general, model routing depends on the pool of available models, the criteria used for selecting a model, and the objective that the router is trying to optimize. Different vendors offer different implementations of this scheme.
Databricks’ Unity Gateway has (in beta) Smart Routing for Codex or Claude Code. Smart Routing automatically selects the lowest-cost model capable of handling each task, so developers do not need to choose a model for every request.
Microsoft’s model router, a trained language model, analyzes your prompts in real time based on complexity, reasoning, task type, and other attributes. It does not store your prompts. It routes only to eligible models based on your access and deployment types, honoring data zone boundaries. You can choose balanced mode, cost mode, or quality mode.
Not Diamond Code uses a lightweight local proxy that computes the derived metadata needed for routing to evaluate the broader session, including expected downstream work and cache state, and select the model and reasoning effort that optimize expected quality and total session cost. It uses more capable models when they are likely to improve the outcome or finish in fewer steps, and more cost-efficient models when additional intelligence is unlikely to add value.
NVIDIA’s NeMo Switchyard has an array of routing algorithms that it can use. These include the LLM classifier, which uses an LLM as a judge to select a candidate LLM and maintains session affinity with that model across later turns; the stage router, which examines recent tool activity to decide how much model capability the agent needs; the escalation router, which starts each conversation with a lower-cost model and moves the session to a more capable model when it detects sustained difficulty; and the trainable prefill router, which learns from workload data to predict model success likelihood.
OpenRouter’s Auto Router analyzes your prompt and selects a model based on what the OpenRouter community, in aggregate, uses for that kind of work. The selection is constrained by your cost tier and guardrails. If the automatic primary pick fails, the router uses a fallback.
Replit’s Intelligent Model Routing matches tasks to the model they’re best suited for, aiming to balance quality, speed, and cost.
Snowflake’s Cortex AI Gateway uses dynamic model routing to select the most affordable model that can confidently complete the task at each step of agent execution. Lower-complexity and repetitive tasks can be directed to more efficient models, while workloads that require deeper reasoning can be routed to frontier models.
Also worth noting: the RouteLLM (2024) paper proposed several efficient router models that dynamically select between a stronger and a weaker LLM during inference, aiming to optimize the balance between cost and response quality. It also proposed a training framework for these routers that takes advantage of human preference data and data augmentation techniques to enhance performance.
The costs and benefits of model routing
Model routing potentially provides massive cost reductions, lower average latency, optimized quality, reduced vendor lock-in, enhanced privacy, and improved operational resilience. Strictly on a cost basis, model routing is likely to pay for itself above 100,000 daily active users. Even if you only save 20% on your LLM bill, the savings can be significant, but 50% cost reduction is common, and a 95% cost reduction is not unheard of.
Inference costs, alas, don’t tell the whole story. Some indirect costs include quality evaluation overhead, human review, and infrastructure.
There’s also a nasty failure mode that can, for example, turn your inference cost savings into customer retention and support costs that outweigh them by a factor of four to five: the Pareto trap. To understand that, we should start with Vilfredo Pareto’s 1906 observation that 80% of the land in the Kingdom of Italy was owned by 20% of the population. That was adapted in 1941 by Joseph M. Juran to apply to quality control: 80% of problems stem from 20% of the causes. The Pareto trap refers to concentrating on and over-optimizing for the top 20% to the exclusion of the remaining 80%.
Essentially, tuning your model router to handle only the top 20% of queries (in complexity) with a frontier model risks sending too many of the edge cases (the moderately complex queries) to the
[…]
Content was trimmed to protect the source. Please visit the original article for the full text.
Read the original article: