How to evaluate LLM routing for your application workload

A model router chooses a backing large language model (LLM) for each request. Evaluate it against the direct model you would otherwise deploy, using your own task mix. Approve routing only for categories that meet their quality and policy requirements, then compare cost and latency under realistic load. A lower bill or a higher overall score alone is not enough to make that decision. Microsoft's workload evaluation guidance makes the same baseline and category-level distinction.

This is an engineering decision worksheet for a product lead, technical buyer, or prime contractor defining an AI application work package. It is vendor-neutral; no model or router is endorsed here.

Start with the decision, not a model leaderboard

Write down the deployment choice first: replace the direct model for all requests, route only routine tasks, change the permitted model pool, or retain the current design. A direct model is a valid outcome if it already meets the workload requirements; adding routing introduces another component to operate and observe. Microsoft's model selection guide describes the same choice between stable single-model use and dynamic selection.

Define the unit of success as a completed application task, not merely a plausible model response. For an answer grounded in a document, that might mean the answer cites the right passage and declines when the source is missing. For a classification action, it might mean the correct label is returned in the required schema. Decide who can judge ambiguous cases before running the comparison.

Use the same prompts and application settings on both paths where their capabilities permit. Scrub secrets and personal information before reusing production prompts. Include edge cases and keep category results visible: these are explicit comparison practices in Microsoft's guidance. If the models cannot support the same tool or context length, record the difference as a workload constraint rather than attributing the result solely to the router.

Worked example: the aggregate passes, the critical category fails

The following 100-case fixture is hypothetical. Its counts and dollar amounts are arithmetic examples, not Rainier client results, observed router performance, or a statistically sufficient sample. Assume a team requires at least 95% task success in its high-impact category. It tests the same 80 routine and 20 high-impact cases on a direct baseline and a router candidate.

Suppose the same fictional run costs $10 for the direct path and $6 for the router, including its retries and fallbacks. That is about $0.105 per successful direct task ($10 ÷ 95) versus $0.0625 per successful routed task ($6 ÷ 96). The lower cost does not erase the high-impact regression. Keep those requests on the direct model while investigating the router's errors; consider routing routine requests only after a larger representative comparison and an operational test. Twenty high-impact cases are too few to establish a durable success rate.

This is why the worksheet uses category gates before an overall cost decision. A published LLM routing benchmark also found that, within its own datasets and model set, many routing methods did not beat simple baselines. That finding is a reason to test a real workload, not a prediction about a particular product.

Run the comparison so another team can reproduce it

1. Freeze the fixture. Version the input set, expected task outcomes, category labels, and redaction method. Split development cases from a held-out decision set so prompt tuning does not quietly optimize to the final test.
2. Fix the application boundary. Record prompt templates, retrieval sources, tools, output schema, timeout, and post-processing for each path. If a change is necessary for one candidate, state it in the result.
3. Score the same tasks. Use exact assertions where possible, such as schema validity or a known classification label. For open-ended answers, document the rubric, automated evaluator limitations, and qualified human review of high-impact and disputed cases. NIST's voluntary AI Risk Management Framework measurement guidance calls for documented test sets, metrics, and conditions relevant to deployment.
4. Inspect failures by category. Keep incorrect, refused, timed-out, malformed, and fallback responses separate. An average can conceal the very cases that set the buying requirement.
5. Measure the full path. Record model selected, tokens or other billable usage, retries, fallback invocations, errors, and end-to-end response time. Compute cost per successful task and p95 latency at representative concurrency, not only the first response from an isolated request.

Record the tested model pool and routing policy with the dataset and results. The candidate models, their availability, and pricing can change; Microsoft recommends reevaluating when the traffic mix, application, routing configuration, supported models, or prices change.

What is enough evidence to deploy?

A useful decision record names the categories approved for routing, the categories pinned to a direct model, each unmet gate, and the next test. For an approved category, run a limited production test with a rollback path. Watch task quality, actual usage cost, p95 latency, errors, failover, and selected-model distribution. Microsoft identifies these production signals; thresholds are your application decision, not universal router benchmarks.

For a subcontracted implementation, put the fixture, acceptance gates, selected-model trace, and rollback procedure in the delivery package. The software subcontractor acceptance checklist shows how to specify reproducible evidence and handover. If a routed request calls a write API, the retry and idempotency test matrix covers a separate integration risk that model-quality scores do not measure.

Rainier's public Martian case study documents our work on model-routing evaluation interfaces, benchmark presentation, and customer portals. It does not claim that we built a router backend or produced the hypothetical results above. For an application or integration scope, our software subcontracting page describes the delivery boundaries we can assess. Bring the task mix, allowed model and data constraints, and decision gates to that discussion.

Discuss an AI application evaluation scope→