Model Routing Is an Evaluation Problem

Model routing is an evaluation problem with a billing department.

The usual routing pitch starts with a price list. A large model handles hard work. A smaller one handles the easy work. The application saves money by putting a classifier in front of the expensive call. This sounds like ordinary infrastructure. Pick the cheaper dependency for the requests that do not deserve the premium one.

That framing leaves out the only question that matters: which requests can the smaller model complete well enough for the job that follows?

“Easy” is usually a feeling somebody had while looking at a few successful examples. A successful example only establishes that a model can produce good work. It Worked Once makes the larger point about why that does not establish a production rate. A request can be short and still need a careful answer. A request can be long and have a mechanical answer. A model can produce a tidy result in the right schema while missing the one fact that matters. If the route is based on a feeling of simplicity, it will quietly turn quality failures into cost savings.

The route needs an acceptance threshold. The cheap path has to meet a stated standard on the work it is actually being asked to do. When it misses, the system needs a known place to send the request next. That makes routing a small evaluation system. The price difference only tells you why it is worth building.

Cost Per Accepted Result

AI Inference Cost Is Architecture makes the larger case for keeping ordinary control flow out of the model. Routing can do that well. A program can send an extraction job to a smaller model, validate the returned fields, and escalate only the records that fail. The expensive model stays available for the residue instead of reading every invoice, form, or support request.

The tempting version uses a label such as simple or complex and calls it done. It often comes from a model prompt: “Classify this request by difficulty, then select a model.” That creates a strange arrangement. The system pays a model to decide whether another model is good enough, then treats the first model’s opinion as a quality guarantee.

Route on acceptance criteria rather than a generic difficulty label.

A shipping-address extraction task may be acceptable when all required fields parse, the address validates, and the order system accepts it. A document-summary task may be acceptable when named claims match the source, the length fits the destination, and a reviewer accepts it without a rewrite. An agent action may be acceptable only when the action passes policy checks and produces the intended state.

Those are different jobs. Each needs its own evidence. The model’s confidence is evidence of very little. So is a clean JSON object. Without an acceptance test, routing is a price cut without evidence that it preserves the required result.

The Evaluation Belongs Before the Route

The evaluation starts with a representative set of real requests. Not the clean examples from the demo. It includes strange formats, incomplete records, requests that tend to be escalated, and cases a human corrected last week. Task classes stay separate when they have different failure costs.

Candidate models then receive the same requests with the same prompt, tools, and output contract. The measurement has to capture what makes the output usable. The OpenAI evaluation system, for example, separates the data source from the graders used to assess a sample.¹ The product names do not matter. The separation does. A route needs a corpus of requests and a test for the result.

The last column catches the trick in token-only thinking. A smaller model that costs one tenth as much but sends half its requests through a human correction loop can be more expensive than a larger model that finishes the job. A route that gives a quick wrong answer can be far worse. The denominator is not calls. It is accepted results.

This also prevents a common accounting lie. Suppose the smaller model handles ninety percent of the easy cases. That number sounds good until the remaining ten percent are the cases that produce refunds, rework, or an angry customer. The average call cost went down. The system may have become less useful. Acceptance rates and failure costs put those facts in the same conversation.

Acceptance Signals Have to Touch Reality

The best routing tasks have a check close to the work.

An extraction result can match a reference record. A classification can be compared with a resolved case. A code change can run tests. A proposed database action can be checked against permissions and invariants before it runs. These checks do not need to be perfect. They need to be correlated with the decision the route is making.

Some work has no immediate mechanical check. A sales email may only prove useful when a person replies. A strategic recommendation may be judged months later. A draft may depend on taste. Those tasks can still be measured, but the feedback is slower and noisier. That raises the bar for routing them to a weaker model. A simple request with no usable acceptance signal belongs on the stronger route by default.

This is the asymmetry of verification showing up in a smaller decision. Cheap checks make cheap routes possible. Without a check, the team is not learning which model is sufficient. It is guessing from prose that looks plausible.

One warning: a rubric model can be useful, but it is still a model. If a model grades another model’s answer, test the grader against human judgments before making it the gate. A weak grader will bless the same polished mistakes the generator makes. The grading system earns trust through calibration, sampled review, and a record of the misses it failed to catch.

Escalation Is Part of the Product

Every route needs a way to decline the cheap answer.

Sometimes the validator fails. Sometimes a calibrated risk signal, tested against the evaluation set, crosses the escalation boundary. Sometimes a request has a feature that the evaluation set shows is risky: a low-quality scan, a new policy version, a foreign-language attachment, an unusual account state. The route sends it to a stronger model, a dedicated process, or a human queue.

That path cannot be an embarrassment the design hides. It is the mechanism that lets the system use a cheaper model safely. A route with no escalation path turns its known failures into customer-facing output. A route with a bounded fallback turns an uncertain request into a more deliberate one.

The escalation record becomes the next evaluation data. If a specific request class keeps escaping to the expensive model, there may be enough volume to build a better prompt, a new deterministic preprocessor, or a task-specific model. If accepted cheap results later get corrected, the evaluation set is missing a case. If the expensive model cannot recover the request either, routing was never the real problem.

The learning loop should change the route cautiously. A fresh model release, a prompt rewrite, a source-data change, or a new validator can all move the acceptance rate. A route is production behavior. A model swap merits comparison on the current corpus, a defined traffic slice, and a look at the failures.

The Classifier Is a Route Too

The request classifier deserves the same suspicion as the generation model.

For many systems, the cleanest classifier is ordinary code. Known language, attachment type, account tier, required tool, data sensitivity, or a hard policy boundary can decide a route without inference. That is cheaper and easier to test.

When the classifier needs semantic judgment, evaluate it separately. A classifier can fail in two directions. It can over-escalate, which wastes money. Or it can under-escalate, which sends a request to a model that has not earned it. Those failures have different costs, so the threshold should not be a generic accuracy score. A benefits-summary request may tolerate more over-escalation. A payment or permission change should be biased toward the stronger path.

This is where route design becomes honest about risk. The team stops asking whether the smaller model is “good enough” in the abstract. It asks whether this particular task, under this check, with this fallback, is cheap enough to accept at this rate. That is a question the system can answer with data.

Price Is the Last Input

Price still matters. Providers charge different rates for input, cached input, and output, and those rates can change the economics of a route.² But pricing comes after the quality boundary. The initial comparison establishes which models clear the acceptance threshold. Cost per accepted result, latency, capacity, and failure behavior can then decide among them.

Tasks without a useful acceptance signal stay on the stronger path. For the rest, total cost per accepted result, including escalation and correction, is the deciding measure.

Footnotes

1. OpenAI, “Evals API reference”, which defines evaluation data sources and graders as separate parts of an evaluation. ↩

2. OpenAI API pricing lists separate input, cached-input, and output rates. ↩