Model tiering: routing to a cheap model first, only escalating on failure

Model tiering means every request starts on the cheapest model that has a reasonable chance of succeeding, and only gets retried on a stronger, pricier model when the cheap attempt actually fails. The mechanism is small: send the request, run a cheap pass/fail check on the response, and if it fails, resend the same request to the next tier up. The hard part isn't the routing logic. It's deciding what counts as failure without writing a bespoke grader for every agent workflow you have.

This matters specifically for coding agents because their traffic is lopsided. A large fraction of the calls a CLI agent makes aren't the hard reasoning step — they're file reads, small edits, git status checks, tool-result summarization, single-function patches. Those don't need a frontier model. The minority of calls that involve multi-file reasoning, ambiguous specs, or long-horizon planning do. If you route everything to one model, you're either overpaying on the easy 70% or under-provisioning the hard 30%. Tiering routes each call to what it actually needs.

The core pattern

The shape is the same regardless of which agent framework you're in:

• Define an ordered list of tiers, cheapest first — e.g. a small fast model, then a mid model, then your frontier model.
• Send the request to tier 0.
• Run a fast, deterministic check against the response: did it error, come back empty, get truncated, fail schema validation, refuse, or land outside expected bounds (like a tool call referencing a file that doesn't exist)?
• If the check passes, return the response. Done, at tier-0 cost.
• If the check fails, resend the identical request to tier 1 and repeat the check. Continue up the tier list until something passes or you hit the top tier, which you always trust as a final answer.

The two design decisions that actually determine whether this saves money without breaking things are the failure check and the retry budget. Get either wrong and tiering either does nothing (everything escalates) or does damage (bad answers ship because the check didn't catch them).

What counts as a failure worth escalating on

Resist the urge to build an LLM-as-judge grader for this. Judging costs a call itself, and if you're trying to cut cost, adding a grading call on every request works against you. Start with checks that are free or nearly free to run:

• Transport failures: non-2xx from the provider, timeout, empty body.
• Schema failures: if you're asking for structured output or a tool call, does it parse and match the expected shape?
• Refusal or hedge detection: cheap string matching against a short list of refusal patterns catches more than people expect.
• Truncation: response cut off at the token limit mid-structure is an obvious signal.
• Downstream validation, when you have it: for code edits, does the patch apply cleanly and does the file still parse? That's a much stronger signal than anything the model says about itself.

That last one is the strongest lever available to coding agents specifically, and it's underused. A model's own output can be self-consistent nonsense, but a patch either applies or it doesn't, and code either parses or it doesn't. If your agent harness already runs the edit before showing it to the user, wire that check into the tiering decision instead of treating it as a separate step.

Retry budget and where tiering breaks

Cap the number of tiers you'll walk through per request, and cap it low. Two escalations is usually enough; three should be a rare ceiling, not a norm. Every escalation adds latency on top of the failed attempt's latency, and past a certain point you've spent more wall-clock time failing cheap than you would have spent just calling the expensive model once. Tiering optimizes cost, not latency, and you have to decide up front which one you're willing to sacrifice for the other on a given workflow. For a background batch job, walk three tiers. For an interactive agent turn where a developer is staring at the terminal, one retry before you give up and go straight to the top tier is often the right call.

The other place this breaks is state. If a call in the middle of an agent turn has side effects — it already wrote a file, called an API, moved a git branch — you can't cleanly retry it on a different model without either making the operation idempotent or rolling back first. Tiering is safe by default on pure generation calls (draft this commit message, summarize this diff, plan the next step) and needs explicit handling on anything that mutates state. Most agent frameworks already separate the "decide what to do" call from the "do it" tool execution, which is the natural place to put the tier boundary — tier the decision call, not the execution.

A minimal implementation

In pseudocode, ignoring provider-specific request shapes:

tiers = [cheap, mid, frontier] for model in tiers: response = call(model, request) if passes(response, checks): return response if model is tiers[-1]: return response # trust the top tier even on a soft fail raise UnreachableError

The part worth getting right is that the request itself doesn't change between tiers. Same prompt, same tools, same context. If you find yourself rewriting the prompt for the stronger model — more instructions, more examples, more scaffolding — you've built two separate systems that happen to share a retry loop, not real tiering. That's fine as a design choice, but be honest that you've added a second maintenance burden, not a free win.

Where this fits in a local proxy

I built Model Tiering as one of the switchable modules in Probe0, a proxy that runs locally on macOS and sits between coding agent CLIs — Claude Code, Codex, Cursor — and the model providers they call. Setup is one proxy plus a certificate installed once; every agent on the machine routes through it without per-tool config. Tiering is one module among several (there's also Local Routing for models already running in Ollama or LM Studio, Exact and Semantic caching, Request Coalescing, and a Spend Guard), and each one reports what it actually saved in the request ledger, not an estimate.

The reason to do this at the proxy layer instead of inside each agent is that the failure-detection logic — transport errors, schema mismatches, truncation — is identical regardless of which CLI generated the request, but each agent would otherwise need its own retry wrapper. Sitting between the agent and the provider means the tiering module sees every call once, in one place, and the semantic cache in the same proxy refuses to match anything carrying tool calls, which matters here too: a bad cache hit on a tool-calling turn is worse than a slow one, same as escalating too eagerly is worse than not tiering at all.

Where a proxy-level approach loses to something purpose-built: if you need per-team budgets, a hosted dashboard multiple people log into, or centralized routing across a fleet of servers, a local single-machine proxy is the wrong tool — that's a real gap, not a hedge. Probe0 is built for a single developer's machine, not a team gateway, and it's private beta on macOS only right now.

The takeaway

Model tiering is a small piece of routing logic — ordered tiers, a cheap failure check, a low retry cap — that pays for itself on any workload where most requests are easy and a minority are hard, which describes most coding agent traffic. Build the failure check out of things you can verify deterministically (parsing, schema, patch application) before reaching for an LLM judge, keep the retry budget short enough that escalation latency doesn't eat the savings, and be careful about tiering anything with side effects. Whether you wire it into your own agent loop or run it as a proxy module in front of whatever CLI you already use, the logic is the same either way.