Auto-retrying to the cloud when your local model gives a weak answer

The fallback pattern is simple: run the request against your local model first, check the response against a quality gate, and if it fails the gate, re-issue the same request to a cloud model before the caller ever sees an answer. The hard part isn't the retry logic. It's defining "weak answer" in a way that catches real failures without triggering on every slightly unusual response.

This matters specifically for coding agents. A chat app can tolerate a mediocre local response because a human reads it and asks a follow-up. A coding agent doesn't do that. It takes whatever the model returns, writes it to a file, and moves to the next step. If a 7B model run through Ollama returns a plausible-looking but broken diff, or truncates mid-function, or drops a tool call it was supposed to make, the agent will often just proceed. You find out an hour later when the build is broken and you're not sure which commit did it.

What actually breaks with local models on agentic workloads

Local models are good at a lot of coding tasks now, but the failure modes on agentic loops are specific and worth naming instead of hand-waving about "quality":

• Truncation on long context. Smaller models run locally often have tighter effective context than their advertised window, and long files get silently cut, producing a partial patch that looks complete.
• Malformed or missing tool calls. Coding agents lean on structured tool-call output (edit_file, run_command). Weaker local models sometimes emit the tool call as prose instead of the expected schema, which the agent either mishandles or ignores.
• Confident wrong answers on unfamiliar APIs. A local model with a smaller or older training cut will invent a plausible-looking method signature rather than say it doesn't know.
• Repetition loops on longer generations, especially at higher context lengths where local inference degrades before a hosted model would.
• Silent degradation under quantization. A 4-bit quant of a strong model can pass casual testing and still fail meaningfully more often on edge-case refactors than the full-precision version.

None of these show up as an HTTP error. The request succeeds, tokens come back, and the agent has no built-in signal that something is off. That's the actual problem a fallback pattern has to solve: catching failure that looks like success.

Building the quality gate

A gate that's too strict burns your cloud budget on every request and defeats the point of running local at all. A gate that's too loose lets broken output through. In practice, a few checks catch most of what matters for coding-agent traffic, and they're cheap enough to run on every response:

• Finish reason. If the local model returns anything other than a clean stop (length-cutoff, content filter, or an error the provider swallowed), treat it as a fail and retry.
• Tool-call schema validation. If the response was supposed to contain a tool call and it either doesn't parse as one or is missing entirely, that's an automatic retry, not a warning.
• Minimum viable shape. For a code edit, does the response contain what looks like a diff or a complete code block, versus a fragment or an apology ("I don't have enough context to modify this file")? A regex plus a length floor catches the obvious cases cheaply.
• Repetition detection. Cheap n-gram repetition checks catch degenerate loops before they get written to disk.
• Optional: a cheap classifier pass. Some teams route the local answer through a second, still-cheap model as a binary judge ("does this look like a complete, syntactically plausible patch: yes/no") rather than a hand-rolled heuristic. This costs a small amount of tokens but catches subtler failures than regex will.

The gate doesn't need to be perfect. It needs to catch the failure modes above without flagging normal variation in style or length. Start strict, watch how often it fires, and loosen the checks that are producing false positives rather than tuning blind.

The retry itself

Once a response fails the gate, the retry has to preserve the original request exactly, including system prompt, tool definitions, and conversation history, and send it to a cloud model. A few details matter more than they look:

• Don't retry against the same local model. If it failed once on this input, it's likely to fail the same way again; escalate rather than retrying in place.
• Log the fact that a fallback happened, not just the final answer. If you don't track how often local fails, you can't tell whether local routing is actually saving money net of the cloud retries it's triggering.
• Pick a fallback model that's meaningfully more capable, not just a different local model. The point is a different quality tier, not a coin flip.
• Set a ceiling. If a request has already failed once locally and once on the fallback, don't keep escalating blindly; surface the failure instead of burning spend chasing a bad prompt.

The retry only pays for itself if you can see, per request, whether it fired and why. A fallback you can't measure is a fallback you can't trust.

Where this fits for a solo developer

I built Probe0 as a local proxy that every coding agent CLI on the machine routes through: Claude Code, Codex, Cursor, all through one proxy and one certificate, no per-tool config. The Local Routing module implements exactly the pattern above: it sends eligible work to whatever model you already have running in Ollama or LM Studio, checks the response against a quality gate, and auto-retries on the cloud when the local answer looks weak. Because it sits at the proxy layer instead of inside any one tool, the gate and the retry logic are shared across every agent you run, instead of being reimplemented per client.

That's also the honest limit of this approach: Probe0 is macOS-only right now, it's in private beta, and it's built for one developer's machine, not a shared team gateway. If you need a fallback pattern that multiple people or CI runners hit through a shared endpoint, you're looking at a hosted or self-hosted gateway with multi-user routing, not a local proxy. Tools like LiteLLM or Portkey are built for that shape of problem and do it well. What Probe0 is for is the developer who's already running a local model for cost reasons and wants the escape hatch built in rather than hand-rolled, plus the other side of that ledger: Exact Cache and Semantic Cache to cut redundant calls before they hit any model at all, Model Tiering for the cheap-first case that doesn't need local, and a Spend Guard that pauses a run before a bad loop turns into a bad bill. Every module reports what it actually saved, on-device, since there's no Probe0 server collecting that data.

The pattern without the tool

If you're not ready to add a proxy, the pattern still holds on its own: define your gate first, log every fallback trigger with a reason, and treat local models as a cost optimization that has to prove itself against the gate, not a default you trust blindly. Local-first routing is a genuinely good idea for most day-to-day coding-agent work. It just needs a safety net that catches the specific ways local inference fails silently, or the savings get eaten by the time you spend debugging output that should never have shipped.