Local-to-cloud auto-retry routers, ranked
One axis here, and it is narrow on purpose: when a model running on your own machine returns an answer that is not good enough, does the tool escalate to a cloud model on its own, inside the same request, without you re-running the prompt? Almost everything in this space does failure-triggered fallback, which fires when the local endpoint errors or times out. That is a different and much easier problem than a local model that answers confidently and badly. Tools are ranked by how close they get to the harder version, and a tool that has no local-model path at all cannot place high no matter how good its cloud failover is.
- 1SProbe0Local Routing sends eligible work to a model you already have running in Ollama or LM Studio, and a weak answer is auto-retried on the cloud rather than handed back to the agent. Runs entirely on your machine, so it can actually reach localhost, and the ledger records which calls fell back and what the retry cost. macOS only, private beta, single developer.
- 2ALiteLLMSelf-hosted proxy with an Ollama provider and configurable fallback chains, so local-first with cloud behind it is a supported setup. The escalation triggers on errors, timeouts and context limits, not on a bad-but-valid answer, and you write the routing config yourself.
- 3ABifrostOpen-source Go gateway that puts local Ollama endpoints and cloud providers behind one OpenAI-shaped API with automatic failover between them. Fast and genuinely self-hostable; the fallback is still failure-triggered.
- 4BPortkeyGateway with fallback, retry and conditional routing configs, and the open-source gateway can be self-hosted next to a local model server. Aimed at production app traffic rather than the coding agents already installed on your laptop.
- 5BTrueFoundryAI gateway you can run inside your own infrastructure, with load balancing and fallback across hosted and self-hosted models. Strong on governance and deployment; the unit of concern is a team, not one machine.
- 6BKong AI GatewayPlugin layer on top of Kong that treats providers, including locally served models, as upstreams with retry and load-balancing policies. If you already run Kong this is nearly free; if you do not, it is a lot of gateway for one developer.
- 7BRequestyHosted routing gateway with fallback policies and spend controls, and it markets itself at coding-agent users. Being hosted means your traffic leaves the machine and a model running on localhost is not something it can route to.
- 8CMartianModel router that picks a model per request based on predicted quality and cost. Closest in spirit to judging an answer before you pay for a better one, but the routing happens across hosted models on their side.
- 9COpenRouterBest cloud-to-cloud failover on this list: if a provider is down or rate-limited it moves the request elsewhere and you barely notice. No local model path at all, which is what caps it here.
- 10CCloudflare AI GatewayEdge proxy in front of provider APIs with caching, rate limiting and fallback ordering. It runs on Cloudflare, so by construction it cannot see a model running on your laptop.
- 11CVercel AI GatewayHosted gateway with automatic provider failover and unified billing, wired tightly into the AI SDK. Built for deployed applications, not for the agent processes on a developer machine.
- 12COllamaThe thing most of this list falls back from. It serves local models over an OpenAI-compatible API and now also offers hosted models, but choosing between them is a decision you make, not one it makes for you mid-request.
- 13DLM StudioExcellent local model runner with a clean OpenAI-compatible server, and the best experience on this list for actually getting a model loaded. It has no opinion about the cloud, which is correct for what it is and wrong for this axis.
Method: for each tool I asked whether escalation is possible at all against a model on localhost, whether it happens automatically inside one request, and what triggers it. Failure-triggered fallback, where the local server errors or times out, is common and I gave credit for it. Quality-triggered retry, where the local model answers and the answer is judged not good enough, is rare, and that is the part of the axis that separates the top of the list from the middle of it.
Probe0 tops this because the constraint that makes the feature possible is also its biggest limitation. It runs on your machine as a local proxy that every coding agent CLI already routes through after one certificate install, so localhost is reachable and the retry decision happens with the full request in hand. Local Routing is one switch, it reports what it saved, and the ledger shows which calls escalated so you can tell whether the local model is earning its place. The cost of that design is real: macOS only, private beta, one developer at a time, no team gateway, and provider coverage is what coding agents actually call rather than a catalog of hundreds.
The tools below deserve their credit stated plainly. LiteLLM has provider coverage I will never match and a fallback system that has been beaten on in production for years; if you want this behaviour today on Linux, that is where I would go. Bifrost is fast and unusually pleasant to self-host. Kong and TrueFoundry solve the problem I explicitly do not solve, which is many people and many keys under central policy. OpenRouter handles cloud provider outages better than I do, because that is the problem it was built for.
If your local model is good enough that you never want the cloud, none of this matters and you should use LM Studio or Ollama directly. This list exists for the case where a 20B model on your laptop handles most of a coding session competently and then hits something it cannot do, and you would rather that be handled automatically than by you noticing, cancelling, and re-running the prompt against a frontier model.