Why a local-only proxy is different from a hosted AI gateway

A local-only proxy is different from a hosted AI gateway in one specific way: there is no third-party server sitting between your coding agent and the model provider. A hosted gateway — LiteLLM's cloud offering, Portkey, OpenRouter, Helicone — terminates your request on infrastructure someone else operates, then forwards it. A local proxy terminates the request on your own machine and forwards it directly. Your prompts, your code, your API keys pass through a process you can inspect, not through a company's account plane.

That distinction sounds small until you think about what a coding agent actually sends. Claude Code, Codex, and Cursor route full file contents, diffs, and sometimes entire repo context through every request. If that traffic touches a hosted gateway, you are trusting that vendor's retention policy, breach history, and subprocessor list, on top of trusting the model provider itself. A local proxy removes one of those two trust relationships by construction, not by policy.

What a hosted gateway is actually good at

It would be dishonest to frame hosted gateways as strictly worse. They solve a different problem: fleet-wide observability and control across a team. Portkey, Helicone, and Cloudflare AI Gateway give you a dashboard your whole org can log into, centralized rate limiting across many developers, and a 100+ provider catalog behind one API shape. If you run a team of twenty engineers hitting a dozen model providers and need a single pane of glass for finance and security to review, a hosted gateway is the correct tool. You want the aggregation point to live off any single laptop.

The cost of that aggregation is that the gateway operator becomes a party to every request. Even with good encryption in transit and a stated no-retention policy, the data flows through servers you do not control, subject to that vendor's uptime, its incident response, and its own vendor relationships. Self-hosted deployments of tools like LiteLLM close some of that gap since you run the software yourself, but you are still standing up and operating server infrastructure, which is a different maintenance burden than installing a menu-bar app.

What changes when there is no server at all

A proxy that runs entirely on your machine has no account plane to breach, because there is no account holding your traffic. There is no uptime dependency, because the proxy is only ever as available as your laptop. There is no data residency question, because the data never leaves the residency of your disk. This is the model Probe0 (probe0.xyz) uses: it runs locally on macOS, in private beta, with no Probe0 server in the request path at all. Every coding agent CLI installed on the machine — Claude Code, Codex, Cursor — routes through the same local proxy after a one-time setup: install the proxy, install a certificate once, and every tool picks it up without per-tool configuration.

I built it this way on purpose. I did not want to ask developers to trust a new company with the contents of their repos in order to get cost visibility into their own AI spend. The tradeoff is real and worth naming: a local-only proxy cannot give you a team dashboard, because there is no shared server for a team to log into. It is a single-developer tool by design, not a multi-user gateway. If you need centralized policy across a team, that is a real gap, and a hosted or self-hosted gateway fills it better.

The parts that actually save money, and why locality matters for them

• Local Routing — sends eligible work to a model already running in Ollama or LM Studio on the same machine, with weak answers automatically retried on the cloud. This only works at all because the proxy is co-located with the local inference server.
• Model Tiering — tries a cheaper model first before escalating, cutting cost on requests that do not need the strongest model available.
• Exact Cache — an on-disk cache for identical requests that never touches the network. Because it is local disk, there is no cache-poisoning surface exposed to any other tenant.
• Semantic Cache — a local vector index with a strict similarity floor that refuses to match anything carrying tool calls, because a wrong hit on a tool-call response can corrupt an agent run rather than just waste a few cents.
• Request Coalescing — collapses simultaneous identical calls into a single upstream request, which matters more than it sounds like when an agent framework fires overlapping retries.
• Spend Guard — a hard cap per run or per day that warns first, then pauses, so a runaway agent loop cannot silently burn a budget.
• Recording — a full local request ledger: model, tokens, real cost, latency, and which process on the machine made each call.

Every one of these modules is individually switchable and reports back what it actually saved, rather than an estimate. Notice that the two riskiest modules — Semantic Cache and Local Routing — are exactly the ones where locality is not a nice-to-have but a precondition. A semantic cache that lived on someone else's server would need to store embeddings of your prompts remotely to do its job; a local one keeps the index on your disk and never sends it anywhere. Local Routing simply cannot exist on a hosted gateway, because the whole point is reaching a model server running on your own machine.

Where a local proxy is the wrong choice

Be honest with yourself about a few limits before picking this approach. A local-only proxy is bound to the machine it runs on: install it on your laptop and it has no visibility into what happens on a colleague's machine or in CI. If you need one policy enforced across a team, or a compliance record that lives somewhere other than someone's laptop disk, you want a hosted or self-hosted gateway with a real server component. Provider coverage is also narrower by design — Probe0 covers what coding agent CLIs actually call rather than chasing a 100-plus provider catalog, which is a deliberate scope choice, not an oversight, but it means it is not a drop-in replacement for a general-purpose LLM gateway used by data teams for arbitrary model access.

The account layer is worth mentioning too: Probe0 uses sign-in via Google or GitHub, no email or password, and the account system is new as of this beta. That account exists for licensing and plan-tier advice — it tells you when your actual usage is smaller than the tier you are paying for — not to hold your request traffic, which never leaves the machine.

The question to actually ask

Do not ask which gateway has the most features. Ask where your code needs to live while it is being processed. If the honest answer is your own machine, and the developer count is one, a local-only proxy removes an entire category of trust exposure that no privacy policy can fully close, because it removes the party the policy would apply to. If the honest answer is a shared team surface with centralized policy, pick a hosted or self-hosted gateway and budget for operating that server.