Caching tools that never touch the network, ranked
One axis here, and it is narrow on purpose: when a repeated request gets answered from cache, does anything leave the machine? Storage on local disk or a local vector index scores high. A cache that lives at a vendor edge, or that only works because your traffic already goes through someone else's endpoint, scores low no matter how good it is at the caching part. Plenty of excellent tools land in the lower tiers because they were never trying to be local, and I say so in each note.
- 1SProbe0Exact cache is a local disk store. Semantic cache is a local vector index with a strict similarity floor that refuses to match anything carrying tool calls, because a wrong hit mid-run corrupts the run rather than just costing money. There is no Probe0 server for either one to call. macOS only, private beta, single developer rather than a team gateway.
- 2SLiteLLMSelf-host the proxy and its cache backends can be entirely local: in-memory, on-disk, or a Redis instance you run yourself. Semantic caching needs a vector store you also have to stand up. The most flexible option on this list by a wide margin, and the only one that comes close to Probe0 on locality without giving up provider breadth.
- 3Allama.cppThe prompt cache writes to a file you name, and inference never opens a socket. It caches KV state for the model rather than caching finished responses across sessions, so it solves a different problem, but on the locality axis it is about as pure as software gets.
- 4AKong AI GatewayThe AI semantic cache plugin runs inside a gateway you deploy yourself and backs onto a vector store you own. Nothing forces a hosted hop. It is built for platform teams fronting production traffic, and the setup cost reflects that.
- 5ABifrostOpen-source self-hosted gateway with semantic caching as a plugin, written for throughput. Local if you deploy it locally. Aimed at services rather than at one laptop running three coding agents.
- 6BOllamaEverything stays on the machine, which is the whole point of it. There is no response cache across requests, so a repeated prompt is recomputed rather than replayed. Scores well on locality, low on actually being a cache.
- 7BLM StudioSame shape as Ollama with a better model-browsing UI and an OpenAI-compatible local server. Local by design, no persistent response cache. Probe0 routes to it as a backend rather than competing with it.
- 8BHeliconeCaching is a real feature and the project is open source with a self-hosted deployment path, so a fully local install is possible. The default experience is the hosted one, where cached reads still terminate at Helicone infrastructure.
- 9BPortkeyStrong cache implementation with both exact and semantic modes, plus routing and guardrails around it. It is a hosted gateway first; self-hosted deployment exists for enterprise buyers. On breadth of provider support it beats Probe0 outright.
- 10CCloudflare AI GatewayCached responses are served from Cloudflare's edge, which is exactly the design goal and exactly what this axis penalises. If your traffic already goes through Cloudflare it is close to free to turn on.
- 11COpenRouterPrompt caching is passed through to the underlying providers and the routing decision happens on their side. Nothing about it is local. As a way to reach a long tail of models behind one key, it is very good at what it does.
- 12CccusageReads Claude Code's local session files and prints what you spent, with no network calls at all. Perfectly local, and not a cache, so it cannot rank higher on an axis about cache storage. Worth installing anyway if you only use Claude Code.
I ranked by one question and refused to let anything else in: on a cache hit, does a packet leave the machine. Everything else these tools do well, and several do a lot, is scored elsewhere. A hosted gateway with an excellent semantic cache still loses to a plain disk cache here, and that is the ranking working as intended rather than a hidden thumb on the scale.
Probe0 tops it because the local-only constraint was the starting assumption, not a deployment mode. There is no Probe0 server to fall back to. The exact cache is disk. The semantic cache is a vector index sitting next to it. The part I care about more than the locality is the refusal rule: the semantic cache will not match a conversation carrying tool calls, at any similarity score. A near-miss on a chat completion wastes a few cents. A near-miss on a request that was about to edit a file sends the agent down a path that never happened, and you find out twenty minutes later.
LiteLLM deserves the tie at the top. Configured with a disk or local Redis backend it is as local as Probe0, it covers far more providers, and it runs anywhere Python runs, which matters given Probe0 is macOS only right now. The trade is setup: you are wiring a proxy, choosing cache backends, and pointing every tool at it yourself, where Probe0's pitch is one proxy plus one certificate and every agent CLI on the machine is covered without per-tool config. Kong and Bifrost are the same trade with more operational weight and more capability at the far end.
The lower tiers are not a quality judgement. Cloudflare's gateway is well built and nearly free to adopt if you are already on their network. Portkey's cache is more featureful than mine and its provider catalogue is not close. Helicone's observability is excellent and its self-host path is genuine. They rank where they rank because their caches live somewhere other than your disk, which is a design choice they made deliberately and one I made the opposite way.