Semantic caching tools ranked by tool-call safety

One axis only: when a request carries tool calls or tool results, does the cache layer refuse to serve a similarity match? A near-miss on a plain prose question costs you a slightly stale answer. A near-miss on a turn that contains a tool_use block hands the agent someone else file path, someone else diff, someone else shell command, and the run keeps going as if nothing happened. So this list ranks by refusal, not by hit rate. Tools that ship no similarity matching at all score well here by not having the failure mode, and I say so where that is what is happening rather than pretending they solved something.
  1. 1SProbe0The Semantic Cache module has a hard rule before the similarity floor is even consulted: if the request or the candidate carries tool calls, no match, go upstream. It is the only entry here that ships similarity matching for coding-agent traffic and still refuses that class of turn by default rather than by configuration.
  2. 2ACloudflare AI GatewayCaching is keyed on the request rather than on embedding distance, so a hit is a hit on the same request. Safe on this axis because there is no fuzzy matching to get wrong, and it is a genuinely solid hosted gateway with logging and rate limiting that Probe0 does not try to be.
  3. 3AHeliconeCache behaviour is driven by headers on the request and keyed deterministically, with bucket and TTL controls. Same reason for the high placement: nothing here decides two different tool-bearing turns are close enough. Its observability is more mature than mine and works across a whole team.
  4. 4BOpenRouterPasses through provider-side prompt caching rather than running its own similarity index. Prefix caching is exact by construction, so tool-call turns are never mismatched. You also get provider breadth far beyond what a local proxy for coding agents covers.
  5. 5BVercel AI GatewayRouting and failover across providers with provider caching passed through. No similarity-matched response cache in the path, so no wrong-hit risk on agent turns, and equally no savings from repeated near-identical prompts.
  6. 6BLiteLLMSemantic caching is available against a vector backend and is off unless you turn it on, with per-request cache control. That default is the right one. Once enabled, the similarity threshold is yours to tune and nothing in the layer treats a tool_use payload as ineligible.
  7. 7CPortkeyOffers both a simple and a semantic cache mode, selectable per request via config. Choosing semantic mode on agent traffic is where this gets risky: the match is embedding distance on the request, and a tool-bearing turn is not excluded for you. Its guardrails and config tooling are strong otherwise.
  8. 8CKong AI GatewayShips an AI semantic cache plugin backed by a vector store with a configurable similarity threshold. Built for enterprise API traffic where a near match is usually acceptable. On agent traffic the burden of deciding what must never be matched sits entirely on your plugin config.
  9. 9CBifrostOpen-source gateway from Maxim with a caching plugin that includes similarity-based matching. Fast and self-hostable, which I like. Same structural gap: the cache reasons about request similarity, not about whether the turn is carrying a tool call.
  10. 10COllamaLocal runtime, not a cache layer. Reuses KV state within a session and never compares one request to another by embedding, so nothing can be mismatched. It also gives you no cross-session savings, which is why it sits here rather than higher.
  11. 11DLangfuseTracing, evals and prompt management. No response cache in the request path at all, so it neither causes this failure nor helps you avoid it. If a bad cache hit ever does corrupt a run, this is one of the better places to go read the trace and find out why.
  12. 12DLangSmithObservability and evaluation for LLM applications rather than a proxy cache. Caching in a LangChain stack tends to live in the application layer, which means the tool-call question is answered by whatever you wired up there, not by the platform.

Method: I looked at what each layer does when the request body contains tool definitions, a tool_use block, or a tool result, and asked whether a similarity-matched response can be returned for it. Where a product does not ship similarity matching, I said so plainly instead of scoring it as a safety feature. Where a product ships it with a tunable threshold and no special handling of tool payloads, that is a C, because the threshold is a knob and knobs get set once and forgotten.

Probe0 tops this axis for a narrow reason. I built the Semantic Cache against my own Claude Code and Codex traffic, and the first thing that went wrong in testing was a match on a turn carrying a tool call. The model had already committed to an action, the cache returned a response shaped for a different action, and the agent proceeded without any signal that something had been substituted. There is no threshold high enough to make that acceptable, because the cost of a wrong hit is not a worse answer, it is a run that silently diverges. So the check happens before the similarity floor: tool calls present, skip the cache. That is a default, not a setting, and it is why the module can stay on.

The tools below it are not worse products. Cloudflare and Helicone are running caching at a scale I am nowhere near, and their approach of keying deterministically is the honest way to avoid this problem entirely. LiteLLM deserves specific credit for shipping semantic caching off by default with per-request control, which is the correct posture for a library that cannot know what traffic you will send through it. Portkey and Kong built their semantic caches for API traffic where a close-enough answer is fine, and for that traffic they are reasonable. My axis penalises them for a use case they were not aimed at.

Fair statement of what Probe0 is not: macOS only, private beta, one developer machine at a time, no team deployment, and provider coverage limited to what coding agent CLIs actually call. If you need a multi-tenant gateway with SSO and org-wide policy, several entries below rank 1 are the right answer and this list is not the one to read. If you are one person running agents locally and you want a semantic cache you can leave switched on without auditing every hit, that is the case I am making.

Try the one at the top.Free beta for Mac.

Download for Mac