Exact-match vs semantic LLM caching: which one is safe for agents

Short answer: turn exact-match caching on everywhere, and treat semantic caching as an opt-in you enable for specific traffic. An exact cache replays a stored response only when the incoming request canonicalises to the same bytes as one you already paid for, so the worst thing it can do is serve a stale answer. A semantic cache replays a response generated for a different request that an embedding model rated as close enough. On agent traffic, where a single wrong turn feeds into the next twenty, that difference decides whether caching saves you money or costs you a debugging afternoon.

The second-order answer is that the two caches are not competing for the same requests. Exact caching wins on repeated deterministic calls: the same file summarised twice, the same lint-explanation prompt fired by two terminal tabs, the same retry after a network blip. Semantic caching only earns anything on paraphrased natural-language questions, which is a real workload for a support bot and a small minority of what a coding agent actually sends. Choosing between them is mostly a question of what your traffic looks like, and agent traffic looks nothing like the demos semantic caching is usually sold with.

What each cache is actually keyed on

An exact cache hashes the request. The interesting engineering is not the hash, it is deciding which fields belong in it. Model, messages, system prompt, tools, temperature, max tokens, stop sequences and response format all change the completion, so they all go in the key. Per-call bookkeeping like a metadata blob, a user id or a request id cannot change the output, so dropping them is what lets two separate agent runs share an entry. Prompt-cache breakpoints are the interesting case: they change your bill, not the model's answer, and agents shuffle them between turns, so normalising them away is a free hit-rate win.

A semantic cache embeds the prompt into a vector, searches an index for the nearest stored prompt, and serves that stored response if cosine similarity clears a threshold. Every part of that sentence is a place where correctness leaks:

• The threshold is a guess. At 0.85 you will match unrelated code questions that share vocabulary and structure. At 0.98 you will barely hit anything. There is no value that is right for all traffic, and the value that is right for prose is wrong for code.
• Embedding models truncate. If your embedding model has a short input window and you feed it a 40k-character conversation, two conversations that agree for the first few thousand characters and diverge afterwards produce nearly identical vectors.
• Short prompts cluster. Anything under roughly a sentence sits in a dense part of embedding space where unrelated requests are numerically close.
• Embeddings only see text. A request with an attached image or a PDF and a two-line caption embeds as those two lines, so two requests with different attachments look identical.
• Negation is nearly invisible. Do not delete the migration and delete the migration are, to most embedding models, extremely similar strings.

The failure modes are asymmetric, so the settings should be too

A missed cache hit costs you a few cents and some latency. A wrong cache hit hands your agent an answer to a question nobody asked. In a chat product a user reads the wrong answer, frowns, and asks again. In an agent pipeline the wrong answer becomes an edit, a shell command, or the premise for the next five turns, and you find out about it two commits later when the tests fail for reasons that make no sense.

I run semantic caching with a strict floor because a missed hit costs money and a wrong hit costs trust. Those are not the same currency, so they should not share a threshold.

That asymmetry is why the most dangerous requests to serve semantically are the ones carrying tool calls. If a cached response contains a tool_use block, replaying it on a near-match hands the agent a concrete action with arguments computed for a different conversation: a file path from someone else's request, a command from a different repository state. Streaming makes it worse, because a server-sent-event stream carries incremental tool call deltas whose ids have to line up with the live conversation. A replayed stream cannot satisfy that.

Hit rates on agent traffic are lower than the pitch suggests

Semantic caching demos use a FAQ. Ten thousand users ask how do I reset my password in nine hundred phrasings, and one stored answer serves all of them. That is a genuinely good fit and the hit rates are genuinely high.

Coding agents send the opposite shape of traffic. Every turn embeds a growing conversation prefix that includes file contents, tool results, diffs and a system prompt that changes with the working directory. Turn twelve of a run is unique by construction, because it contains the output of turns one through eleven. In my own measurements on tool-call-heavy conversations, semantic hit rates land in the 5 to 15 percent range, and most of those hits are on the short, generic turns where the payoff is smallest. Spending correctness risk to chase that is a bad trade. Meanwhile the exact cache quietly does well on the same traffic, because agents genuinely do resend identical requests: retries, parallel subagents on the same file, and the same warm-up prompt at the start of every session.

This is also where provider prompt caching matters, and it is worth keeping the concepts separate. Anthropic and OpenAI both discount input tokens that repeat a prefix they have already processed. That is a server-side billing mechanism, it does not skip the call, and it is orthogonal to a proxy cache. A proxy cache skips the call entirely. One thing to watch: any middleware that rewrites or compresses your prompts fights prefix caching, because a rewritten prefix is a cache miss upstream. Dropping billing-only annotations from your local cache key is fine. Editing the prompt body to save tokens usually is not.

The tooling landscape, honestly

Most gateways ship both cache types. LiteLLM is an open-source proxy with exact and semantic caching backed by a vector store, and it covers a far wider provider catalogue than anything local-first. Portkey is a hosted gateway with a semantic cache option and the operational maturity that comes with running as managed infrastructure. Helicone gives you caching alongside observability. GPTCache is the open-source library that popularised the pattern. If you need a team gateway with shared keys, org-level budgets and audit trails, those tools win and I would not pretend otherwise.

What none of them are optimised for is one developer, on one laptop, whose traffic is entirely coding agents. That is a different distribution: no user prompts in the classic sense, long tool-heavy conversations, and source code in every request that you may not want leaving the machine at all.

How Probe0 draws the line

I built Probe0 as a local proxy on macOS that every coding agent CLI on the machine routes through, Claude Code, Codex and Cursor included, via one proxy and a certificate installed once. Caching is two separate switchable modules because the two caches deserve different rules.

• Exact Cache is on local disk and never touches the network. It keys on an explicit allow-list of parameters that can change a completion, keeps any field it does not recognise so a new vendor parameter cannot cause a false hit, and drops billing-only annotations like cache breakpoints.
• Semantic Cache uses a local vector index with a strict similarity floor. Configured thresholds below the floor are clamped up rather than honoured, because below roughly 0.9 the embedding model starts rating unrelated code questions as similar.
• Conversations longer than the embedding model can take are refused outright rather than embedded truncated, and requests with images, documents or a temperature that signals intentional sampling variety are skipped.
• No response carrying a tool call is ever served from the semantic cache, and streaming plus tools is refused with no override. The worst case degrades to answering a slightly different question in text, which is the ordinary semantic risk everyone already accepts.
• Recording logs every call with model, tokens, real cost, latency and which process made it, so you can see the hit rate each module produced instead of trusting a marketing figure.

The honest limits: macOS only, private beta, single machine. It is not a team gateway, there is no shared deployment, and provider coverage is what coding agents actually call rather than a hundred-provider catalogue. If you need any of those, use one of the gateways above.

A rule you can apply today

Enable exact caching unconditionally. It has no correctness cost beyond staleness, which a TTL handles. Then look at your own logs before enabling anything semantic: if your traffic is short paraphrased natural-language questions with no tool calls, semantic caching will pay for itself. If it is long agent conversations with tool results in them, set a floor above 0.9, exclude tool-carrying requests entirely, and measure the hit rate for a week before you decide it was worth the risk. A cache you cannot audit is not a saving, it is a bet.