Setting a similarity floor so semantic cache never corrupts an agent run

The direct answer: for a coding agent, a semantic cache should only serve a cached response when the cosine similarity between the incoming request and the cached one is at or above roughly 0.97 to 0.99 on a normalized embedding, and it should refuse to match anything at all when either the cached turn or the incoming turn carries a tool call. Below that floor, or in the presence of tool calls, the wrong tokens go back into the model's context and the agent silently starts working from bad state. That failure is invisible in a way a wrong chat answer isn't, because the agent keeps executing on top of it.

Semantic caching for chat products can afford to be loose. If a user asks a slightly different phrasing of the same question and gets a near-duplicate answer, the cost of a miss is a mildly off response the user can just re-ask. Coding agents don't get that grace period. A tool call result, a file diff, a shell exit code — these become facts the agent reasons over for the rest of the run. If a semantic cache hands back a cached tool result that resembles the real one at 0.91 similarity but isn't the same file, isn't the same directory, isn't the same error, the agent proceeds as if it is. It writes the next patch against imaginary state. By the time that shows up as a broken build or a deleted file, the actual cause is three turns back and doesn't look like a cache bug at all.

Why 0.97-0.99 and not 0.85

Most semantic cache tutorials online default to something like 0.8 to 0.85 similarity, because that's a reasonable threshold for FAQ-style retrieval where near-paraphrases genuinely share an answer. Coding prompts don't behave like FAQ questions. Two requests that embed at 0.9 similarity can differ by one changed variable name, one different file path, or one flipped boolean — details that change the correct answer entirely but barely move the embedding. Natural language is forgiving of paraphrase; code and file state are not. Pushing the floor up into the 0.97-0.99 range cuts the hit rate substantially, but it's the right trade: a cache that answers less often and never lies is more useful to an agent than one that answers often and is occasionally wrong in a way nothing catches.

• 0.99+: near-exact rephrasing, safe to serve almost always
• 0.97-0.99: workable floor for coding-agent traffic where wrong hits are costly
• 0.90-0.97: fine for consumer chat FAQ retrieval, too loose for tool-using agents
• Below 0.90: not really a cache hit, it's a different request that happens to embed nearby

The threshold isn't the real gate — tool calls are

Tuning the number is the easy 80%. The harder rule, and the one that actually prevents corruption, is refusing to match anything that includes a tool call, on either side of the comparison. A high similarity score tells you the request text looks alike. It tells you nothing about whether the underlying files, environment, or command output are still the same as when the cached turn ran. A grep result from ten minutes ago is not a grep result now if the file changed in between. An agent mid-run has no way to know the cache handed it stale ground truth, because nothing about the response format signals staleness.

So the safer design isn't a smarter similarity model, it's a narrower scope: cache eligibility should be gated by turn shape before it's gated by embedding distance. Plain text-in, text-out turns are eligible for the similarity check. Anything carrying a tool call, tool result, or file read/write is excluded outright, full stop, regardless of how close the embedding lands. That single exclusion rule removes most of the actual risk, and it's a rule you can implement without touching the embedding model at all.

What to still cache exactly, not semantically

The turns that get filtered out of semantic matching aren't wasted — they're just better served by exact caching instead of similarity matching. If an agent issues the identical request byte-for-byte (same prompt, same tool definitions, same context) within a short window, an exact-match cache keyed on a hash is safe to serve regardless of tool calls, because there's no interpretation step where a near-miss can sneak in. The two caching strategies solve different problems: exact cache catches repeats and retries, semantic cache catches paraphrase, and only the second one carries the corruption risk. Keeping the two paths separate, with separate eligibility rules, is simpler than trying to make one model do both jobs safely.

A cache that answers less often and never lies is more useful to an agent than one that answers often and is occasionally wrong in a way nothing catches.

How Probe0 draws this line

I run into this directly because Probe0 is a local proxy that sits in front of Claude Code, Codex, and Cursor and caches at the request level. The semantic cache module uses a local vector index with a strict similarity floor, and it refuses outright to match any turn carrying a tool call — the exclusion described above isn't a tunable option, it's the default because a wrong hit corrupts the run and there's no way to undo an agent's next three actions once it's happened. The exact-match cache is separate, reads off local disk, and never touches the network, so retries and identical repeats get served without going anywhere near the similarity threshold at all. Everything runs on the machine; there's no Probe0 server, so the cached content and the embeddings never leave your disk either.

None of this is exotic engineering — a strict cosine floor plus a hard tool-call exclusion is maybe twenty lines of logic. The reason it's worth writing down is that the default advice available for semantic caching is tuned for chat products, and applying that advice unchanged to an agent that executes tool calls on top of what the cache returns is how you get a cache that looks like it's working right up until it silently isn't.