Why semantic caching breaks agent tool calls (and how to avoid it)

Semantic caching breaks agent tool calls because it answers a different question than the one you asked. Exact caching returns a response only when the request is byte-identical. Semantic caching returns a response when the request is close enough in embedding space, and "close enough" is a similarity score, not a guarantee of matching intent. A coding agent's tool call is a function name plus a specific set of arguments: a file path, a line range, a shell command, a diff. Two tool calls can be 94% similar by cosine distance and still point at two different files. Serve the cached one and your agent edits the wrong file, or reports a grep result from a different repo, or replays a stale git diff as if it just ran. The agent has no way to tell the cached answer apart from a fresh one, so it keeps building on top of it.

Why tool calls are a worse fit for semantic caching than chat

Semantic caching was built for a different traffic shape: FAQ bots, support deflection, RAG answers to natural-language questions where two rephrasings of "how do I reset my password" should legitimately hit the same cached answer. That works because the answer's correctness doesn't depend on exact surface form. A tool call is the opposite. The correctness of "read src/auth/session.ts lines 40-80" depends entirely on the exact string src/auth/session.ts and the exact numbers 40 and 80. There is no fuzzy version of a file path that's still correct. Embedding models compress semantic content, not identifiers, so they will happily rate "read src/auth/session.ts" and "read src/auth/sessions.ts" as near-duplicates. A human skimming a diff catches that typo. An agent replaying a cached tool result does not.

The failure compounds because agent loops are stateful. A chat cache miss costs you one wrong answer in a conversation a human is reading and can correct. A tool call cache hit that returns the wrong file contents gets fed straight into the next planning step, which generates a new tool call based on bad information, which the agent executes, which changes real files on disk. By the time a human looks at the diff, the corruption is three or four steps removed from the cache hit that caused it, and the actual root cause never shows up in the diff you're staring at.

What actually goes wrong, concretely

• Stale file reads: a cached response for "read config.ts" served after the file was edited two tool calls ago in the same session.
• Wrong-repo bleed: two projects with structurally similar prompts (both have a src/index.ts, both use similar system prompts) produce embeddings close enough to collide.
• Phantom command output: a cached shell command result gets replayed for a command with different flags, so the agent believes a build passed when it never ran with the flags it thinks it used.
• Silent divergence: nothing errors. The agent's internal state and the actual filesystem state quietly disagree, and the disagreement surfaces three or five tool calls later as a confusing, hard-to-reproduce bug.

That last one is the expensive part. A cache bug that throws an error is a five-minute fix. A cache bug that returns a plausible, well-formatted, wrong answer can cost an afternoon of debugging the agent's reasoning before anyone thinks to check the cache layer at all.

The fix isn't a higher similarity threshold

The instinctive fix is to crank the similarity threshold up, 0.97 instead of 0.90, and assume that filters out the dangerous near-misses. It doesn't reliably. Embedding similarity and semantic equivalence aren't the same axis, and the threshold that's safe for chat prose is not the same threshold that's safe for a tool call payload where a single character difference in a path is the whole difference between correct and destructive. Raising the bar buys you fewer false positives at the cost of fewer cache hits overall, and you still can't prove the remaining hits are safe. You're tuning a knob against a failure mode you can't fully observe.

The more reliable fix is structural: don't run tool call traffic through semantic matching at all. Route it through exact caching, where a hit means the request was identical, or skip caching entirely for anything carrying tool calls or tool results and reserve semantic matching for prose the agent generates that never touches the filesystem. Exact caching is boring and that's the point. It has zero ambiguity. If the key doesn't match byte for byte, it's a miss, and a miss is always safe, just slower.

How Probe0 handles this

I built Probe0 as a local proxy that sits in front of whatever coding agent CLIs are already on your machine, Claude Code, Codex, Cursor, all routed through one proxy and one certificate, no per-tool config. It runs entirely on your laptop; there's no Probe0 server collecting your prompts. One of its modules is a local vector-index semantic cache, and it refuses to match any request carrying tool calls. Not a high threshold, a hard rule. If the request has a tool call or tool result in it, semantic matching doesn't run on it, period. Exact caching still applies underneath it, so an identical repeated request, the same read on the same file with the same arguments, still gets served from local disk without hitting the network. You get the caching win where it's safe and none of the risk where it isn't.

That's a real tradeoff, not a free lunch. A hosted gateway with a more permissive semantic cache will show a higher hit rate on paper, because it's willing to match things Probe0 won't. If your traffic really is mostly prose, that gateway's cache will look better in a benchmark. Probe0's bet is that for coding agent traffic specifically, the composition of a session is heavy on tool calls, and a cache that's fast but occasionally wrong on that traffic is worse than a cache that's more conservative and always right. Spend Guard and Recording exist for the same reason, visibility into what actually happened per run, because a cache you can't audit is a cache you eventually stop trusting.

What to check if you already run a semantic cache

• Look at whether your cache keys on the raw request or on a normalized version that strips tool call arguments before embedding.
• Ask your provider or self-hosted setup whether tool calls are excluded from semantic matching, or just subject to the same threshold as everything else.
• If you can't get a straight answer to that question, assume they're not excluded and treat any agent run that used the cache as unverified until you diff the actual file state.
• For anything touching a live filesystem or a git repo, prefer exact-match caching over semantic, even if it means a lower hit rate.

A cache hit that's wrong is more expensive than a cache miss, because a miss costs you latency and a wrong hit costs you a debugging session.

Semantic caching is a genuinely good idea for the traffic it was designed for. The mistake is applying it uniformly to everything an LLM proxy sees, prose and tool calls alike, because they don't share a correctness model. If you're running coding agents through any kind of caching layer, the one question worth asking first is whether tool calls are excluded from fuzzy matching. If the answer is no, that's where your next confusing bug is going to come from.