Prompt caching vs response caching: what actually saves money

Two different things get called caching on an AI bill, and they do not overlap. One is applied by the provider on the input side of a request that still runs, still generates output, and still costs you money. The other stops the request before it leaves your machine. People turn on the second one, watch their invoice barely move, and conclude caching does not work.

The short answer: prompt caching makes a repeated prefix cheaper on a call that still reaches the model. Response caching returns a stored answer and skips the call entirely. Prompt caching is a discount. Response caching is an avoided purchase. For coding agents specifically, prompt caching is already doing most of the work, and your main job is to stop breaking it. Response caching pays off on a narrow slice of traffic that genuinely repeats, and the size of that slice is the whole question.

Prompt caching is a prefix discount, applied by the provider

Prompt caching works on prefix match. The request is rendered in a fixed order (tools, then system, then messages) and the provider hashes the bytes up to each cache breakpoint. If those bytes match a previous request, the matched span is billed at a heavily reduced rate. Anything after the first differing byte is billed at full price. That is the entire mechanic, and every practical rule about caching falls out of it.

On Anthropic's API the economics are explicit. A cache read costs roughly a tenth of the base input price. A cache write costs more than an uncached request: 1.25x for the default five-minute TTL, 2x for the one-hour TTL. With the five-minute TTL you break even on the second request. With the one-hour TTL you need at least three. You verify hits by reading the usage fields on the response: cache_read_input_tokens is what you saved on, cache_creation_input_tokens is what you paid a premium to write, and input_tokens is only the uncached remainder, not the total prompt.

There are also floors that fail silently. Anthropic caps you at four breakpoints per request, and there is a minimum cacheable prefix that varies by model, from 512 tokens on the newest Opus models up to 4096 on some earlier ones. Below the floor, nothing caches and you get no error, just a cache_creation count of zero. OpenAI's version is automatic rather than opt-in, with no request changes required and discounted pricing on the cached portion, but the same prefix-match logic governs whether you hit.

Response caching skips the call

Response caching keys on the request and returns a previously stored response. No tokens are billed, because no request is made. Latency drops to whatever a disk read or vector lookup costs. There are two common forms. Exact caching hashes the request and returns a stored answer only on a byte-identical match. Semantic caching embeds the request, searches a vector index, and returns a stored answer when similarity clears a threshold.

The honest problem is hit rate. In an agent run, every turn carries the growing conversation, so byte-identical requests are rarer than the intuition suggests. The places where response caching genuinely fires are specific: a retried call after a timeout, several processes issuing the same call at once, a repeated one-shot classification or summarization step, and a developer re-running the same command against an unchanged file.

Where each one sits, and what each one costs you

• Prompt caching lives on the provider side. You cannot implement it locally, only preserve or destroy it by how you build the request.
• Response caching lives wherever you put it: in your app, in a gateway, or in a local proxy. It is entirely under your control.
• Prompt caching reduces the price of tokens you still send. Output tokens are unaffected.
• Response caching removes both input and output cost for the calls it hits, and returns them in milliseconds.
• Prompt caching has a real downside risk: a cache write costs more than a plain request, so churning breakpoints on traffic that never repeats makes things worse.
• Response caching has a correctness risk that prompt caching does not. A wrong hit returns a plausible answer to a question nobody asked.

The expensive mistake is fighting your own prefix cache

The most common way to lose money is to install something that rewrites the prompt to make it smaller. Prompt compression, history summarization, dedup middleware, anything that touches bytes near the front of the request. Every one of those changes the prefix, and a changed prefix means the whole span after it is billed at full rate again, plus a cache write premium if you re-mark a breakpoint.

A middleware that trims a few percent off your token count and invalidates the prefix cache has raised the price of that request.

The quieter versions of the same mistake are worth grepping for. A current timestamp interpolated into the system prompt. A request ID near the top of the message. JSON serialized without sorted keys, so the byte order drifts between runs. A tool list built per user, which renders at position zero and means nothing caches across users. Switching models mid-conversation, since caches are model-scoped. And in long agentic turns, the breakpoint lookback window is finite (twenty content blocks on Anthropic), so a turn that emits thirty tool_use and tool_result blocks can silently miss the previous entry.

What coding agent traffic actually looks like

A coding agent request is a long stable head and a short volatile tail. System prompt, tool schemas, project instructions, and file context sit at the front and barely change across a session. The user's new instruction and the latest tool results sit at the end. That shape is close to ideal for prompt caching and close to hostile for exact response caching, which is why the two feel so different in practice.

• Keep stable content first and volatile content last. Ordering matters more than breakpoint placement.
• Do not rebuild the system prompt per request. Inject changing context later in the message array instead.
• Serialize tools deterministically and keep the set fixed for the life of a conversation.
• When a side task forks off (summarization, a subagent), copy the parent's system prompt, tools, and model verbatim so the fork reuses the parent's prefix.
• Check cache_read_input_tokens across two consecutive identical-prefix calls before believing any of it works.

How Probe0 treats the distinction

Probe0 is a local proxy for coding agent CLIs. It runs entirely on your machine, and every agent on the machine (Claude Code, Codex, Cursor) routes through one proxy plus a certificate you install once. It does not implement prompt caching, because it cannot: that is the provider's side of the wire. What it does is refuse to interfere with it. There is no prompt compression module, because on this traffic shape compression usually costs more than it saves.

What it does own is the response side. Exact Cache stores hits on local disk and never touches the network. Semantic Cache uses a local vector index behind a strict similarity floor, and it refuses to match anything carrying tool calls, because a wrong hit in the middle of a tool loop corrupts a whole run rather than producing one bad paragraph. Request Coalescing collapses simultaneous identical calls into a single upstream request, which is where a surprising share of the duplicate traffic actually lives. Every module is individually switchable and reports what it saved, so you can turn one off and see the number move.

The part that matters most for this topic is Recording. It keeps a full request ledger with model, tokens, real cost, latency, and which process made each call. That is what lets you tell prompt-cache reads apart from full-price input tokens, per agent, and catch the day a config change quietly stopped your prefix from matching.

The limits are real: macOS only, private beta, single developer machine, and provider coverage is what coding agents actually call rather than a hundred-provider catalog. If you need a hosted team gateway with shared cache state across a fleet, a server-side gateway is the right tool and Probe0 is not it. If you are one developer trying to find out where a four-figure monthly agent bill is going, the local ledger is the fastest path I know to an answer.

The one-line summary I would keep: protect the prefix, and cache responses only where repetition is real. Everything else is a rounding error compared to accidentally invalidating a cache you were already getting for free.