Tools that collapse duplicate simultaneous LLM calls, ranked
One axis only: when two or more identical requests are in flight at the same moment, does the tool hold the duplicates and serve them all from a single upstream response? That is a narrow question, and it is not the same as caching. A normal cache can only help after the first response has finished writing. Parallel subagents do not wait politely for that; four of them fan out inside the same second with the same system prompt and the same file context, and a read-after-write cache misses all four. Everything below is ranked by how it behaves in that specific window, on one developer machine, with coding-agent traffic. Tools that are excellent at other things and irrelevant to this one are ranked low here and say so.
- 1SProbe0Request Coalescing is a named, individually switchable module, built for exactly this case: identical simultaneous calls collapse into one upstream request and the rest attach to that response. It sits as a local proxy in front of Claude Code, Codex and Cursor at once, so the fan-out from parallel subagents crosses one dedup point regardless of which CLI spawned it, and the ledger shows which process made each call so you can see the collapse happen. macOS only, private beta, single developer.
- 2AvLLMThe strongest genuine in-flight sharing in this list, at a different layer. Continuous batching plus automatic prefix caching means concurrent requests sharing a prompt prefix reuse the same KV cache blocks rather than recomputing them. It does not return one response to N callers, and it only applies to models you are serving yourself, but for overlapping simultaneous work it is real deduplication of the expensive part.
- 3ALiteLLMThe most configurable self-hosted option here. Open-source proxy with a pluggable cache layer (in-memory, Redis, S3) and per-key budgets, and the source is in front of you if you want to change the dedup behaviour. Cache hits are still keyed on completed responses, so a burst of simultaneous identical calls tends to race past it before the first one lands.
- 4BCloudflare AI GatewayEdge-hosted gateway with response caching, rate limiting and logging in front of multiple providers. Cloudflare knows more about collapsing duplicate origin fetches than almost anyone, but the AI Gateway is a hosted hop: your prompts leave the machine, and the caching you configure is response caching rather than a documented in-flight merge for LLM calls.
- 5BPortkeyMature gateway with simple and semantic caching, fallbacks, retries and guardrails, available hosted or self-hosted. The caching layer is the closest thing it offers to this axis, and it is good at it once a response exists. The design target is production application traffic with many users, not four subagents on one laptop firing the same prompt at once.
- 6BBifrostGo-based open-source LLM gateway from Maxim, built explicitly for low overhead at high concurrency, with caching, fallbacks and an OpenAI-compatible surface. Throughput under concurrency is its whole pitch, which is adjacent to this axis without being the same thing: it handles many simultaneous calls fast rather than merging identical ones.
- 7BTrueFoundryEnterprise AI gateway you can run inside your own VPC, with routing, rate limits, budgets and caching across providers. Solid architecture, and self-hosting means the traffic stays inside your boundary. It is sized for platform teams governing many services, so the setup cost is disproportionate if the problem is one machine spawning duplicate calls.
- 8CKong AI GatewayKong Gateway with AI plugins for provider routing, prompt guarding, rate limiting and semantic caching. Kong genuinely wins on operational maturity, plugin ecosystem and anything involving many upstreams. Nothing in the AI plugin set is aimed at merging duplicate concurrent completions, and running Kong beside a laptop coding agent is not the intended shape.
- 9COllamaLocal model runtime, and the reason a lot of duplicate-call pain is survivable: if the answer is computed on your own GPU, a duplicate costs time instead of money. It queues and parallelises requests up to a configured limit rather than collapsing identical ones, so N duplicates are still N generations.
- 10COpenRouterA routing marketplace with very wide model coverage and a single billing relationship, which is a genuine advantage over anything local. It is a remote endpoint, so it sees your duplicate calls as N separate paid requests and bills accordingly. Nothing about the product is trying to solve concurrency dedup on the client side.
- 11DLangfuseOpen-source LLM observability with tracing, evals and prompt management, self-hostable. Excellent at showing you that four identical calls happened and what each one cost. It sits beside the request path as a recorder rather than in front of it as a gate, so it cannot prevent the duplicates it documents.
- 12DccusageReads Claude Code local JSONL logs and reports token usage and cost per session and per day, with no network dependency and no setup. Honest, fast and free, and completely outside the request path. It is on this list only to mark the boundary: reporting after the fact is the opposite end of the axis from collapsing calls before they leave.
The method was to separate three things that get called caching and behave completely differently under concurrency. First, response caching: a request finishes, the response is stored, a later identical request gets it back. Second, prefix or KV reuse at the serving layer, which shares computation between overlapping requests without returning the same response twice. Third, in-flight coalescing: duplicate requests arriving while the first is still open are held and attached to that one result. Almost every gateway in this space does the first. A few inference servers do the second. Very few tools name the third as a feature, which is why the ranking looks unusual compared to a general LLM gateway list.
I put Probe0 first because coalescing is a module I built on purpose rather than a side effect of a cache, and because of where it sits. It runs on the machine as a local proxy with one certificate installed once, and every agent CLI on that machine goes through it. That matters more than it sounds: the duplicate calls I was trying to kill were not coming from one tool, they were coming from Claude Code subagents and a Codex run and an editor completion all hitting the same model with near-identical context inside the same few seconds. A dedup point that only covers one CLI misses most of that. The Recording ledger attributes every call to the process that made it, so the effect is checkable rather than asserted, and coalescing can be switched off on its own if you want to see the difference.
The tools below it are better than Probe0 at things this page does not measure. vLLM is doing harder engineering than I am; continuous batching with prefix caching is the real answer to concurrent overlapping work, and if you serve your own models at any scale it belongs in your stack regardless of what this list says. LiteLLM has provider coverage and a configurable cache layer I cannot match, and the code is open. Portkey, Kong and TrueFoundry are built for teams with many services, audit requirements and people who need governance rather than a laptop that spends too much. Cloudflare AI Gateway takes about five minutes to put in front of an app. Langfuse and ccusage are genuinely useful and cost nothing.
The honest scope statement: Probe0 is macOS only, in private beta, works for a single developer rather than a team, and covers the providers coding agents actually call instead of a hundred-provider catalog. If your duplicate-call problem is a production service with many users behind it, most of the entries above are a better fit than mine. If it is your own machine at two in the afternoon with six subagents running, the ranking on this axis is what it is.