Probe0 vs llama.cpp
llama.cpp is the engine that runs the model. Probe0 is a local proxy that sits in front of every coding agent CLI on your Mac and decides what each request costs, where it goes, and whether it needed to leave the machine at all. Most people who compare these end up running both.
An open-source C/C++ inference engine (MIT) that runs GGUF models locally, with llama-server exposing an OpenAI-compatible HTTP API.
| Feature | Probe0 | llama.cpp | Edge |
|---|---|---|---|
| What it is | Local proxy for coding agent traffic | Local inference engine plus HTTP server | Even |
| Local inference speed | None of its own; calls out to Ollama or LM Studio | GGUF, quantization, batching, speculative decoding | Competitor |
| Cost and token accounting | Per-request ledger: model, tokens, real dollars, latency, process | Prometheus counters for tokens and timings, no dollars | Probe0 |
| Spend limits | Hard cap per run and per day; warns, then pauses | No budget concept; docs suggest fronting it with a gateway | Probe0 |
| Local-to-cloud routing | Tries local first, retries weak answers on the cloud | Routes by explicit model name only, no fallback chains | Probe0 |
| Setup for one developer | One proxy plus a certificate; Claude Code, Codex, Cursor all covered | Point each tool at a base URL yourself | Probe0 |
| Response reuse across agents and providers | Exact and semantic cache on local disk, cloud calls included | KV prefix reuse inside one server, per loaded model | Probe0 |
| Identical calls fired at once | Coalesced into one upstream request | Parallel slots run each one separately | Probe0 |
| Which process spent the money | Per-process attribution in the local ledger | Metrics are server-wide, not per caller | Probe0 |
llama.cpp wins on inference and it is not close: hand-tuned Metal, CUDA and Vulkan kernels, quantization choice, continuous batching, parallel slots, speculative decoding with byte-identical output, one static binary with no Python, MIT licence. What it has no notion of is dollars, because inference on your own hardware is free at point of use. That is a reasonable design decision for an engine and a serious problem for someone running Claude Code, Codex and Cursor against paid APIs all day. Probe0 is the layer that handles the paid half. It records every call with model, token counts, real cost, latency and which process made it, so a bill stops being a single number and becomes a list of the agents that produced it. Spend Guard takes a hard cap per run or per day, warns, then actually pauses instead of emailing you afterwards.
The routing is where a proxy earns its place above an engine. llama-server dispatches on the explicit model name in the request; there is no fallback and no local-versus-cloud arbitration, and router mode has no warm pool yet, so alternating between two models reloads on every request. Probe0's Local Routing sends work to a model already loaded in Ollama or LM Studio and retries on the cloud only when the local answer comes back weak, which is the decision llama.cpp deliberately leaves to you. Model Tiering tries the cheap model first and escalates when it has to. Exact Cache lives on local disk and a hit never touches the network at all. Semantic Cache uses a local vector index with a strict similarity floor and refuses to match anything carrying tool calls, because one wrong hit corrupts a whole agent run. Request Coalescing collapses simultaneous identical calls into one upstream request. Every module switches on and off on its own and reports what it saved, so you can judge each one instead of trusting an aggregate.
The axis I would pick on is who you are. If you are serving models to a fleet, squeezing tokens per second out of a GPU, or running on Linux or Windows, llama.cpp is the correct tool and Probe0 does not apply. If you are one developer on a Mac with three coding agents and a bill you cannot explain, a faster engine does not help; the missing piece is the layer that knows a request happened, what it cost, whether it could have been answered locally, and whether it should have been allowed at all. Setup is one local proxy and one certificate, installed once, with no per-tool config, and there is no Probe0 server anywhere, so the ledger and both caches stay on your machine. Because it sees real usage, it will also tell you when you are paying for a plan tier above what you actually consume. Honest limits: macOS only, private beta, not a team gateway. Running both is the normal configuration. llama.cpp serves the local model, Probe0 decides when to use it.
The verdict
Pick llama.cpp if the job is raw local inference speed on your own hardware. For everything else a developer running coding agents on a Mac needs, pick Probe0: spend caps that pause, a per-request ledger with per-process attribution, caching that covers cloud calls too, and local-first routing with cloud fallback. I run Probe0 in front and let llama.cpp, via Ollama, take the local half of the traffic.
Questions
- Is Probe0 a llama.cpp alternative?
- Not really, and I would not sell it as one. llama.cpp is the inference engine; Probe0 is an LLM proxy that sits above whatever engine you run. If you want faster local inference, use llama.cpp. If you want spend caps, caching across agents and a cost ledger, put Probe0 in front of it.
- Can llama.cpp limit how much my AI coding agent spends?
- No. llama-server has no budgets, no quotas and no cost accounting, since local inference has no per-token price. The standard answer in the project's docs is to front it with a proxy or gateway, which is exactly the job Probe0 does for a single machine.
- Does Probe0 cache better than llama.cpp?
- Differently, and worse at the tensor level. llama.cpp reuses KV prefixes inside the attention cache, which a proxy cannot touch. Probe0 caches whole responses on local disk, exact and semantic, which works across tools and across providers including cloud models llama.cpp never sees.
- Do I have to choose between them for local LLM routing?
- No. llama.cpp routes only on the explicit model name in the request, with no local-versus-cloud fallback. Probe0 makes that decision for you: it tries a model already running in Ollama or LM Studio, and retries on the cloud when the local answer comes back weak.