Probe0 vs Arize
Arize watches your LLM traffic from the side and tells you what happened to it. Probe0 sits in the middle of that traffic on your own machine and changes what happens, by serving a cached answer, sending the call to a local model, or pausing the run when spend crosses a cap.
An LLM observability and evaluation company: Arize Phoenix, a self-hostable source-available tracing and eval platform built on OpenTelemetry, and Arize AX, the commercial hosted version with enterprise governance and classic ML monitoring.
| Feature | Probe0 | Arize | Edge |
|---|---|---|---|
| Position relative to the request | In the path. Every agent call goes through it. | Beside the path. A span sink you ship traces to. | Even |
| Setup for one developer's coding agents | One local proxy plus one certificate. Claude Code, Codex and Cursor all covered, no per-tool config. | Instrument each app with an SDK or an OTel collector. | Probe0 |
| Caching | Exact cache on local disk plus a semantic cache with a strict similarity floor. Hits never touch the network. | None. Reads cache token counts, cannot create a hit. | Probe0 |
| Tool calls in cached traffic | Semantic cache refuses to match anything carrying tool calls. | No cache, so no such risk and no such saving. | Probe0 |
| Routing and fallback | Local Ollama or LM Studio first, weak answers retried on the cloud. Cheap model tier before the expensive one. | Not a routing layer. Arize docs pair it with a gateway for that. | Probe0 |
| Spend control | Hard cap per run and per day. Warns, then pauses the run. | Cost observability with monitors and alerts. No blocking. | Probe0 |
| Output quality evaluation | None. Records cost, tokens, latency and which process made the call. | LLM-as-a-judge evaluators, datasets, experiments, prompt playground. | Competitor |
| Cost figures | Measured at the wire from real responses. | Computed from configured rates, and pricing config has to exist before ingestion. | Probe0 |
| Attributing spend to a process | Local ledger names the process behind each call. | Attribution by project or span, as instrumented. | Probe0 |
Arize is genuinely strong at one thing Probe0 does not attempt: judging whether an answer was any good, with prebuilt hallucination and relevance evaluators, custom Python evals, versioned datasets you can promote production traces into, and experiments that replay the same inputs against a changed prompt and diff the results. What that machinery cannot do is take a dollar off the bill, and for a developer running coding agents on a Mac the bill is the problem. Probe0 sits in the request path on that machine: Exact Cache and Semantic Cache answer repeat calls without a network round trip, Local Routing hands work to a model already loaded in Ollama or LM Studio and retries weak answers on the cloud, Model Tiering tries the cheap model before the expensive one, Request Coalescing collapses simultaneous identical calls into one upstream, and Spend Guard pauses a run instead of paging you about it. There is no Probe0 server anywhere, so nothing about your prompts leaves the laptop except the provider call itself.
The structural difference is that Arize never touches the request. It reads what already happened. That means it cannot return a cached answer, cannot fail over to a cheaper model, cannot collapse two identical simultaneous calls into one, and cannot stop a run that is burning money. Its cost numbers are inferred by matching the model name against a pricing config, and that config has to be in place before ingestion because cost is not backfilled. It also requires you to instrument every app you want to see. That is a fine trade for a service you own the source of, and a bad one for a coding agent CLI you did not write. Probe0 captures Claude Code, Codex and Cursor because they all speak HTTPS through the same local proxy, with no code change to any of them, and the cost it reports is what the provider actually billed for the response that came back.
The details are where I think Probe0 earns the install. The semantic cache refuses to match anything carrying tool calls, because a wrong hit there corrupts a whole agent run and a few saved cents are not worth that. The ledger records model, tokens, real cost, latency and the process that made the call, so a runaway Cursor loop is identifiable rather than a line item. Every module switches on and off independently and reports what it alone saved, which is how you find out a feature is not paying for itself. Because Probe0 knows your real usage it can tell you when you are on a plan tier above what you consume. Sign-in is Google or GitHub with no password. The limits are real too: macOS only, private beta, no team deployment, and provider coverage is whatever coding agents call rather than a catalogue of a hundred backends.
The verdict
If you are shipping a multi-service LLM product and need evals, datasets and regression testing on output quality, use Arize. That is the one case where it is the better pick. For a developer whose Claude Code and Codex bills keep climbing, install Probe0: it is a local proxy that caches, routes to a model already running on your Mac, and pauses the run at a hard cap, which is the difference between watching the number and lowering it. The two coexist fine, and Arize's own docs pair Phoenix with a gateway for caching and routing. Probe0 is that gateway for a laptop.
Questions
- Is Probe0 an Arize alternative?
- Only for cost, not for tracing quality. If you came to Arize for LLM cost control and hit the limits of alerting, Probe0 replaces that part by enforcing caps and caching in the request path. If you came for evals, distributed traces or team governance, Probe0 does not cover any of it.
- Can Arize cache LLM responses or block a request that costs too much?
- No. Phoenix and AX are passive span sinks that sit outside the request path, so they can report cost and fire a monitor on an expensive trace but cannot serve a cache hit or refuse a call. Arize's own integration docs point at gateways for caching and routing.
- Does Arize work with local models like Ollama or LM Studio?
- Yes, for tracing and the prompt playground: point the OpenAI base URL at your local endpoint and the OpenAI instrumentor picks it up. It observes local model calls rather than deciding to make them, which is the difference from Probe0's local LLM routing.
- Do I need to change my AI coding agent setup to use Probe0?
- No per-tool config. You install one local proxy and a certificate once, and every coding agent CLI on the machine routes through it, including Claude Code, Codex and Cursor. It is macOS only and in private beta right now.