Probe0 vs Braintrust
Braintrust exists to answer whether your LLM output is good. Probe0 exists to answer where the money went on your own machine and to stop it leaving. Both sit on the request path, and that is roughly where the overlap ends.
A hosted LLM evaluation and observability platform, with an MIT-licensed AI proxy you can run yourself.
| Feature | Probe0 | Braintrust | Edge |
|---|---|---|---|
| Where your prompts go | Nowhere. Everything stays on your Mac; no Probe0 server exists. | Hosted cloud platform; the proxy component is open source. | Probe0 |
| Setup for one developer | One local proxy plus one certificate. Claude Code, Codex and Cursor all route through it after that. | Point a base URL at the gateway, or instrument via SDK. | Probe0 |
| Local model routing | Sends work to a model already running in Ollama or LM Studio, retries weak answers on the cloud. | Built around commercial providers. No native local runtime story. | Probe0 |
| Spend caps before the call | Hard cap per run and per day. Warns, then pauses the run. | Observes cost after inference. No documented pre-request budget. | Probe0 |
| Caching agent traffic | Exact cache reads off local disk without touching the network. Semantic cache refuses to match anything carrying tool calls. | Gateway response caching, encrypted and per-user scoped. | Probe0 |
| Evaluation workflow | None. Probe0 does not score output quality. | Datasets, 25+ scorers, custom scorers, experiment comparison, release gates. | Competitor |
| Which tool spent the money | Local ledger records the calling process, so Cursor spend separates from Codex spend. | Traces grouped by project and span from instrumented apps. | Probe0 |
| Attributing the savings | Every module toggles on its own and reports what it saved. Reads real usage and flags an overpriced plan tier. | Dashboards report cost after the fact. | Probe0 |
| Access and pricing | Private beta, macOS only. Sign in with Google or GitHub. | Free Starter tier, paid Pro, Enterprise quoted. | Even |
Braintrust owns evals, and I will not pretend otherwise: 25+ built-in scorers, custom scorers defined in natural language, side-by-side prompt comparison, and quality gates that block a release when a score regresses. That is a product-team problem. My problem was the bill on my own laptop, and it has a different shape. A coding agent on a Mac makes thousands of calls a day, most of them repeats of something it already asked, several of them to a model far stronger than the edit needs. Probe0 sits in front of all of it as one local proxy with one certificate, and from that moment Claude Code, Codex and Cursor are all covered with nothing configured per tool. Requests go to a model already loaded in Ollama or LM Studio when one will do, weak answers get retried on the cloud, a cheap tier runs before an expensive one, exact cache hits are served off disk without a packet leaving the machine, and identical simultaneous calls collapse into one upstream request.
Braintrust watches inference and reports on it afterwards. Independent reviews of its gateway make this plainly: it routes, it load-balances across providers, it fails over on 429s and 5xx errors, and it does not gate spend on the request path. There are no virtual keys, no per-agent budgets, no hard cap. For a team with a finance function that is a rounding error. For one person watching Claude Code chew through a refactor at two in the morning, it is the entire problem. Probe0 warns at the threshold and then pauses the run at the cap. The semantic cache carries the matching restriction I care most about: it refuses to match anything carrying tool calls, because a wrong hit there corrupts an agent run rather than merely embarrassing it.
The rest is accounting you can act on. The local ledger records model, tokens, real cost, latency and which process made each call, so Cursor spend and Codex spend are separate numbers rather than one monthly total. Every module switches off independently and reports what it saved, so you can turn one off, watch the figure move, and decide whether it earned its place. Probe0 also reads your real usage and tells you when your subscription tier is above what you actually consume. The limits are real and worth stating: macOS only, private beta, one machine, no multi-user gateway, and provider coverage is whatever coding agents actually call rather than a catalogue of a hundred.
The verdict
Pick Probe0 if you are one developer on a Mac and your coding agent bill is the problem: local routing, zero-network cache hits, per-process attribution and a cap that pauses instead of emailing you. Braintrust is the better pick in one situation, and it is a real one: you are shipping an LLM product with other people and need scorers, datasets and regression gates on output quality. That is a different job from cost control on your own machine, and Probe0 is the one that does this job.
Questions
- Is Probe0 a Braintrust alternative?
- Only for the gateway half. Probe0 is a local LLM proxy for AI coding agent traffic with caching and spend caps, and it has no eval, scoring or dataset features at all. If you came to Braintrust for evals, Probe0 does not replace it.
- Can Braintrust cap how much my coding agent spends?
- Not before the request runs. Braintrust records cost and latency after inference, and reviewers consistently point elsewhere when hard budgets, virtual keys or per-key rate limits are the requirement. Probe0 enforces a cap per run and per day, warning first and then pausing.
- Does either one route requests to a local model?
- Probe0 does, natively: it sends work to a model already loaded in Ollama or LM Studio and automatically retries weak answers on a cloud model. Braintrust is oriented around commercial providers, and while a self-deployed proxy could in principle point at an internal endpoint, local LLM routing is not a first-class feature.
- Do I have to configure Claude Code, Codex and Cursor separately?
- No. Probe0 installs one local proxy and one certificate, and every coding agent CLI on the machine routes through it after that. Braintrust generally means changing a base URL or adding an SDK per application you want instrumented.