Probe0 vs Weights & Biases Weave
Weave measures what your LLM code did and stores it centrally for a team. Probe0 sits in the request path on one Mac and changes what happens: caching, local LLM routing, cheaper model tiers, and a hard spend cap.
An LLM and agent observability and evaluation toolkit from Weights & Biases: you instrument your code with the Python or TypeScript SDK, and traces, token counts, cost estimates, and evaluation runs are stored on a W&B backend.
| Feature | Probe0 | Weights & Biases Weave | Edge |
|---|---|---|---|
| Setup for one developer's agent CLIs | One local proxy plus one certificate, once. Claude Code, Codex, and Cursor all route through it with no per-tool config. | weave.init() plus @weave.op decorators in code you control | Probe0 |
| Where prompts end up | Ledger stays on the machine. Nothing is uploaded. | Inputs and outputs go to a W&B backend unless you run self-managed | Probe0 |
| Response caching | Exact cache on local disk. A hit never touches the network at all. | No provider-response cache. The optional client cache stores Weave API reads. | Probe0 |
| Semantic cache safety | Local vector index, strict similarity floor, refuses to match anything carrying tool calls | No semantic cache | Probe0 |
| Local model routing | Routes work to a model already loaded in Ollama or LM Studio, auto-retries weak answers on the cloud | Traces Ollama and LM Studio calls, but does not route them | Probe0 |
| Spend limits | Hard cap per run and per day: warns, then pauses | Cost estimates and rollups. No cap on your provider bill. | Probe0 |
| Cost attribution | Model, tokens, real cost, latency, and which process made the call | Attributed to the ops you decorated, in projects you set up | Probe0 |
| Knowing your plan tier is wrong | Reads real usage and flags a tier above what you use | No view of your provider subscription | Probe0 |
| Offline evaluation | None. Probe0 has no view on whether a prompt change improved quality. | Versioned datasets, scorers, LLM judges, leaderboards, side-by-side runs | Competitor |
Weave's evaluation layer is the real product and I would not try to talk anyone out of it: versioned datasets, scorers, LLM judges, leaderboards, and drill-down into the individual examples a prompt change broke. Everything else about it assumes a shape I do not have. It expects you to instrument your own code, which works when you wrote the agent and does not work when the agent is Claude Code or Codex or Cursor, because there is nothing of yours to decorate. Probe0 installs below that line. One local proxy and one certificate, and every coding agent CLI on the Mac routes through it, including the ones that ship as a binary.
The other gap is that Weave observes and does not intervene. It has no provider response cache, no router, no fallback, and no enforced budget. It reports cost by multiplying token counts against known model prices, which tells you what you already spent, and then you spend it again on the next run. And outside the licensed Kubernetes and ClickHouse self-managed install, your prompts and completions leave the machine and land on a vendor backend, with the ingestion meter priced by volume, so the more of your code you send the more the observability costs. Probe0 runs entirely on your Mac. There is no Probe0 server to send anything to.
What that buys you concretely: Local Routing sends work to a model already running in Ollama or LM Studio and retries weak answers on the cloud. Model Tiering tries the cheap model first. The exact cache lives on local disk and a hit never reaches the network, and the semantic cache holds a strict similarity floor and refuses to match anything carrying tool calls, because a wrong hit there corrupts a whole agent run rather than returning a slightly stale answer. Request Coalescing collapses simultaneous identical calls into one upstream. Spend Guard warns at a threshold and then pauses. Recording keeps a full ledger of model, tokens, real cost, latency and originating process on local disk, which is how Probe0 can tell you your usage sits below the plan tier you are paying for. Every module switches on and off independently and each one reports what it saved, so you can check the claim instead of taking it.
The verdict
If you have a quality question that only evaluation runs can answer, and sending trace data to a hosted backend is acceptable, use Weave. That is its home ground. For everything else in a solo developer's day on macOS, use Probe0: it takes one certificate to cover every agent CLI on the machine, it cuts the bill instead of reporting it, the cap actually pauses, and the ledger never leaves your disk.
Questions
- Is Probe0 a Weights & Biases Weave alternative?
- Only for the cost and visibility half of the job. Probe0 is an LLM proxy that caches responses, does local LLM routing, and enforces a spend cap for AI coding agents on one Mac. It has no evaluation framework, no team workspace, and no production monitoring, so if that is why you use Weave, Probe0 will not replace it.
- Does W&B Weave reduce LLM costs or just report them?
- It reports them. Weave derives per-call cost from token usage and known model prices and rolls it up in the UI, and add_cost() lets you register prices for custom models. There is no response cache for provider calls, no router, and no budget enforcement, so the number it shows you is a record rather than a lever.
- Can I use Weave without sending data to Weights & Biases?
- Not on the default path. The Weave client is open source but the backend is not, and traces ship over HTTPS to a W&B server. Keeping data in-house means the self-managed install, which needs a Weave-enabled license, Kubernetes, the Altinity ClickHouse operator, and an S3 bucket. Probe0 keeps its ledger on local disk with no account required for the proxy itself.
- Do I need to change my code to use Probe0 or Weave?
- Weave needs code changes: weave.init(project) plus @weave.op on the functions you want traced, in Python or TypeScript. Probe0 needs none. You install one local proxy and a certificate once, and every coding agent CLI on the machine routes through it, which is the only option when the agent is a binary you did not write.