How do you detect a weak LLM answer well enough to auto-retry it

You detect a weak answer by scoring the output on cheap, structural signals the moment it comes back, not by asking the model how confident it is. Self-reported confidence from an LLM is close to useless for this: models are consistently overconfident on exactly the answers that are wrong, because the token-probability distribution that produces the text doesn't know anything about whether the text is correct. What works instead is a small stack of checks that look at the shape and behavior of the output, not its self-assessment.

This matters specifically for coding agents because the retry decision has to happen inline, in milliseconds, without a human reading the diff. A routing layer sitting between the agent and the model has maybe four kinds of evidence available to it, and none of them require asking the model to grade itself.

The four signals that actually predict a weak answer

• Truncation and malformed output — the response hit a token limit mid-structure, a JSON block doesn't parse, a code fence never closes, a function call has missing required fields. This is the single most reliable signal because it's binary and cheap to check.
• Refusal and hedge language — pattern-matching against phrases like "I cannot determine", "it's not clear from the context", "you may need to check", repeated apologetic hedging, or a response that restates the question without answering it.
• Degenerate output — empty diffs, a response that's just the input echoed back, repetition loops (the same clause 40 times), or a tool call the agent didn't ask for and can't use.
• Downstream failure — the agent tries to apply the patch and it doesn't apply, the tool call references a file or function that doesn't exist in the repo, the test the agent runs immediately after fails. This one is the strongest signal of all because it's not a heuristic, it's ground truth, just delayed by one round trip.

Notice what's missing from that list: log-probabilities and perplexity. In theory, low per-token confidence should correlate with wrong answers, and there's real research behind that (semantic entropy, self-consistency sampling). In practice, for a local proxy sitting between a coding agent and a small local model, you often don't get log-probs at all, streaming responses make them awkward to aggregate, and the correlation is noisy enough on short structured outputs like diffs and tool calls that it's not worth the complexity for a v1. Structural checks catch the failure modes that actually show up in coding-agent traffic, which skews heavily toward tool calls, patches, and short factual lookups rather than long free-text reasoning where perplexity-based methods do better.

Why this is a routing problem, not a prompting problem

The obvious fix for a weak answer is to prompt around it: add "think step by step", raise the temperature down to zero, add a self-critique pass. Those help, but they don't solve the actual problem, which is that some questions are genuinely too hard for the model you sent them to. A 7B local model asked to trace a race condition across three files is going to produce a confident, well-formatted, wrong answer no matter how you phrase the prompt. The fix there isn't prompt engineering, it's escalation: detect the weak answer, then resend the same request to a stronger model.

That reframes the problem as a routing decision made after the fact rather than a model-selection decision made in advance. You don't need to predict difficulty upfront, which is genuinely hard and gets it wrong constantly on real prompts. You send the cheap or local model first, cheaply, and only pay the strong-model cost on the subset of requests that actually fail the weak-answer checks. On typical coding-agent traffic — lots of small, well-scoped tool calls and short edits — that subset is a minority, which is the entire economic case for doing this instead of just always calling the frontier model.

The retry loop, concretely

A working implementation looks like this: request comes in, gets sent to the fast/cheap/local tier first, response comes back, gets run through the structural checks above before it's handed back to the agent. If it fails a check, the same request — same messages, same tool schema — goes to the escalation tier, usually a frontier cloud model, and that response is what the agent actually sees. The agent itself doesn't know a retry happened unless you choose to surface it. The critical design constraint is that this has to be cheap and fast enough that the check itself doesn't become the bottleneck; regex and JSON-parse-attempt checks run in single-digit milliseconds, which is negligible next to a network round trip to any model, local or cloud.

The harder edge case is the downstream-failure signal — the patch that doesn't apply, the test that fails. By the time you know that, the agent has already acted on the bad answer, so "retry" there means something closer to "flag this exchange and let the agent's own retry logic decide whether to re-ask," not a silent swap. Most routing layers, ours included, treat structural and refusal detection as the auto-retry path and treat downstream failure as a signal you log and expose, not one you act on unilaterally, because acting on it means guessing at intent the agent hasn't stated yet.

Where Probe0 fits

I built the Local Routing module in Probe0 around exactly this loop, because it's the piece that makes sending coding-agent traffic to a local model (Ollama, LM Studio) viable instead of a novelty. Without auto-retry, routing to local-first means occasionally watching Claude Code or Codex silently accept a bad tool call from a small model that had no business handling that request. With it, the weak answer gets caught by the structural checks and re-sent to your cloud model before the agent ever sees it, and the run gets logged either way. Probe0 is a local proxy — it runs on your Mac, there's no Probe0 server, and every coding CLI on the machine routes through it after a one-time proxy-plus-certificate setup, no per-tool config. Local Routing is one of several switchable modules; you can run it alongside Model Tiering (cheap-model-first on the cloud side), Exact and Semantic Cache, and Spend Guard, and each one reports what it actually saved rather than an estimate.

It's honest to say this doesn't replace real evals. Structural weak-answer detection catches malformed and refused output well; it does not catch a confidently wrong answer that's syntactically perfect, which is the harder and more interesting problem, and one that generally needs the downstream test-failure signal or a human in the loop to catch at all. If you're building evaluation infrastructure rather than a retry heuristic, that's a different and heavier tool than what a routing proxy should try to be.

The retry decision should never depend on the model telling you it's unsure — check the shape of the output, not its opinion of itself.

What to build first if you're rolling your own

• Start with parse/truncation checks — they're nearly free and catch the loudest failures.
• Add a small refusal-phrase list tuned to your own traffic; generic lists overfire on legitimate hedged answers.
• Wire in downstream failure as a logged signal before you wire it in as an auto-retry trigger — watch it for a week before trusting it.
• Skip log-prob-based confidence scoring until the first three stop being enough; it adds real complexity for a marginal gain on short structured outputs.