When can an agent trust its own confidence to abstain?

The short version. A common assumption in agent design is that a model's own confidence tells it when to act and when to hold back — answer if confident, abstain if not. We tested the cheap version of that signal — verbalized, single-shot confidence — on a contamination-free task. On arithmetic, a weak or mid model's verbalized confidence barely separates right from wrong (AUROC ~0.50), while one frontier model's tracks correctness well (AUROC ~0.90, though on only n=48, 95% CI [0.81, 0.99]). Two honest caveats blunt the clean story: the weak-model 0.50 is partly a saturation artifact — those models say ~100% to almost everything, and a sampling signal recovers discrimination even for them (~0.97 on arithmetic) — and on a harder factual benchmark (SimpleQA, below) the gradient flattens: a mid model matches the frontier (both 0.74). So the honest reading is narrower than "only the frontier can trust itself": verbalized single-shot confidence is a weak abstention signal that improves with capability but is task-dependent, and this re-measures a known result — calibration and self-knowledge scale with capability (Kadavath 2022, Xiong 2023). The probe and raw per-item data are public, so every number is re-checkable.

The setup

We want one question: does higher confidence actually mean a higher chance of being right? The clean way to measure it is AUROC of confidence against correctness — the probability that a correct answer was given more confidence than an incorrect one. 0.5 means confidence carries no signal (you cannot threshold it to abstain); 1.0 means it perfectly separates right from wrong.

The task is multi-step integer arithmetic of escalating difficulty — generated from random numbers, so it is contamination-free (no model has memorized the answers) and graded exactly. Each item: the model returns an answer and a confidence (0-100, its own probability of being correct).

One methodology point that matters for trusting the result: AUROC is computed only on items where the model emitted a real, parsed confidence — parse-failures are excluded, not defaulted. (An earlier version of this analysis defaulted unparsed answers to 0.5-confidence-and-wrong, which can inflate AUROC by separating parsed from unparsed instead of right from wrong. We caught that and fixed it; the numbers below are the clean version.)

The result on arithmetic: discrimination scales with capability

The weak and mid models emit maxed-out confidence (~1.0) on almost everything, including answers that are wrong (the weak model returned exactly 1.0 on all 71 clean items) — so their verbalized confidence cannot separate right from wrong (AUROC ≈ 0.5) and they are massively overconfident (+0.7 to +0.8). One honest caveat here: with the confidence scale saturated, that 0.5 is partly a degenerate readout, not proof the model has no internal uncertainty — a sampling signal recovers discrimination for the very same models (see the multi-sample note below). The frontier model is near-perfectly calibrated (overconfidence +0.02) and its confidence tracks correctness (AUROC 0.90): in this run it assigned about 2% confidence to most of its wrong answers and about 77% to its right ones. Read that 0.90 with its width: it is on n = 48 (25 right / 23 wrong), 95% CI ≈ [0.81, 0.99] — clearly above the coin-flip line, but a point estimate, not a precise constant, and resting on one clean frontier model. A second frontier model, glm-5.2, also discriminates (AUROC 0.73) but emitted no parseable confidence on 34% of items (an exclusion that can bias its score upward), so we treat Claude — which gave a real confidence every time — as the clean anchor.

What it means in practice (risk–coverage)

The same result without the jargon: if a model answers only the questions it is most confident about, does its accuracy go up? For a weak model, no — selecting its most-confident half is no better than answering everything. For a frontier model, accuracy climbs sharply, and you can safely answer a large share at high accuracy:

This is the operational version of the AUROC table above: AUROC summarizes whether confidence ranks correctness; risk–coverage shows what that buys you — how much you can answer, at what accuracy, if you gate on the model's own confidence. Below the frontier, gating buys you nothing; at the frontier it buys a lot.

A harder test: SimpleQA (n=150)

Arithmetic is contamination-free but computable — a model can re-derive the answer, which may exaggerate how cleanly confidence tracks correctness. So we ran the same probe on SimpleQA (OpenAI's deliberately-hard short-answer factual benchmark), where models err on most items and the numbers are robust (the small-model AUROC no longer rests on a handful of correct cases). Verbalized confidence, plus a multi-sample signal (sample N=5, use answer-agreement), at n=150:

This refines the arithmetic headline in two honest ways. First, verbalized discrimination is not a clean weak→mid→frontier gradient: the 7B model is still at chance (0.47), but the 30B model matches the frontier on raw AUROC (both 0.74). Discrimination switches on above ~7B and is itself task-dependent (the same 30B scored only 0.54 on arithmetic). Second, the robust capability gradient is operational: gating to the most-confident quarter lifts accuracy 5% → 21% → 62% across the tiers, and only the frontier can answer any share at ≥90% accuracy. The mid model discriminates yet still cannot gate to useful accuracy, because it does not know enough — discrimination is not the same as usable abstention. That is the case for an external retrieval / grounding layer even on a model whose confidence ranks correctly. (One honesty note on the anchors: the SimpleQA run covers weak / mid / glm-5.2 — we did not run Claude on SimpleQA, so the 0.90 sits on arithmetic only, and no single model anchors both the top of the arithmetic gradient and the harder factual test.)

A note on the multi-sample signal: on arithmetic, sampling-consistency recovered discrimination strongly (AUROC ~0.97 even on small models). On SimpleQA it does not (0.57–0.71) — re-sampling helps when a model can re-derive an answer, but not when it simply does not know the fact. So multi-sample confidence is task-dependent, not a universal small-model fix.

Why it matters: the cost of acting on bad confidence

Put a price on it. Reward a correct answer +1, penalize a wrong one −c, score 0 for abstaining. The textbook-rational rule is to answer when confidence exceeds c/(1+c). Applied to a weak model's verbalized confidence — which here carries no rank information about which answers to keep — that rule loses heavily: under any real penalty the only utility-positive policy is to abstain on everything. Applied to the frontier model's arithmetic confidence, answering is utility-positive and lands close to the best achievable.

The practical reading is not simply "small models need an external gate, frontier models don't." Two things are true at once. First, the cheaper a self-confidence signal is (verbalized, single-shot), the more capability it takes to be worth trusting — so a weak or local agent gets little from its own number and more from an external check (corroboration across independent sources, a separate verifier, or retrieval against ground truth). Second — and this is the correction — an external gate is insurance, and insurance is priced by the cost of a wrong action, not by the model's capability. Frontier agents are deployed on harder, higher-stakes, more-irreversible tasks, where even good discrimination leaves a costly residual error, and where self-confidence is measured in-distribution but degrades on the hard / out-of-distribution tail (Kadavath's own P(IK) falls under distribution shift). So the gate's value tracks reversibility and stakes first, model capability only second. That is the design principle behind our memory core (inspeximus) and benchmark (RAMR): the corroboration gate earns its keep where a wrong write is costly and self-confidence is not enough.

Auditable

The probe is one file and the result JSONs include the raw per-item rows (expression, answer, gold, confidence, correct), so you can re-run it on your own models or re-check every number above: research/probes/overconfidence_tax (MIT).

Honest prior art

This sits on a well-worked literature, and the honest framing is re-measurement, not discovery. Calibration and self-knowledge improving with scale is Kadavath et al. 2022 (models "mostly know what they know"; P(True) calibration scales). The exact quantity we report — AUROC of verbalized confidence for failure prediction, rising with capability — is Xiong et al. 2023 (ICLR 2024): single-shot verbalized AUROC ≈0.51 (GPT-3) → ≈0.63 (GPT-4), with the higher ≈0.73+ numbers coming from sampling / self-consistency, not the single-shot verbal number. That is the honest ceiling on our own signal — our arithmetic 0.90 is a computable-task best case; on factual recall the single-shot verbalized ceiling is far more modest. Tian et al. 2023 showed RLHF models' verbalized confidences beat their token probabilities. The distinction we lean on — calibration (do stated probabilities match reality) vs discrimination (AUROC — can you rank right above wrong) — is textbook signal-detection theory (Green & Swets 1966), invariant to any monotone recalibration; the abstain-below-a-cost-threshold rule is the reject option (Chow 1970); and that optimization power (deeper nets, RLHF) re-breaks calibration is Guo et al. 2017 and the GPT-4 report (pre-training calibrated, RLHF degrades it). What is ours is narrow: a runnable, contamination-free re-measurement with public per-item data, and the operational framing of when the cheap self-confidence signal is worth trusting versus paying for an external gate.

Honest limits

• Two task types (arithmetic + SimpleQA factual recall), three models. The gradient is directional, not a scaling law. Arithmetic may exaggerate confidence-maxing (models treat computation as deterministic and say ~100%); SimpleQA (above) was the harder cross-check and refined the claim — raw discrimination can appear at mid-size, and the gradient that robustly holds is operational (only the frontier gates to useful accuracy).
• Scope: this is verbalized, single-shot confidence — the cheap signal an agent gate usually reads (ask the model its confidence, once). Multi-sample / sampled-consistency confidence (e.g. a Monte-Carlo temperature sweep: sample N times, use answer-agreement as confidence) is a separate, more expensive signal that is more predictive on computable tasks (arithmetic AUROC ~0.97 even on small models) — but task-dependent: on SimpleQA factual recall it only reached 0.57–0.71. So the claim is scoped to single-shot self-reports, and multi-sample is not a universal fix.
• Falsifier (pre-registered). The claim dies if a weak model's AUROC were ≫0.5 (its confidence is usable after all), or if the frontier model's AUROC collapsed to ~0.5 in its error regime (confidence useless even at the frontier → the claim becomes "universal", not a gradient). Neither happened: weak/mid ≈0.5 (with 51 and 66 clean errors), frontier 0.90 (23 clean errors, zero parse-failures).

FAQ

Can an LLM agent trust its own confidence to decide when to abstain? Not reliably on the cheap signal. Its verbalized, single-shot confidence barely separates right from wrong below the frontier (AUROC ~0.5 — partly because it saturates at ~100%), and while discrimination improves with capability it is task-dependent: on factual SimpleQA a mid model matches the frontier (both 0.74). Even the frontier's single-shot verbalized ceiling on real factual QA is modest (~0.63 in the prior literature). A sampling-consistency signal, or an external gate keyed to the cost of a wrong action, is the more reliable lever.

What is AUROC of confidence here? The probability that a correct answer received higher confidence than an incorrect one. 0.5 = confidence is useless for telling right from wrong; 1.0 = perfect separation. It is the right metric for selective prediction / abstention, distinct from calibration (whether "80% confident" is right 80% of the time).

Are LLMs overconfident? The weak and mid models were massively so (confidence +0.72 and +0.84 above their accuracy); the frontier model was near-perfectly calibrated (+0.02). But overconfidence (a calibration gap) matters less than discrimination (AUROC) for the abstain decision — a model can be overconfident yet still rank its right answers above its wrong ones.

What should a weak/local agent do instead? Not rely on self-confidence. Use an external gate — corroboration across independent sources, a separate verifier, or retrieval against ground truth — which is exactly where it pays off most.

Related research

• Memory that won't let an uncorroborated fact become permanent — measured
• Why RAG serves stale facts: the supersession blind spot, reproduced
• Does long context kill RAG? We measured it