We built a dose-response 'grounding meter' for AI - ordered by how sure the model already is
Two failures look identical from the outside: a model that ignores a correct document and one that swallows a wrong or poisoned one. We built a dose-response 'grounding meter' that reads how much a model follows graded in-context evidence (a fixed-wording ladder, '1 source' to '6 sources'), bias-cancelled. The follow-curve is ordered by the model's prior strength - fictional facts saturate almost immediately, near-axioms resist - and this replicates on frontier models (glm-5.2, deepseek-v4-pro). We have dropped an earlier, near-tautological 'confidence can't see it' headline and corrected one cross-model claim; the honest version is below.
The problem. A confidence score cannot distinguish two failures that look identical from the outside: a model that ignores a correct document, and a model that confidently swallows a wrong or poisoned one. Both just look like “a confident answer.” So we built an instrument that measures the difference directly - a dose-response grounding meter.
What it measures. For a two-option question we supply an external context that asserts one option at graded strength - a fixed-wording ladder from “1 source reports X” to “6 sources report X” - and flip which option the sources push. We read how much the model goes with the context at each dose, averaged over both option orders and both push-directions so position and option bias cancel by construction. We read the answer probability either from token logprobs (on an open model) or, where logprobs are not exposed, by sampling the model a few times and counting - so unlike a logprob-only method the meter also runs black-box on closed or frontier models. The output is a grounding curve: a dose-response of deference to evidence.
The core result (open model qwen2.5-7B, a 14-question bank spanning fictional to near-axiom facts): the grounding curve is continuous and ordered by how strong the model's prior is. Fictional or weak-prior facts half-saturate almost immediately; near-axiom facts - “water boils at 100 C”, “H2O is water” - resist even six agreeing sources. As a half-saturation dose (an EC50 for evidence):
| Prior strength of the fact | Half-saturation dose (lower = grounds faster) |
|---|---|
| Fictional / no prior | ~0.08 |
| Strong / near-axiom | ~0.26 |
That is the meter's real signal: how much evidence it takes to move the answer, as a function of how sure the model already is. Mechanically it is an Asch conformity curve fitted with A.V. Hill's 1910 dose-response equation - the “drug” is evidence, the “receptor” is the model's prior. (Runnable analysis, reproduced from cache.)
The ordering replicates on frontier models. We re-ran the max-dose test on two strong cloud models (eight facts, six sources all asserting the FALSE option, five samples, both orders). This is a coarse max-dose check, not the full curve - but the ordering by prior strength holds on both. Mean fraction of answers that follow the false context, by stratum:
| Prior strength | glm-5.2 follows false @6 sources | deepseek-v4-pro |
|---|---|---|
| Fictional | 1.00 | 1.00 |
| Common fact | 0.80 | 0.42 |
| Near-axiom | 0.55 | 0.25 |
Confidence is blind to this - but here is the honest version. The model's own token confidence barely tracked whether it grounded or resisted (correlations 0.15-0.36). An earlier version of this post went further and claimed the grounding signal “predicts confident-wrongness” at r = -0.93 where confidence cannot. That number was near-tautological: it correlated “follows the context at max dose” with “fails to resist a false context” - two names for the same quantity - so we have dropped it. The defensible point is narrower and older (Lichtenstein & Fischhoff, 1977; calibration): token confidence does not encode whether an answer is riding on the context, and the dose-response does.
A correction about strong models. The earlier version said a frontier model (GLM-5.2) “resists plausible-but-wrong sources on facts it knows.” That rested on a single-sample, four-item slice our own notes flagged “not commensurable.” The proper five-sample re-run above shows the opposite: glm-5.2 follows a six-source false context on “water boils at 100 C” (0.70), the largest planet (0.90), the capital of Japan (1.00) and the tallest mountain (1.00); only the most canonical items (H2O) resist. deepseek-v4-pro resists somewhat more. So grounding is a property of (model x how the context is framed), ordered by prior strength within each model - but no frontier model here broadly “resists what it knows.” (Runnable probe.)
What is new, and what is not
The phenomenon is established: whether a model adopts in-context evidence over its parametric memory depends on evidence strength and the model's prior confidence (Xie et al., ICLR 2024); models override a correct prior more than 60% of the time given wrong retrieved content, scaling inversely with confidence (ClashEval, Wu et al., NeurIPS 2024); and “caving to graded pressure” is the sycophancy literature. Context attribution is already a scored quantity (ContextCite). What we add is the instrument - a fixed-wording, bias-cancelled dose-response curve with a per-fact half-saturation dose - and the clean half-saturation-dose-vs-prior-strength scaling. The idea is older still: Asch's conformity-vs-majority-size curve (1951-56) and Hill's dose-response (1910).
The falsifierIf the follow-curve did not order by prior strength, the meter would be measuring nothing. It did, on an open model and on two frontier models. The open question is whether it measures grounding or mere suggestibility to one prompt wording - which is why the honest next step is a wording ablation, not a model leaderboard.
Honest scope. One open model in full plus a frontier re-measurement; a 14-item bank; one fixed wording. We are not claiming a leaderboard, and we owe the wording ablation that separates grounding from sycophancy. All numbers here come from runnable probes (qwen dose-response, frontier re-run), not field data.
FAQ
What is a grounding meter for AI? A dose-response measure of how much a model's answer bends to graded external evidence versus its own prior - an EC50/Asch curve for evidence. It runs black-box, from token logprobs or from sampling, so it works on closed and frontier models too.
Does an AI's confidence tell you when it is wrong? Not reliably. Token confidence barely tracked whether the model grounded or resisted (0.15-0.36) and does not encode whether an answer is riding on the context - a known calibration failure (Lichtenstein & Fischhoff, 1977). The dose-response reads what confidence misses.
How do you measure grounding without model internals? Supply an external context that asserts one option at graded strength (a ladder from “1 source” up) and watch how the answer moves, reading the answer probability from logprobs or by sampling. The dose-response curve separates fictional or weak-prior facts (which saturate almost immediately) from stubborn priors that resist.
Do strong models resist a wrong document on facts they know? Mostly no. In a five-sample re-run, glm-5.2 followed a six-source false context on “water boils at 100 C”, the largest planet, the capital of Japan and the tallest mountain; only the most canonical facts (H2O) resisted, and deepseek-v4-pro resisted somewhat more. Grounding is (model x framing), ordered by prior strength.
Is the phenomenon new? No - it is knowledge-conflict / sycophancy behaviour (Xie et al. 2024; ClashEval 2024). The contribution is the instrument: a bias-cancelled dose-response curve with a per-fact half-saturation dose.