More samples, worse answers: when scaling best-of-N backfires

A cheap way to make an AI answer better: sample N candidate answers and keep the one a verifier (or reward model) scores highest — "best-of-N". Folklore says more samples is more better. We measured when that's false, with a minimal, fully-reproducible model — and it turns out this is a textbook effect (reward-model overoptimization / Goodhart) that our model just makes legible.

Two kinds of "imperfect verifier"

Generate N candidates, each correct with probability g. The verifier scores each and we keep the argmax. "Imperfect" splits into two very different cases.

A — error independent of correctness (its score tracks correctness with alignment α plus noise unrelated to whether the answer is right). Best-of-N is safe: accuracy climbs toward 1.0 for any α > 0 (monotonically across our sweep). What matters is that the noise is independent of correctness, not that it is small — even unbounded Gaussian noise averages out under the argmax, because the fixed signal gap outweighs the shrinking spread of the extreme draws (which grows only like √log N), so the signal still decides. Noise magnitude is not the danger.

B — error correlated with wrongness. In the sharp version, a small fraction h of candidates are "hacks": wrong, but scored higher than any correct answer can be. Now accuracy peaks, then collapses as N grows:

The chance of drawing at least one hack is 1 − (1 − h)^N → 1, so the large-N accuracy is just P(no hack) = (1 − h)^N — an arithmetic identity, not a discovered law. Its scale is 1/h, so the optimal N shrinks as the exploitable tail grows; at h = 8%, best-of-128 scores zero. "≈ 1/h" is a restatement of that geometric decay.

The "scored above every correct answer" tail is a sharp caricature. The real regime is continuous: whenever a wrong-but-plausible answer can merely out-score a correct one — a verifier whose error is correlated with wrongness (length, confident tone, sycophancy), not a discrete hack — best-of-N collapses just the same, with no tail required. Our probe sweeps this continuous knob; it is the ordinary imperfect reward model of Gao 2022 and the shared-vs-idiosyncratic error axis of Eisenstein 2023.

The mechanism (it's textbook)

Under selection, verifier error independent of correctness is averaged out, but error correlated with wrongness — bias, or in the limit an exploitable tail — is amplified: best-of-N drives the sample toward whatever the verifier over-rewards. The risk is not verifier imperfection or noise magnitude; it is a high-proxy / low-truth direction the argmax can climb. None of this is new — it is Goodhart / Campbell's law (1979), and empirically reward-model overoptimization: Gao, Schulman & Hilton (2022) fit the best-of-N gold-reward curve as a downward parabola in √KL, so the optimal N is already finite; Khalaf et al. (2025) prove the rise-then-fall is inevitable for a broad class of best-of-N mechanisms and give an algorithm (HedgeTune) that recovers the optimal N. Our contribution is only the minimal runnable model that isolates the two idealized extremes.

If you spend test-time compute

Before scaling best-of-N or reward-model reranking, do not just look at the verifier's average accuracy — the argmax climbs the tail, not the mean. You usually cannot measure the exploitable-tail rate directly (if you could, you would filter it), so the deployable moves are indirect: watch the gap between the proxy score and a held-out gold signal you did not select against (a second reward model, hidden tests, spot-checks), and cap N where that held-out curve stops rising; add a KL / length-and-style debias so the winner's-curse tail cannot win; use reward-model ensembles (they damp idiosyncratic but not shared error — Coste 2023, Eisenstein 2023); and above all prefer verifiable / executable rewards (unit tests, checkers), where the exploitable tail is bounded by construction. The tail is non-stationary: optimizing against a verifier moves the policy toward its blind spots, so re-measure the safe N.

The falsifierTwo pre-committed predictions: under a verifier whose error is independent of correctness, accuracy must be non-decreasing in N (it held even for unbounded Gaussian noise — noise magnitude is not the danger); and under error correlated with wrongness, accuracy must peak then fall. Both held across the sweep — had best-of-N's own argmax manufactured a collapse from pure independent noise, the "exploitability, not noise" claim would be wrong. It did not: the extreme draws grow the same for correct and wrong candidates, so the fixed signal gap still wins.

FAQ

Does best-of-N sampling always improve answers? No. When the verifier's error is correlated with wrongness (a wrong answer that scores high), more samples make it worse — the argmax climbs toward the high-score / low-truth region. In our model accuracy collapses toward zero by N=128 once such a tail exists (0.28 at a 1% hack rate, 0.00 at 8%).

Why do more samples backfire? Because best-of-N takes the maximum proxy score. If the proxy only partly tracks truth (alignment α), a larger N is more likely to surface a sample that games the proxy without being correct — exactly the tail the argmax seeks.

Is there an optimal N? Yes, and it shrinks as the exploitable tail grows. With a well-aligned verifier more N helps; with a weak proxy the optimal N is small — past it, exploitation dominates.

How do I use best-of-N safely? Improve verifier alignment before scaling N; cap N at the point where added proxy score still tracks true reward; and treat a verifier that rarely disagrees with the policy as a red flag, not a green light.

Related research

• Diversity is noise for answers, signal for ideas
• The same hidden law in four AI-memory mechanisms
• Does long context kill RAG? We measured it