Diversity is noise when you want the right answer — and the engine when you want new ideas

Part 1 — the wall: cheap tricks don't fix hard answers

The standard advice for an unreliable LLM is "combine your way out": sample it many times and vote, try diverse prompts, add another model, let it check itself. On hard reasoning — multi-hop MuSiQue and graduate MMLU-Pro, strict grading, single-model accuracy in the 0.3–0.6 band — we measured each one, on local models plus a frontier voter.

One mechanism explains all of it: LLM errors are systematic, not random. Aggregation only cancels random, independent noise; LLM errors are biased and shared — across a model's own samples and across different families. A model cannot average, verify, or check its way to an answer it could not produce in the first place.

Part 2 — what works, and the trap

Hand the model the gold facts and multi-hop accuracy jumps +21 points (0.47 → 0.66). But that is an oracle upper bound — perfect retrieval. Real retrieval that returns only some of the needed facts (flat semantic search recovered about 42% of them) scored below full context (0.22 vs 0.47): multi-hop needs the complete chain, and a partial chain breaks the answer. Bad RAG is worse than no RAG. The lever is complete information, not more opinions — and reaching that oracle is itself the hard part.

Part 3 — the flip: the same diversity is the engine for ideas

Here is the turn. The model diversity that bought ≈ 0 for finding the answer pays when you want new answers. On open-ended ideation, three diverse model families cover +14–16% more unique ideas than resampling one model at equal budget (replicated across two model trios); their cross-family idea-overlap is about 2× lower than one model's self-overlap; this survives semantic de-duplication (distinct concepts, not paraphrases); and a judge rated the extra ideas as valid as the base ones (0.65 vs 0.65 — validity only; novelty and usefulness were not separately scored, and this is two trios with no confidence intervals).

Honest about size: the first-order win is fanning out to more generators at all — even a second model from the same family adds most of the gain; a different family is a real but modest edge (about 13% more marginal coverage). On convergent tasks the families converge on the same, often wrong, answer; on divergent tasks they diverge to different, valid ideas.

One rule underneath both, and it is textbook: aggregation only cancels decorrelated error — so diversity is noise when you want agreement and signal when you want exploration. The governing variable isn't the task type, it's the error correlation ρ; the convergent/divergent split is a proxy for correlated-vs-decorrelated error, not a mechanism of its own. This is one object wearing several names — the ensemble ambiguity decomposition ("ensemble error = average error − diversity"; Krogh & Vedelsby 1995; Brown et al. 2005), Guilford's convergent/divergent thinking (1950), wisdom-of-crowds needing independence (Galton 1907; Condorcet), and Ashby's requisite variety (1956). What is ours is only measuring both signs on the same LLM population in one cycle.

What to actually do

• Don't ensemble or self-check for hard answers. It targets random noise you don't have — spend on a stronger model or complete grounding instead.
• Invest in retrieval quality, not answer-voting — and remember that partial retrieval can hurt.
• Do fan out for ideation — more generators, ideally across diverse families, when you want to cover an idea space.

Caveats and prior art: two benchmarks, specific models, strict-substring and judge grading; "hard" means the headroom subset (single-model acc 0.3–0.6). The convergent "≈0" is scope-conditional — on easier items, or with methods like Mixture-of-Agents / LLM-Blender, ensembling can still gain; the governing mechanism is decorrelated-vs-correlated error, not "answers vs ideas". The correlations here (ρ ≈ 0.62–0.72) are from our own runs, not a cited paper. On the divergent side the first-order win is fan-out (one sample → three roughly doubles coverage); the +14–16% is the diversity increment at equal budget, from two trios with no confidence intervals, and idea novelty beyond validity was not separately scored — so treat it as suggestive, not established. Every number comes from a runnable experiment — the probe (scripts + result files) is at research/probes/diversity_flip.

FAQ

Does ensembling multiple LLMs improve accuracy? On this hard, matched-compute subset (single-model acc 0.3–0.6): ≈0. Models from different families err on the same items (cross-family ρ ≈ 0.62–0.72, our runs), self-consistency saturates at ~1.6 effective independent samples, and “family independence” carries no accuracy premium — aggregation has little independent error to cancel. On easier items, or with Mixture-of-Agents / LLM-Blender, ensembling can still gain.

When does model diversity actually help? For generating ideas — but mostly it's fan-out. Adding generators at all is the first-order win (one sample → three roughly doubles unique-idea coverage); making them diverse families adds +14–16% more unique ideas on top, at equal budget (two trios, no CIs). The extra ideas were rated equally valid — validity only, novelty not scored.

Why don’t more samples fix a wrong LLM answer? Because the errors are systematic, not random: a model is largely blind to its own failure mode, so resampling concentrates on the same wrong answer. That is the “bootstrap wall” — self-consistency saturates around 1.6 effective independent samples.

What is the practical rule? Ensemble diverse model families for ideation and divergent work; never for convergent answers. For hard multi-hop accuracy, fix retrieval instead — supplying the gold facts jumped accuracy +21 points (0.47→0.66), while partial retrieval (~42% of facts) scored below full context.

Related research

• The same hidden law in four AI-memory mechanisms
• Does long context kill RAG? We measured it
• Multi-hop recall on LoCoMo: put the model in the retrieval loop