How Much of Chatbot Arena Is Style? The Votes Are Biased; the Order Mostly Isn't

The short answer. Chatbot Arena (now LMArena) Elo, built from millions of human pairwise votes, is the leaderboard developers cite to pick a model — taken to rank LLMs by genuine quality. On the real public votes we found two things that pull in opposite directions. At the vote level, style is a real bias: a judge that sees only the answer's style (length, markdown) and nothing about which model wrote it predicts the human winner 61.5% of the time — and, crucially, the longer answer keeps winning ~62% even between the same two models (quality held fixed), so this is a genuine length preference, not just "better models write longer." **At the leaderboard level, style mostly is not what's ranked:** the style-only model reproduces ~74% of the raw order (Spearman 0.74), but that is a correlational ceiling — style and quality co-move across models — and the decisive test, LMSYS's style-controlled Elo, reorders the board only modestly. So: bias the individual votes, mostly not the order. Use style controls; the rank is mostly real skill.

The claim under test. Arena Elo measures quality, so a higher Arena rank means a better model — and its votes are a clean quality signal.

What we measured — three tests, no model identity, no content

Data: lmarena-ai/arena-human-preference-140k — 28,084 decided battles (ties dropped). Features: style only — assistant length (tokens), markdown header / list / bold counts — as side-A-minus-B differences. A logistic classifier; held-out split. Runnable: research/probes/arena_style_only.py.

[A] Style predicts individual votes.

A judge that understands nothing about correctness beats chance by ~11 points — and the markdown features add essentially nothing beyond raw length. This is a real but modest per-vote effect. The same length signal we found faking the GPT-4 judge on MT-Bench.

[B] The length bias isn't just that better models are wordier — the within-pair control. The obvious objection (and the one that reversed our sibling post on LLM judges): maybe longer answers win because better models write longer, so "style" is just a proxy for skill. So we held quality fixed — among battles between the same two models — and asked whether the longer answer still wins:

Holding the model pair constant barely dents it: the longer answer still wins ~62%. So the length preference in human votes is not just an artifact of stronger models being wordier — it holds at fixed average model quality. (Honest caveat: this fixes which two models are competing, not which answer is better on a given prompt — on any single prompt the longer answer is often the more complete/correct one, so ~62% is a strong association, not a clean isolation of length from per-response quality.)

[C] Style tracks the leaderboard order — but that's a ceiling, not a decomposition. Rank the models by the style-only classifier's win-propensity and correlate with their real win-rate ranking:

A judge that never sees which model wrote an answer reproduces ~3/4 of the leaderboard order from stylistic form. It is tempting to read this as "74% of the ranking is style" — but that would repeat the exact error we had to reverse on the LLM-judge length post. Style and quality co-move across models: better models genuinely write longer, more complete, better-formatted answers, so a style-only rank tracks the order because style is a valid proxy for the quality the votes reward. Reproducing the order via a proxy does not partition it into style-vs-skill.

The decisive order-level test: style-controlled Elo

The only thing that separates confound from proxy at the leaderboard level is to remove style and see if the order holds. LMSYS did exactly this (their August-2024 style-control analysis regresses length + markdown out of the Elo). The result is a modest reorder, not a scramble: the very top (GPT-4o, Claude, Gemini) stays near the top; Claude 3.5 Sonnet even rises (6→4); the models that fall are mostly the ones that leaned on formatting — GPT-4o-mini (6→11), Grok-2-mini (6→18). LMSYS calls style "a strong effect" but their own controlled order largely survives — and they explicitly warn of "positive correlation between length and substantive quality," i.e. they refuse to call it a pure confound. So the honest verdict is two-sided: style genuinely biases individual votes (test B), but it does not drive most of the leaderboard order — the order is mostly skill, with style a partial confound that moves specific models.

What this does and does not say

• It does not say Arena is useless or "ranks by style, not skill." The order is mostly quality; the style-only reproduction is a correlational ceiling inflated by the style–quality entanglement.
• It does say the individual votes carry a real, model-independent length bias (~62% even within a fixed pair), so a single Arena vote is not a clean quality signal — and a model that wins by writing longer, heavily-formatted answers can climb some on style alone (the mini-models under style control).
• Practically: read the style-controlled leaderboard, not the raw one, and weight Arena against task-specific evals — especially if you need terse output, where the raw board's length preference is exactly wrong for you.

The style/verbosity bias itself is well known — Zheng et al. (2023) flagged it and LMSYS shipped the style-control adjustment in 2024. Feuer et al.'s "Style Outweighs Substance" (2024) showed LLM judges over-weight style (not the human Arena — different setting). What is ours here is narrow: the runnable within-pair control (the length bias survives holding model quality fixed) and the no-identity order reproduction (a style-only rank tracks ~74% of the raw order) — a quantification, framed against the decisive style-controlled result, not a discovery that style matters.

The falsifierWe predicted style-controlled Elo would leave "a large share of the order style-driven." It does not — LMSYS's controlled order largely survives, which is why this post corrects that read. What would still move the verdict: a full recomputation of the style-controlled ranking against our style-only rank (does ρ drop sharply against the controlled board? our prediction now: yes, it should), and whether the within-pair length effect holds on harder-prompt subsets where length is less likely to track quality.

FAQ

Does this mean Chatbot Arena is broken? No. Individual votes carry a real length bias (~62% even between fixed models), but the leaderboard order is mostly quality — under LMSYS's style control the top largely survives. Use the style-controlled board and task-specific evals.

Is style just a proxy for quality, then? At the leaderboard level, largely yes — better models write better-formatted answers, so a style-only rank tracks the order. At the single-vote level, no: the within-pair control shows the longer answer wins ~62% even with model quality held fixed, which is a genuine bias.

Is 61.5% impressive? It's ~11 points over the 50.8% baseline — a real but modest per-vote effect, and essentially all of it is length (markdown adds nothing). It is enough to track the raw order (ρ=0.74), but that's because style co-moves with quality.

Isn't verbosity bias already known? Yes (Zheng 2023; LMSYS shipped style control in 2024). The new, runnable pieces are the within-pair control (bias survives fixed model quality) and the no-identity order reproduction — quantifications, not the discovery.

Is this just a model you trained? A trivial logistic classifier on real public Arena votes, no model identity, no content — the weakest possible "judge." Runnable: research/probes/arena_style_only.py.