We Tried to Debunk LLM-as-Judge as a Length Trick. Our Own Control Refuted It.
A length-only null recovers half of GPT-4's above-chance agreement with humans on MT-Bench (68% vs 86%), which looks like a verbosity confound. But our pre-registered control — length-matched pairs — refuted it: with length neutralized, GPT-4 still agrees ~80% while the null drops to chance. The agreement is largely semantic. A debunk that debunked itself.
The short answer. The foundational LLM-as-judge result (Zheng et al., 2023) is that GPT-4 agrees with human preference judgments about 85% of the time (ties removed) — a hair above the 81% two humans agree — so a strong model looks like a valid stand-in for human quality evaluation. On the same released data we built a judge with zero understanding — it just picks the longer answer — and it already agrees with humans 68% of the time, seeming to recover about half of the judge's above-chance margin. That looks damning. So we ran the control our own post pre-registered as the falsifier — comparing only length-matched pairs — and it refuted us: with length neutralized, GPT-4 still agrees with humans ~80% while the length rule collapses to a coin flip. The agreement survives length-matching, so it is largely semantic, not a length trick. This is a debunk that debunked itself.
The claim we set out to test. ~85% GPT-4–human agreement ≈ human–human agreement ⇒ the LLM judge is measuring quality, not a shared shortcut. Our worry: humans prefer longer answers, and an LLM judge trained on human preferences inherits the same bias, so the two could agree ~85% while both just reward length.
Step 1 — the length-only null (this looked damning)
We used the original lmsys/mt_bench_human_judgments data — 3,355 human and 2,400 GPT-4 pairwise votes — and a null judge that picks the response with more characters. Ties excluded.
| Judge | Agreement with… | Score | n |
|---|---|---|---|
| GPT-4 (the famous judge) | human majority | 86.3% (reproduces Zheng's ~85%) | 798 |
| Length-only null (pick longer) | human votes | 68.1% (word-count: 66.4%) | 2,562 |
| Length-only null (pick longer) | GPT-4's own votes | 73.5% | 1,792 |
| chance | — | 50% | — |
Read naively, a rule with no understanding at all reaches 68% — apparently ~50% of the judge's entire above-chance margin recovered by counting characters, with the GPT-4 judge itself agreeing with "pick the longer one" nearly three times in four. That is the number our first draft led with. It is also the wrong way to read it.
Step 2 — the control that flipped our verdict
A length-only null agreeing 68% proves nothing on its own, because length correlates with genuine quality: on MT-Bench a longer answer is frequently the more complete, more correct one. So the null could be recovering real signal that both the judge and humans correctly track — not a shared bias. The one experiment that separates "shared confound" from "length is a valid proxy" is to look at length-matched pairs, where the length cue carries no information. If the agreement were a length trick, it should collapse toward the length floor there. We ran it:
| Length gap between the two answers | GPT-4 vs human | Length-only null vs human |
|---|---|---|
| matched (<5%) | 87.8% (n=41) | 60.2% |
| matched (<10%) | 79.7% (n=74) | 53.0% (≈ chance) |
| moderate (10–30%) | 75.0% (n=124) | 54.3% |
| imbalanced (>30%) | 89.5% (n=600) | 73.2% |
(The <5% row is nested inside <10%; the non-overlapping bins — <10%, 10–30%, >30% — sum to the n=798 above.)
On length-matched pairs the length rule falls to a coin flip (53%) — as it must, since the lengths are equal — yet GPT-4 still agrees with humans ~80%. The agreement does not collapse to the length floor; it survives with the length cue removed. By our own pre-registered falsifier — "if GPT-4–human agreement stays near 80% on length-matched pairs while the length-only null drops to chance, the judge's agreement is genuinely semantic and this verdict is wrong" — the "it's just length" reading is falsified.
Why the length-only null misled us
The 68% is a correlational upper bound, not a causal decomposition. Because length co-varies with quality on this data, a length-only rule "recovers half the agreement" by riding a valid proxy, not by exposing a fooled judge. This is the textbook shared-method-variance trap (Campbell & Fiske, 1959): when two measures share a nuisance dimension, their convergence looks inflated — but you cannot attribute the shared part to bias without a control that removes it. We ran the control, and it attributes most of the agreement to semantics, not length. The honest residue is a mild "length-easiness": agreement is highest on imbalanced pairs (89.5%) and dips to ~80% when lengths match, so some of the headline rides on longer-usually-being-better — but the core is real judging.
The verbosity bias is real — this just isn't where it wins
None of this says LLM judges are unbiased. Verbosity/length bias is well-documented and worth controlling: Zheng et al. flag it in the original paper (and show a "repetitive list" attack most judges fail); Singhal et al. (2023) find a length-only reward reproduces most of RLHF's downstream gains; Dubois et al. (2024) built length-controlled AlpacaEval, which raised its correlation with Chatbot Arena from 0.94 → 0.98 and cut length-gameability ~21% → ~6%; Wang et al. (2023) show position bias large enough to flip rankings. The lesson stands — use length controls, position-swaps, and per-criterion rubrics (recent multi-judge audits find verbosity bias can shrink substantially under a fixed rubric). What our control shows is narrower and, for once, reassuring: on MT-Bench the specific ~85% human-agreement number is mostly earned, not a length artifact.
What this does and does not say. It does not say LLM judges have no length bias — they do, and it should be controlled. It does correct our own initial over-read: a length-only null recovering half the agreement is not evidence that half the agreement is fake, because the length-matched control shows the agreement survives when length is neutralized. The number to trust is the controlled one (~80% on matched pairs), not the raw null (68%).
The falsifierThe pre-registered test was: length-match the pairs; if agreement collapses toward the length floor, the confound story holds; if it stays near 80% while the null drops to chance, the story fails. It stayed near 80% (0.797 on matched-<10% pairs) while the null hit chance (0.530). What would flip it back: a larger length-matched replication (our matched-set n is only 74, 95% CI ≈ ±9pp) that shows agreement actually collapsing — or a design that also removes position and self-preference confounds, which this control does not.
FAQ
So is LLM-as-judge a length trick? No — that was our initial hypothesis and our own control refuted it. A length-only rule recovers half the raw agreement, but on length-matched pairs (where length is uninformative) GPT-4 still agrees with humans ~80%. The agreement is largely semantic.
Then why does the length-only null hit 68%? Because length correlates with quality on MT-Bench — longer answers are often genuinely better — so a "pick the longer one" rule rides a valid proxy. Recovering agreement ≠ exposing a confound.
Do LLM judges have verbosity bias at all? Yes, well-documented (Zheng, Singhal, Dubois, Wang). It should be controlled with length normalization, position-swaps, and rubrics. Our point is only that on MT-Bench the ~85% headline is mostly earned, not that judges are unbiased.
Did you reproduce the original number? Yes — GPT-4 vs human majority came out 86.3% (strict majority, ties dropped), matching Zheng et al.'s ~85%.
Is this just a simulation? No — real released human and GPT-4 votes, a trivial length-only null, and a length-stratified control. Every number is re-runnable: research/probes/llm_judge_length_null.py.