The “55% Faster” AI Coding Claim Is an Operating-Point Trap

The short answer. "AI coding assistants make developers ~55% faster" is the most-cited productivity number in software. It's real — and non-representative. It comes from a vendor study on a single greenfield task; the only independent randomized trial of experienced developers in mature codebases found the opposite: 19% slower. These two results are not in conflict. They are two points on one curve — and reporting either as "the" effect is the error.

The claim. A widely-repeated reading of a GitHub/Microsoft study: AI coding assistants deliver a large, universal speedup (~55%).

What the evidence actually says (verified against primary sources).

The headline 55.8% is a preprint on the easiest possible case. The peer-reviewed evidence (+26%) is for typical/junior devs on scoped tasks. The one randomized test on experts in big, familiar codebases found a slowdown — and those developers still believed they were ~20% faster. In the one RCT that measured both, self-reported speedup diverged sharply from measured output (−19% real vs +20% felt).

The contradiction dissolves into a sign-flip

The +26% and the −19% look like a fight. They aren't — they're two operating points of a single mechanism. Model a task's time as write vs review: doing it without AI costs your self-write time; doing it with AI costs a fixed "read the draft + review/rework it" overhead instead. Experts write faster (less to gain) and AI adds a fixed read overhead, so the net flips negative at high context — while novices, who'd be slow writing it themselves, gain.

We ran the smallest version of that model. Net speedup vs developer context k (0 = novice/unfamiliar, 1 = expert/own mature repo):

• A robust junior-gain / expert-loss sign-flip with a crossover around k ≈ 0.6.
• It flips even with no "expert review tax" — purely because experts write faster and AI adds a fixed read/prompt overhead. (Adding a context-rising review tax only steepens it.)
• Robust in 79% of plausible parameter sets (we drew the review-tax from 0 upward, so we did not assume it).

So the same tool helps and hurts depending on where on the curve you measure. This is the operating-point trap: a single number is meaningless when the effect changes sign across the operating range. (The same shape recurs across our Crucible — a celebrated number that is really a property of one operating point of the measurement.)

What this does and does not say

• It does not say AI coding is useless — it genuinely helps juniors and greenfield work (+26%, peer-reviewed).
• Revealed preference cuts the other way: developers broadly adopt and pay for these tools, so real value clearly exists somewhere on the curve. The point is that it is not the universal +55%, and the labor-relevant unit is team throughput, not one senior's task-time.
• It does say the universal "~55% faster" claim fails: the speedup flips sign with developer context, and the canonical number is a vendor preprint on one toy task.
• The model is illustrative — it reconciles the two RCTs' directions (they use different metrics: task-count vs time), not a new measurement. Which driver dominates the expert loss — "less to gain" vs a review-reconciliation tax — is underdetermined.
• The one thing that is robustly measured: the perception–reality gap (−19% real, +20% felt). Any org sizing AI ROI from developer surveys is measuring belief, not output.

Why it matters (and where the value actually is). The binding constraint at the hard end isn't model capability — it's the context the model lacks that the expert holds in their head. That's an argument for treating memory/context quality as the lever: AI helps most exactly where it can supply missing context, and hurts where it can't. The high-value work is the high-context work — the opposite of where the benchmarks point.

The falsifierA large, pre-registered, peer-reviewed RCT on experienced developers in mature repos, measuring time-to-merged plus downstream defect/maintenance cost over 6–12 months. If experts show a robust positive effect there, the sign-flip framing is wrong. None exists yet. METR's own late-2025 re-run (n=57) shrank the slowdown toward zero, with confidence intervals now crossing zero — but its authors call this only weak evidence (selection effects) and expect newer models to help more. So −19% is one early-2025 snapshot, not a durable law.

FAQ

Is the "55% faster" number fake? No — it's a real measured result, but on a single greenfield JavaScript task in a vendor (GitHub/Microsoft) preprint with a small completion sample (35 of 95 accepted; ~63% attrition) and a wide confidence interval (21–89%). It does not generalize to experienced developers working in large, familiar codebases.

So do AI coding tools help or not? Both — depending on context. Juniors and greenfield tasks gain (peer-reviewed +26% task completion); experienced devs in mature repos can lose (−19% in the one RCT). The sign flips with developer expertise and codebase familiarity.

What's the "operating-point trap"? When an effect changes sign across the operating range, any single headline number is misleading. AI coding speedup is positive for novices/greenfield and negative for experts/mature code, so "AI makes devs X% faster" has no single true value.

What is the perception–reality gap? In the METR trial, developers were 19% slower with AI yet believed they were ~20% faster — a ~39-point gap. It means self-reported productivity is an unreliable measure of actual output.

Is your model proof? No — it's the smallest model that shows the two RCTs are consistent (one mechanism, two operating points), robust across parameters. The verified facts are the studies; the model explains why they don't conflict. Code and the full verification dossier are in the Crucible.