The Operating-Point Trap: methods break exactly where they are neededPasca prevádzkového bodu: metódy zlyhávajú presne tam, kde ich potrebuješ

There is a famous result in basketball that everyone half-remembers: the "hot hand" is a myth. Players who feel hot, the story goes, are fooling themselves — a made shot tells you nothing about the next one. For thirty years this was settled science, taught as a clean parable about human overconfidence.

It was wrong, and the way it was wrong is the most useful thing about it.

The original analysis measured a simple quantity: after a streak of made shots, how often does the next shot go in, versus after a streak of misses? The difference came out near zero, so: no hot hand. The trouble is that this estimator is biased on a fair coin. Rebuild it and run it on a shooter with provably no hot hand — pure randomness — and it does not read zero. It reads about −8 percentage points at the sample sizes those studies used, and the bias grows worse, to about −17 points, when you look at longer streaks. A measurement of "zero" in that procedure is therefore evidence for a real hot hand of roughly the size that was being denied. The parable about human error was itself the error.

We rebuilt that analysis in code, and then we kept going — rebuilding claim after claim from finance, network science, machine learning, and cognition as the smallest runnable model we could, and measuring where each one holds and where it breaks. After a couple of dozen of these, a pattern surfaced that we did not go looking for. It is sharp enough to name.

The pattern

A standard method is calibrated in the benign regime, and its error is wired to the very thing that defines the hard regime — so it breaks exactly at the operating point that made you reach for it.

Watch it repeat:

• Diversification. The textbook says thirty stocks gives you essentially complete diversification. Measured, that is true for ordinary volatility — about 96% of the achievable risk reduction. But for tail risk, under the heavy-tailed returns that real markets have, thirty stocks captures only about 85%, and you need closer to a hundred. The rule fails in the fat tail — which is the only part of the distribution diversification was supposed to protect you from.

• The wisdom of crowds. Averaging many independent estimates is genuinely powerful. But let the estimators watch each other, and accuracy collapses; in simulation the crowd needs roughly 80% independence before the averaging buys you much at all. The method fails under correlation — and correlation is the normal condition of any crowd that can see itself.

• Forgetting, in an AI's memory. A popular trick keeps memories alive by how recently they were accessed. We pit that against keeping memories by value under a shrinking budget. At a tight budget the access-based policy retained only 3% of the rare-but-critical memories and a fifth of the total value; the value-aware policy kept all of them and three times the value. Recency-based forgetting fails precisely when memory is scarce — the only time forgetting policy matters at all.

Three domains, three estimators, one skeleton. The list is longer — venture-capital returns, early-warning signals for tipping points, the conditions under which a diverse team beats an expert one — and every time the failure has the same shape.

Why this isn't just "be careful with statistics"

The instinct is to file this under sampling noise: bad luck you can beat with more data. That instinct is exactly what makes the trap dangerous, because the error here is not random. It is systematic, and it is monotone in the stress. The fatter the tail, the smaller the sample, the tighter the budget, the more correlated the observations — the larger the bias gets. You cannot average it away, because the regime where you'd have enough slack to be careful is the regime where you didn't need the method in the first place.

That coupling is the whole point. In each case the structural feature that defines the hard case — a heavy tail, a short streak, a dependency, a scarce slot — is the same feature that biases the estimator. The headline number you get in the demo is a benign-regime mirage. It was measured where the method works and quoted where it doesn't.

What to do with it

Two habits fall out, and they cost nothing:

1. 1. Test at the operating point, not the demo. Validate an estimator under the conditions you'll actually run it in — the small sample, the tail, the dependency, the constraint — not the comfortable average where everything behaves.

1. 2. Ask one question of any metric: what is the stress variable, and does the bias grow or shrink in it? If the error gets worse as conditions get harder, the headline figure is telling you about a world you don't live in.

The honest part

We could be wrong, and here is exactly what would change our mind: a domain where a standard method's bias shrinks as the stress rises — where it becomes more reliable as samples shrink, tails fatten, dependence climbs, or budgets tighten. We have not found one; every case we've rebuilt worsens monotonically toward the hard regime. A clean counterexample would demote this from a law to a tendency, and we'd publish that too. Every claim above is a small program you can run; the failures are not anecdotes, they are reproductions.

The deeper reason we keep doing this: a measured number feels like the end of an argument, and it is usually the middle of one. The number is true — in the regime where it was taken. The mistake is carrying it, unexamined, to the regime where the decision actually gets made.