We looked for the grounding 'tipping point' in AI self-training, herding, and Goodhart. Mostly, it isn't there.
A popular story says systems that lose touch with reality fail at a tipping point: an AI that trains on its own output collapses past a threshold; a crowd that watches itself flips into a bubble; a metric that gets gamed breaks suddenly. We tested four minimal models directly, with a matched positive control — and mostly, there's no cliff: three degrade smoothly, the fourth (metric-gaming) has an unresolved caveat we can no longer independently confirm the direction of.
A popular story says systems that lose touch with reality fail at a tipping point: an AI that trains on its own output collapses past a threshold; a crowd that watches itself flips into a bubble; a metric that gets gamed breaks suddenly. The tipping-point framing is everywhere. We built four minimal models of the most-cited mechanisms and tested it directly — across a range of system sizes, with the standard physics toolkit for detecting phase transitions — and mostly found no tipping point. Three fail smoothly; the fourth (gaming) shows at most mild path-dependence we could not fully resolve.
What we tested
Four minimal models under one shared protocol: self-training (a distribution re-estimated from a mix of real and its own synthetic data), herding (agents weighing a private signal against the crowd), metric-gaming (selection on a proxy with contagious gaming), and — as a negative control — misspecified statistical inference (an omitted confounder). We chose these four because each is publicly described as having a tipping point, so the null is a direct test of the popular claim, not a convenient sample.
For each we swept a "grounding" knob, measured an order parameter from 0 (collapsed) to 1 (truth-tracking) at sizes from 250 up to 64,000 (16,000 for most domains; 500–8,000 for gaming), and applied the finite-size-scaling battery that distinguishes a true critical transition from a smooth slope: Binder-cumulant crossing, susceptibility growth with size, and whether the curve sharpens as the system grows.
What we found
None of the four sharpens. The order-parameter slope is size-independent; the susceptibility does not diverge; the Binder cumulants do not cross.
- Self-training tracks its closed-form fixed point
std = sqrt(g / (1 − (1−g)·s))with retentions = 0.7, to 3–4 decimals at every setting. Caveat: this closed form is the fixed point of a contraction map, which is smooth by mathematical construction whenever s<1 — it can't by itself rule out a real cliff from mechanisms this simple a model excludes, like a hard quantization step or discrete mode-collapse (see "what would change our mind"). A related model in the amplifying regime, s>1, DOES have a genuine deterministic threshold, g* = 1 − 1/s — different regime, not a contradiction; see our follow-up on eight systems. Separately: Dohmatob et al. (ICLR 2025, "Strong Model Collapse") found even a ~1% synthetic-data fraction can cause a sharp scaling-law breakdown — a real threshold, but in the MIXTURE PROPORTION of an otherwise-growing dataset, a different manipulated variable than our fixed-retentiongsweep, so it doesn't contradict this result either; it's a reminder that "smooth in this parameter" isn't "smooth in every parameter." - Herding's apparent steepening is a fixed crossover, not a critical point.
- The misspecified-inference control degrades smoothly too, exactly as a non-critical system should.
- Coupled gaming did not collapse in our sweeps — selection efficiency stayed in [0.84, 0.96] even under runaway coupling, because uniform gaming cancels in the ranking. This is the one caveat: we also saw weak hysteresis (a 0.065 → 0.10 gap between starting ungamed vs. starting fully-gamed) and a slowly-growing susceptibility peak that our finite-size scaling did not fully resolve — possibly a weak first-order effect at larger size or coupling, which we flag as open. Audit update (2026-07-01): the original contagion script could not be relocated; an independent from-scratch reconstruction (a standard threshold-cascade model) reproduced the bounded-efficiency direction but not the specifics: efficiency landed lower and narrower (0.78–0.81, not [0.84, 0.96]), the hysteresis gap shrank with size instead of growing, and the susceptibility peak shrank too (not the slowly-growing peak originally reported). All three point the same way — toward LESS evidence of an emerging cliff, not more — but the exact contagion mechanism was never fully specified in this post, so a reconstruction under a different mechanism is a genuine possibility, and this doesn't retroactively prove the original measurement wrong. Net effect: treat the entire gaming-hysteresis story as unconfirmed pending the original code, not as a settled hint of a larger-scale cliff in either direction.
| System | Grounding knob | Verdict | Key number |
|---|---|---|---|
| Self-training | retention s=0.7 (contraction regime) | no transition — smooth | fixed point matches to 3–4 decimals |
| Herding | private-signal weight | no transition — fixed crossover | susceptibility ratio ≈1.0 across a 64× size range |
| Misspecified inference (negative control) | confounder coverage | no transition — smooth, as expected | slope ratio ≈0.90 across the same range |
| Metric-gaming | contagion strength | bounded; hysteresis direction unresolved | efficiency 0.84–0.96 (original) vs. 0.78–0.81 (reconstruction) |
| Mean-field Ising (positive control) | temperature | real transition found | β≈0.43 (finite-size; exact 0.5), susceptibility grows 11.6× |
The control that makes this trustworthy
A null result is only as good as the instrument. So we ran a positive control — a system known to have a sharp transition (a mean-field Ising model with no external field) — through the identical pipeline. It found the transition cleanly: a Binder crossing at the right place and the textbook critical exponent β ≈ 0.5. The method sees a cliff when there is one. (Our four models are themselves well-mixed / mean-field, so the mean-field Ising is the matched positive control; networked or spatial versions are out of scope here — see the caveat below and "what would change our mind.")
Honest limitation: Ising is canonically critical by construction — it has a genuine double-well free energy with two competing macrostates on purpose. Confirming our pipeline finds its known transition proves the Binder/susceptibility code works; it doesn't as strongly guarantee the same pipeline would find a transition in an unrelated, differently-structured system if one existed there. Three of our four target models (self-training, herding, misspecified inference) never had multiple competing macrostates to begin with — closer to testing a smoke detector against a bonfire and inferring it would catch a candle. Take the positive control as reassurance the method isn't obviously broken, not as proof it would catch every real cliff.
Prior art we're building on
None of these four questions are new. Self-training collapse has been characterized as progressive and gradual rather than a sharp cliff since Shumailov et al. 2023, "The Curse of Recursion" — our smooth result in the retention regime (s<1) is consistent with that, not a fresh discovery. Cascade theory has long noted that information cascades are fragile rather than robustly sharp — Bikhchandani, Hirshleifer & Welch 1992 introduce the mechanism, and their 1998 follow-up ("Learning from the Behavior of Others," J. Economic Perspectives 12(3)) is explicit that agents in a cascade sit close to indifferent, so even a small new public signal can dislodge one; Centola's work on social tipping (Science, 2018) does find a real ~25% threshold, but in a different network-committed-minority setup than our well-mixed model — so our herding null doesn't contradict Centola, it just tests a different question. Applying statistical-physics finite-size-scaling tools (Binder cumulants, susceptibility scaling) to test whether social or AI dynamics are genuinely critical is itself an established cross-disciplinary technique — see Castellano, Fortunato & Loreto's review, Rev. Mod. Phys. 81, 591 (2009). We haven't found a prior study running this specific four-system, one-protocol comparison with a matched positive control; that combination — not any single result in isolation — is the contribution here.
Why
The systems differ, but the reason is shared — a shared explanation, not a shared universality class: the four do not collapse onto one curve or share critical exponents. Any real grounding signal enters as a symmetry-breaking field, and a field rounds a sharp transition into a smooth slope. A genuine cliff only reappears in the singular limit of zero grounding (perfect self-reference) — which our zero-field positive control demonstrates as a separate, idealized system, and which real systems never quite reach. The intuition is inverted: grounding doesn't push you toward a tipping point; its absence is what manufactures one.
What would change our mind
A faithful version of any of these systems in which the susceptibility grows with size toward a single threshold, the Binder cumulants cross there, and the slope diverges — with the same critical exponent in two or more of them. We didn't find it in the standard mechanisms; the regimes we did not rule out (a hard quantization step, runaway variance inflation, exactly-zero grounding, and the gaming hysteresis whose size-direction a later audit could no longer confirm) are where to look — plus one we underweighted at first: all four models here are well-mixed (mean-field). Real self-training corpora, real social herding, and real gamed metrics all run on structured, heterogeneous networks, and our own related work on network epidemic thresholds shows topology alone can create or erase criticality that a well-mixed test structurally cannot see — a well-documented pattern in the physics literature: Pastor-Satorras & Vespignani (2001) showed the classic epidemic threshold that homogeneous-mixing models predict can vanish entirely on real heterogeneous (scale-free) contact networks, and explosive percolation shows the reverse can also happen — a small structural rule change (limited-choice edge selection) turns the classic smooth transition into one with an extremely steep, anomalous scaling (reported as discontinuous at first; later rigorous work showed it's technically continuous, just with a much sharper approach than the classic case). Either direction is possible; a well-mixed null doesn't settle it. That is now the single biggest open direction, not a footnote.
The practical takeaway
If you worry about model collapse, crowd bubbles, or a gamed KPI, you're probably not facing a hidden cliff you'll fall off without warning. You're facing a smooth, measurable decline — which is better news: it's visible early, and a little real-world grounding buys a lot of margin. The danger isn't a sudden tip; it's slow, unnoticed drift toward the zero-grounding limit.
All figures from simulation; models and protocol are minimal and re-runnable. This piece reports a negative result with both a negative and a positive control.
FAQ
Do systems that lose touch with reality fail at a tipping point? Mostly no. Across the four systems people claim have one — AI self-training, herding crowds, metric-gaming, misspecified inference — each degrades smoothly, with no sharp threshold. The one exception (gaming) is still open, not settled.
Is the method just too weak to detect a cliff? No. On a real phase transition it found the cliff cleanly — a Binder crossing at the right place and the textbook critical exponent β ≈ 0.5. It sees a tipping point when one genuinely exists — though a canonically-critical system like Ising is an easier target than an arbitrary one, so treat this as reassurance the pipeline isn't broken, not proof it would catch every real cliff.
Was there any sign of a real threshold? One honest caveat: weak hysteresis (a 0.065 → 0.10 gap) hinted a true transition could emerge at larger scales than we tested. A 2026-07-01 audit could not relocate the original script; an independent reconstruction reproduced the bounded-efficiency direction but not the specifics — efficiency lower/narrower than claimed, hysteresis shrinking not growing, and the susceptibility peak shrinking not growing — all three pointing toward less evidence of a cliff, not more. We flag the whole gaming-hysteresis story as unresolved pending the original code, not as a settled hint either way.
Why does this matter? “Tipping point” framing invites waiting for a threshold that never comes. Smooth degradation means you must monitor continuously, not watch for a cliff. We later extended this to eight systems.