Your AI might be training on itself — and we measured the two ways that ends badly
Model collapse, measured. Any system that learns from its own output is a strange loop. We built the smallest runnable model and found two failure modes — and two knobs that prevent each: a ~5% real-data anchor pulls the collapse rate (in an unfiltered loop) from ~94% to ~6–10%, and keeping the self-trust exponent p≤1 prevents permanent lock-in. Both halves are established results (Shumailov 2024; Arthur 1989) — we add the runnable packaging.
The claim. Any system that learns from its own output is a strange loop — a model retrained on synthetic data, an agent whose memory is its own past answers, a RAG store indexing the system's prior generations, a recommender fed by the clicks it created. In 2026 this stopped being a research curiosity: model collapse is now a documented production concern. So we built the smallest honest model of it and measured the two ways a self-referential system fails — and the two knobs that prevent each.
Failure 1 — collapse (the data-mix law). Retrain a model recursively on its own outputs and its diversity drains away — the "curse of recursion" documented by Shumailov et al. (Nature 2024). In our minimal simulations — two independent toy models that agree — with no real data ~92–94% of runs collapsed to near-zero diversity. The cure is a floor of real/external data, and the floor is low: in an unfiltered resampling loop (the worst case; verifier/selection filtering can reverse it) a ~5% real-data anchor pulls the collapse rate to ~6–10%, and 20% makes it clean (0–1%) — and peer-reviewed theory backs the low floor: iterative retraining is stable exactly when the share of real data is large enough (Bertrand et al. 2024). The lesson isn't "never use synthetic data" — it's "never replace real data with it." Accumulate, don't substitute — exactly what Gerstgrasser et al. (2024) prove keeps the error bounded.
Failure 2 — lock (increasing-returns lock-in, applied to self-trust). If a system weights its own prior belief faster than fresh evidence can correct it (a self-trust exponent p > 1), a fixed fraction of any initial bias is never washed out. This is not a new law — it is classic increasing-returns lock-in (Arthur 1989; Pemantle 2007), the same reason stochastic approximation needs the step sizes to sum to infinity to forget where it started. The "closed form" is just the infinite-product theorem: ∏(1 − 1/(n+1)p) is nonzero iff p > 1, telescoping to exactly 1/2 at p = 2, ≈0.81 at p = 3, ≈0.18 at p = 1.5. Those numbers are a textbook identity for the chosen 1/np weighting — not a discovered constant, and the schedule is an assumption, not a measured property of any real system. The genuinely usable part is the tell: inject a known bias, keep feeding unbiased data — if it doesn't decay, your update rule is super-linearly self-weighting.
The honest caveat. These are minimal models, not your training run — but they're runnable, the thresholds are reproducible, and the peer-reviewed literature agrees on the cure: mixing/accumulating real with synthetic provably bounds the error (Gerstgrasser et al. 2024), while replacing real with synthetic grows it without bound (Shumailov et al. 2024). Both halves are established results, not our discoveries — collapse (Shumailov 2024; Gerstgrasser 2024; Alemohammad et al. 2024) and lock-in (Arthur 1989; Pemantle 2007); all we add is the runnable packaging. Two more honest caveats: (1) the ~94% collapse figure is an unfiltered loop (resample with no quality filter) — the regime few real pipelines run; selecting synthetic outputs (verifier filtering, best-of-N, RL with a reward model — why systems like AlphaZero improve under near-pure self-generation) injects outside signal and can reverse collapse, which our single external_fraction knob does not model — and conversely, in an accumulating corpus even a ~1% synthetic fraction can shift the scaling laws (Dohmatob et al. 2024), so the low floor is not a universal safety guarantee. (2) The two failures are separate mechanisms (variance destroyed by finite-sample noise vs. a deterministic mean-bias contraction), not two faces of one law. The point is to turn "model collapse" from a vibe into a number you can put a threshold on.
We packaged the check as selfref: one zero-dependency file, plus an MCP server so an agent can ask — before it retrains on itself — am I about to collapse or lock? It's open-core and free, a sibling of our memory, RAG-freshness and statistics tools. One call, audit(external_fraction, self_trust_p), gives you a verdict and the fix.
FAQ
What happens when an AI trains on its own output? Two distinct failures: collapse (a data-mix law) where output degrades into nonsense, and lock (a self-trust law) where the system over-trusts its own past answers and stops correcting against reality.
Can you prevent model collapse? In our minimal models, yes — cheaply. In an unfiltered resampling loop (the worst case; verifier/selection filtering reverses it), a ~5% real or external-data anchor pulls the collapse rate from ~94% to ~6–10%, and 20% makes it clean (0–1%). The floor of real data needed is surprisingly low — and peer-reviewed theory agrees (Bertrand et al. 2024).
What is the self-trust “lock”? When a system's memory is its own prior answers, it locks onto them — each cycle treats yesterday's output as evidence, so errors compound instead of being corrected by outside signal.
Where does training-on-itself show up? Any strange loop: a model retrained on synthetic data, an agent whose memory is its own past answers, or a RAG store indexing the system's prior generations.