Skip to content
vivek pal

wiki / self-improving models

how self-improvement goes wrong

Failure modes of models trained on their own outputs, the symptom each one leaves, and the guard for it.

updated · 5 min read

A model that trains on its own outputs can get worse in ways its own scores won’t show. Each failure below has a symptom you can measure and a guard you can build before the first round runs. The loop itself is in the self-improvement loop.

model collapse

Shumailov et al. (Nature, 2024) trained each generation of a model on data produced by the generation before it. The models lost the tails of the distribution first, then drifted toward narrow, low-variance output, and later generations produced sequences the original model would never have produced. In their OPT-125m language experiments, keeping a random 10% of the original data in every generation led to only minor degradation.

Gerstgrasser et al. (2024) pinned down when this happens. Collapse follows when synthetic data replaces the real data each round. When synthetic data accumulates alongside the original real data, test error stays bounded no matter how many rounds run.

  • Symptom: rare cases get worse first. Outputs grow more uniform. Loss on real held-out data rises across rounds while scores on self-generated data keep improving.
  • Guard: never train on synthetic data alone. Keep the real data in every round’s mix and add to it instead of replacing it. Track accuracy or perplexity on a fixed set of real examples every round.

reward hacking

Krakovna et al. (DeepMind, 2020) define specification gaming as behaviour that satisfies the literal specification of an objective without achieving the intended outcome. Their example is an agent rewarded for the height of a red block’s bottom face, which flipped the block over instead of stacking it. Gao, Schulman and Hilton (2022) measured the language-model version: as a policy is optimized against a proxy reward model, the score from a held-out gold-standard reward model can rise and then fall, even while the proxy score keeps climbing.

  • Symptom: training reward keeps going up while held-out task success flattens or drops. Outputs start matching the checker’s surface, for example a test that only checks that a file exists.
  • Guard: check outcomes, not surface signals. Add a KL penalty toward a reference model; InstructGPT used a per-token KL penalty from the SFT model to limit reward-model over-optimization. Evaluate with a different checker from the one used in training, and read a sample of outputs by hand each round.

verifier drift

When the verifier is learned, or is the model itself as in Self-Rewarding Language Models, it changes as the policy changes. Nothing in the loop stops the judge from coming to prefer whatever the generator happens to produce. Scores then stay high because the judge moved, not because the outputs got better.

  • Symptom: self-assigned scores climb faster than scores from a grader that wasn’t trained.
  • Guard: keep a frozen reference evaluator that never trains, such as a fixed checkpoint, a fixed test suite, or a human-labeled set, and compare every round against it.

diversity loss

Kirk et al. (2023) found that RLHF generalizes better than SFT to new inputs but significantly reduces output diversity. Filtering to passing samples pushes the same way, because the easiest strategy that passes gets the most copies. In a self-improvement loop this compounds: the next round samples from a narrower model, so fewer new strategies show up for the filter to keep.

  • Symptom: pass@1 rises while pass@k at large k stalls or falls. Samples for the same prompt start to look alike.
  • Guard: track pass@k and the number of distinct passing solutions per prompt. Sample at a higher temperature when generating training data, deduplicate kept samples, and cap how many examples one prompt can contribute.

benchmark contamination

Generated tasks can reproduce benchmark items the base model memorized, and the loop then trains on the test. Yang et al. (2023) showed that rephrased test samples get past n-gram decontamination, that a 13B model trained on rephrased benchmark data can reach GPT-4-level benchmark scores, and that synthetic datasets generated by GPT-3.5 and GPT-4 also contained benchmark overlap.

  • Symptom: large jumps on public benchmarks without a matching gain on your own fresh tasks.
  • Guard: decontaminate generated data against every eval set with embedding or LLM-based matching, not only n-grams. Keep a private eval set written after the training data was collected.

catastrophic forgetting

Fine-tuning on narrow self-generated data can erase skills that data never covered. Biderman et al. (2024) compared LoRA with full fine-tuning on code and math. LoRA learned less on the target domain but forgot less of the base model’s abilities, mitigated forgetting better than weight decay and dropout, and kept generations more diverse.

  • Symptom: the target eval goes up while unrelated evals (general questions, formatting, other tools) go down.
  • Guard: run a broad regression suite every round, not only the target eval. Prefer LoRA adapters when the training data is narrow, mix in general data, and keep the previous checkpoint so you can roll back.

a minimal release gate

A round ships only if all of these hold:

  1. The target held-out eval improves by more than run-to-run noise.
  2. The frozen reference eval and the regression suite don’t drop by more than noise.
  3. Loss on the fixed real-data set doesn’t rise.
  4. pass@k and distinct solutions per prompt aren’t falling.
  5. Someone read a sample of the new training data and the new outputs.

sources

← all notes