Skip to content
vivek pal

wiki / self-improving models

the self-improvement loop

How a model trains on its own checked outputs, and why the checker sets the ceiling.

updated · 5 min read

the shape

Most self-improvement methods for language models run the same loop. The model generates candidate outputs for a set of prompts. Something checks them. The outputs that pass (or pairs of better and worse outputs) become training data. The model is fine-tuned on that data and evaluated, and the next round starts from the new checkpoint.

policy = base_model
for round in 1..N:
    prompts   = sample_tasks()
    samples   = [policy.generate(p, k=8) for p in prompts]
    scored    = [(p, s, verify(p, s)) for p, s in samples]
    keep      = [(p, s) for p, s, ok in scored if ok]   # or preference pairs
    candidate = finetune(policy, keep)
    if beats(candidate, policy, heldout_evals) and no_regressions(candidate):
        policy = candidate

The papers below differ in three places: what verify is, what gets kept, and how the update is made.

the lineage

STaR (Zelikman et al., 2022). The model writes rationales for questions with known answers, and the rationales that reach the right answer are kept. For questions it got wrong, the model is given the correct answer and asked to produce a rationale for it, which the paper calls rationalization; those are kept too if they reach the answer. Fine-tune on everything kept and repeat. The verifier is a match against the known answer.

Rejection-sampling fine-tuning (Yuan et al., 2023). Sample many reasoning paths from a supervised model, keep the ones with correct final answers, and fine-tune on them. The paper found that more distinct correct paths helped more than more copies of similar ones, and that weaker models gained more from the extra data.

ReST (Gulcehre et al., 2023). Framed as growing-batch offline RL. A Grow step samples a dataset from the current policy. Improve steps filter that dataset with a reward model at rising thresholds and fine-tune on what passes. The experiments in the paper are on machine translation.

ReST-EM (Singh et al., 2023). A simpler version for tasks with binary feedback: generate, filter by correctness, fine-tune, repeat a few times. On MATH and APPS with PaLM 2 models it beat fine-tuning on human-written solutions alone, and it scaled well with model size.

Self-Instruct (Wang et al., 2022). Here the model also writes the tasks: new instructions, inputs, and outputs, bootstrapped from a small seed set. Invalid or near-duplicate examples are filtered out before fine-tuning. The filter is a set of heuristics rather than a correctness check, so it teaches instruction following better than it teaches reasoning.

Self-Rewarding Language Models (Yuan et al., 2024). The model scores its own responses with an LLM-as-a-Judge prompt, builds preference pairs from those scores, and trains on them with iterative DPO. The paper reports that both instruction following and judging improved across iterations. Generator and verifier are the same weights.

SPIN (Chen et al., 2024). Self-play against the previous iteration. The model learns to tell the human-written SFT responses apart from its own earlier outputs and to move toward the human ones. It needs no human data beyond the SFT set, and it also can’t go past that set: the paper proves the training objective reaches its optimum only when the policy matches the target data distribution.

GRPO and RL with verifiable rewards. DeepSeekMath (Shao et al., 2024) introduced Group Relative Policy Optimization, a PPO variant that drops the value network and computes each sample’s advantage relative to the other samples drawn for the same prompt. DeepSeekMath scored those samples with a reward model. DeepSeek-R1 (DeepSeek-AI, 2025) used GRPO with rule-based rewards instead: an accuracy reward checked by a deterministic rule, such as whether the final answer matches or whether the tests pass, plus a format reward. The R1 paper says it left out neural reward models for reasoning because they were prone to reward hacking in large-scale RL.

why the verifier sets the ceiling

Each round, the model learns only from what passed the check. If the verifier accepts a wrong answer, the model is trained to produce more wrong answers of that kind. If the verifier rejects a correct but unusual answer, that behaviour gets filtered out and is never reinforced. The loop climbs toward whatever the verifier rewards, and that matches the real goal only as far as the verifier is right.

This is why the strongest results come from domains with cheap, exact checks: math with known answers, code with unit tests, and environments that report success. It also explains the spread across the methods above. A match against a known answer (STaR, ReST-EM, R1’s accuracy reward) leaves little room for error. A learned reward model (ReST, DeepSeekMath) can be exploited, which is the over-optimization problem. A model grading itself (Self-Rewarding) moves with the policy it is grading, so its mistakes can compound instead of staying fixed.

What follows for a small model:

  • Work on the verifier before the generator. A better check raises the ceiling of every later round.
  • Prefer checks you can execute, such as running the test, reading the file back, or comparing the final screen, over checks that ask a model for its opinion.
  • Keep the release gate separate from the training verifier. If one check both filters the training data and decides what ships, you can’t see it being gamed.
  • Sample several attempts per prompt. Filtering only works if some attempts pass, and GRPO’s group-relative advantage needs variation within each group.

The failure modes that show up when these rules are skipped are in how self-improvement goes wrong.

sources

← all notes