As a model gets better at posing and solving problems, it still needs a trustworthy way to check its answers.
Part 9 asked whether AI can even generate its own training inputs. Now we examine self-play: one role proposes problems, another solves them, and their results feed into the next round of training.
What matters more than the number of problems is practice worth learning from and feedback worth trusting. Repeating familiar problems rarely builds a new skill. The same goes for posing impossible problems or learning incorrect answers as if they were true. How does that balance break down, and what can restore it?
Video timestamps by slide
| Slide | Video | Topic |
|---|---|---|
| 46 | 45:39–48:12 | Proposer, solver, verifier and appropriate difficulty |
| 47 | 48:12–49:00 | Difficulty and pseudo-labels of R-Zero’s generated problems |
| 48 | 49:00–50:24 | Plateaus and declines after early gains |
| 49 | 50:24–51:24 | An unexpected generation in Absolute Zero |
| 50 | 51:24–52:17 | External information in SPICE and R-Few |
#Slide 46 — A good problem is challenging and still solvable
At 45:39, the lecture introduces Absolute Zero, R-Zero and Self-Questioning Language Models and distinguishes three roles. The proposer creates inputs, the solver answers them and the verifier supplies a training signal for each answer.
| Role | What it does | Training question |
|---|---|---|
| Proposer | Creates practice problems | What can the current solver learn from this? |
| Solver | Solves a given problem | Which answers and solutions should become more likely? |
| Verifier | Judges the suitability of an answer | How well does the score reflect actual success? |
The three robots need not represent three networks. In Absolute Zero Reasoner, one model proposes and solves problems and a Python executor checks the results. R-Zero trains a Challenger and a Solver separately from the same base model. Distinct roles can share parameters or use separate ones. Absolute Zero §3 · R-Zero §2
#A solver’s failure is not always a proposer’s success
The lower graph puts solver loss on the horizontal axis and proposer loss on the vertical axis. Proposer loss is high on easy problems to the left and overly difficult problems to the right. Minimizing its loss draws the proposer toward the middle.
A purely adversarial goal rewards whatever makes the solver fail. Ambiguous or unsolvable problems may drive up the failure rate while providing little useful practice. The goal here is a problem that slightly stretches current ability and still allows meaningful feedback.
Papers implement that condition differently. The curve does not imply that every method peaks at 50% accuracy. Start with R-Zero’s specific design.
#Further reading — R-Zero estimates difficulty from answer agreement
The upper row fixes the Solver and trains the Challenger. The lower row fixes the updated Challenger and trains the Solver. Because people do not label each generated problem, the most frequent answer from repeated Solver attempts becomes a pseudo-label. R-Zero §2.2–2.4
Suppose we obtain m final answers for problem x. The modal answer and its agreement rate are defined as follows, assuming a procedure for recognizing equivalent final answers.
The paper calls this empirical accuracy. Without a known correct answer, the directly observed quantity is the proportion agreeing with the modal answer. The Challenger receives this uncertainty reward.
The reward peaks at agreement 0.5. Here are examples from ten responses.
| Count of modal answer | Agreement | Uncertainty reward |
|---|---|---|
| 1 | 0.1 | 0.2 |
| 3 | 0.3 | 0.6 |
| 5 | 0.5 | 1.0 |
| 7 | 0.7 | 0.6 |
| 10 | 1.0 | 0.0 |
Unanimous answers bring little reward to the proposer. Highly dispersed answers also reduce reward. In the experiment, the researchers obtained ten responses per problem and kept problems whose modal answer occurred three to seven times for Solver training. Passing this filter alone cannot guarantee the quality of the problem or its pseudo-label. R-Zero §3.1
A separate term discourages the proposer from repeating problems. Let B be the batch size and C a cluster of similar problems. For problems that pass the format check, the final reward is
The paper clusters by BLEU-based similarity and uses λ=1. If uncertainty reward is 1 and three of ten items belong to the same cluster, the final reward is 0.7. Matching difficulty and providing varied practice are separate objectives. R-Zero §2.2
The Solver then receives reward 1 if its answer matches the selected problem’s pseudo-label and 0 otherwise.
A shared misconception can make an agreed-upon wrong answer the training target. Suppose seven of ten responses give incorrect A and three give correct B. Agreement is 70%, while actual accuracy is 30%. This problem even passes the three-to-seven filter. Majority voting aggregates answers. It provides no independent check on the chosen answer.
#Slide 47 — Answer quality can change as problems get harder
At 48:12, the lecturer fixes one solver and reads down its column. If the same solver answers fewer later-stage problems correctly, the proposer may have begun to create harder problems.
Read the final column too. For each stage, R-Zero sampled a set of 200 generated problems and treated GPT-4o’s answers as the reference. The reported accuracy is relative to that reference. The dataset was not fully verified by humans. R-Zero §4.4, Table 5
| Problem generation stage | Base | Solver 15 | Solver 30 | Solver 45 | Pseudo-label accuracy |
|---|---|---|---|---|---|
| Step 15 | 48.0% | 59.0% | 57.0% | 61.0% | 79.0% |
| Step 30 | 52.5% | 53.0% | 51.5% | 53.5% | 69.0% |
| Step 45 | 44.0% | 47.0% | 45.0% | 50.5% | 63.0% |
#The same solver on different problems
The Solver 15 column goes from 59% to 53% to 47%. Later problems were harder for this solver. The Base column goes from 48% to 52.5% to 44%, so the decline is not monotonic in every column. Each row uses different problems, making the table alone insufficient to establish whether the Solver improved.
Read across the first row to compare solvers on the same problems. Solvers 15, 30 and 45 reach 59%, 57% and 61%. A later checkpoint does not always outperform the preceding one. Changes in problem difficulty and changes in solver performance require separate comparisons.
#Are harder problems better training data?
Pseudo-label accuracy fell from 79% to 63%.
For samples of 200, the numbers of pseudo-labels that disagree with the reference are
The count grows from 42 in the early stage to 74 in the later stage, or 32 more training targets that disagree with the reference. This is a comparison of equal-sized samples drawn at different stages. It does not mean that the same 32 problems changed from correct to incorrect.
Harder problems alone do not establish a better curriculum. The learner can receive increasingly difficult problems and increasingly unreliable targets at the same time. Track difficulty and target reliability together in self-play.
#Slide 48 — Early improvement need not continue
From 49:00, the lecturer asks whether repeated training raises performance indefinitely. All three model sizes improve initially. The two smaller models plateau early, and the 4B model also declines at the final point shown. R-Zero §4.2, Figure 3
| Model | Base | Step 15 | Step 30 | Step 45 | Step 60 |
|---|---|---|---|---|---|
| Qwen3-0.6B-Base | 16.77 | 27.39 | 27.15 | 25.05 | 25.15 |
| Qwen3-1.7B-Base | 32.71 | 39.19 | 39.20 | 37.03 | 38.27 |
| Qwen3-4B-Base | 42.58 | 48.06 | 48.44 | 49.07 | 46.52 |
We compare the mean math benchmark scores shown in the graph. The highest displayed score for the 4B model is 49.07 and the final score is 46.52.
A model can finish above its starting point and below its peak. The 0.6B model initially gains 10.62 points but loses 2.24 points from step 15 to step 60. Distinguish the final checkpoint from the best one.
The figure reports observations on three models under this training setup. It neither proves that all self-play must stall nor isolates pseudo-label errors as the cause. Declining target quality from slide 47, a narrowing problem distribution and the model’s initial capabilities are possibilities to investigate together.
#Recording training reward separately from actual performance
Reward on self-generated data can change meaning during training. A generator that returns to easier problems may raise average reward. More answers sharing the same mistake can raise agreement. Keep a separate fixed held-out set as a reference.
With evaluation problems and answers held fixed, define accuracy as
The evaluation problems stay fixed as the proposer and solver change. In practice, also match decoding settings and the number of samples per answer. Select among checkpoints on a validation set and reserve the final test set for evaluation to preserve the comparison.
Three measures help track self-play.
- Solver performance. How well does it solve fixed, unseen problems?
- Target reliability. How often do generated answers match an independent reference?
- Problem coverage. Do topics, difficulty and phrasing remain diverse?
Together they help distinguish genuine capability gains from adaptation to the model’s own evaluation system.
#Slide 49 — Checking answers by execution leaves some behavior unscored
At 50:24, the lecture discusses an Absolute Zero example. While generating a complex programming problem, the model produced derogatory reasoning text that cast another model and humans as competitors. The paper labels the example from step 132 of a Llama-3.1-8B-based model an “Uh-oh Moment.” Absolute Zero, Figure 34
What was observed directly was generated text. This observation alone establishes neither an internal intention nor actual aggressive behavior. It raises a narrower question: does the score used to check an answer or program evaluate every expression and action in the generating process?
#Further reading — Absolute Zero uses execution to construct problems and check answers
The basic unit in Absolute Zero Reasoner (AZR) is a triple consisting of program P, input I and output O.
Hiding different parts of the relationship produces different reasoning tasks. Absolute Zero §3.2
| Task | Given | Find | Verification |
|---|---|---|---|
| Deduction | Program and input | Output | Compare with the execution result |
| Abduction | Program and target output | A possible input | Run the proposed input and check for the target output |
| Induction | Some input-output examples and a description | Program | Test execution results on hidden inputs |
For P(i)=i²+1 and input 2, the output is 5. Asking for the output is deduction. Given only the program and output 5, both 2 and −2 work. The verifier should check whether the proposed input actually produces the same output, even if it differs from the hidden input.
Induction needs additional care. Many programs can match a few input-output examples. AZR supplies a description and some examples, then checks the others in secret. This reduces simple memorization. Passing finitely many tests does not prove correctness on every input.
#A different proposer reward from R-Zero’s
AZR averages solver rewards from execution-based verification. For G attempts the mean binary success signal is
The proposer’s learnability reward is
A problem with no successes earns zero. Once at least one attempt succeeds, a lower success rate gives a higher reward. The following example uses G=8, the paper’s setting for estimating accuracy. Absolute Zero §3.1, equation (4) · Hyperparameters, Table 3
| Successful attempts | Success rate | Proposer reward |
|---|---|---|
| 0/8 | 0.0 | 0.0 |
| 1/8 | 0.125 | 0.875 |
| 4/8 | 0.5 | 0.5 |
| 8/8 | 1.0 | 0.0 |
AZR’s reward therefore does not peak at 50% success. It favors difficult problems the solver can occasionally solve, with a different shape from R-Zero’s reward. R-Zero measured agreement with a modal answer, whereas AZR measures success under execution. The table shows the base learnability reward. The full training reward also includes format and validity conditions. Absolute Zero, equations (4)–(6)
“Zero Data” here still assumes pretraining. The method starts with a pretrained model, a program execution environment and specified task formats. The implementation includes an identity-function example to initiate generation. Without a person supplying each new problem and answer, the executor can give feedback independent of the model’s current beliefs. Absolute Zero §3.3.1, Figure 5
The example on slide 49 illustrates the scope of that verification. Checking format and execution results leaves the appropriateness of some generated text outside the reward. A system designer needs criteria for problem validity, the final answer and behavior during generation.
#Slide 50 — External information connects self-play to the world
At 51:24, the lecture turns to reference documents and small sets of human examples. SPICE and R-Few bring external information into problem generation to reduce the risk of training getting stuck on shared mistakes or a narrow range of topics.
#Further reading — SPICE reads a document to ask the question and answers without it
SPICE (Self-Play In Corpus Environments) uses one model in two roles. The Challenger reads document d and produces question q and reference answer a*. The Reasoner receives only the question. The question-generation instructions call for questions answerable without the document. SPICE §2.1–2.3
The roles have asymmetric information despite shared parameters. The proposer grounds its problem in the document. The solver must answer without looking it up. The document grounds the question and target, and the solver earns a reward for matching the reference answer.
The Challenger’s reward depends on the variance of solution outcomes. For a valid question, the paper defines it as follows.
The variance of a binary success signal with probability p is
At p=0.5 the variance is 0.25 and the proposer reward is 1. At p=0.1 or 0.9 the variance is 0.09 and the reward is about 0.278. At p=0 or 1 the reward is about 0.044. These numbers use the binary distribution’s variance. Variance from finitely many responses depends on the estimator. R-Zero’s triangular reward and SPICE’s variance-based reward have different formulas. SPICE, equation (1)
Documents can contain errors, and extraction or question generation can introduce new ones. Compared with repeatedly accepting the model’s current modal answer, a document nevertheless supplies external evidence for checking the problem and its answer.
#Further reading — R-Few uses a few human examples to guide generation
R-Few keeps a small set of human-labeled anchor examples. The Challenger takes some of them as in-context examples and generates problems. The Solver trains on human examples and generated problems selected at an appropriate difficulty for its current ability. The arithmetic problem in the figure illustrates the setup. R-Few §3.2–3.3, Figure 2
Write the set of human examples as
The generator receives between zero and five in-context examples. These guide the form and content of the questions. For solver training, the model attempts each problem several times and measures the proportion of responses matching the reference. It uses data whose estimated success rates lie in this range.
The condition applies to both generated and human examples. Human examples use supplied answers. Synthetic problems use majority-vote pseudo-labels. Their success-rate estimates therefore rely on references of different reliability. R-Few §3.3, equations (7)–(9)
The denominator matters when interpreting “Few.” The paper’s 1% and 5% are fractions of WebInstruct’s 232,000 problems selected for the pool of human examples. They are not fixed human-data fractions in every training batch. A reward weight of 2.0 for human-labeled answers likewise does not specify a 2:1 data mix. R-Few §4.1
#Which external reference should we add?
| Method | Basis for problems and answers | Main answer check | Limitation |
|---|---|---|---|
| R-Zero | Challenger-generated problems | Solver’s modal answer | A shared wrong answer can become a pseudo-label |
| Absolute Zero Reasoner | Generated programs and input-output relations | Program execution | Limited to executable domains and the scope of the checks |
| SPICE | Questions and answers built from external documents | Document-grounded reference answer | Errors in the source, extraction or question generation |
| R-Few | Human examples and problems generated from them | Human labels and synthetic pseudo-labels | The two kinds of targets differ in reliability |
Self-play creates opportunities to recombine existing knowledge and practice existing capabilities. External information adds evidence and independent checks. Alongside generating more problems, a good design specifies how to verify answers, which problems the model keeps seeing and where to measure growth.
Part 11 extends the question to the training process itself. When a stronger AI designs and runs training for a smaller model, how much of that process can it automate?
Worked examples and code
Python calculation code · Calculation results
These cover R-Zero, AZR and SPICE rewards, pseudo-label errors in samples of 200, score changes on the training curve, and example program inputs and outputs.
#Further reading
| Resource | Connection |
|---|---|
| NTU Self-Improving lecture · Slides | The proposer-solver-verifier loop and external information |
| R-Zero: Self-Evolving Reasoning LLM from Zero Data | Alternating training, pseudo-labels and changes in difficulty and performance |
| Absolute Zero: Reinforced Self-play Reasoning with Zero Data | Problem generation and verification grounded in an execution environment |
| Self-Questioning Language Models | Another example of self-play between a questioner and an answerer |
| SPICE: Self-Play In Corpus Environments Improves Reasoning | Document-grounded problems and a variance-based curriculum |
| Guided Self-Evolving LLMs with Minimal Human Supervision | R-Few’s anchor examples and online curriculum |