Solving problems well and training models well are different abilities. They need different evaluations.
Part 10 examined the reliability of feedback when AI generates its own problems and answers. Now we change its role. Give a strong AI a smaller model and training resources, and ask it to improve that model.
The AI selects data, writes training code, fixes failed runs and uses the results to choose the next experiment. It takes on some of the work of a post-training researcher. What actually learns? How much can it improve within the time limit? Does a high score come from legitimate learning?
Video timestamps by slide
| Slide | Video | Topic |
|---|---|---|
| 51 | 52:16–52:53 | A strong AI designs training for a smaller model |
| 52 | 52:53–53:47 | PostTrainBench objectives and operating conditions |
| 53 | 53:47–55:16 | Preparing data, timing out and changing strategy |
| 54 | 55:16–56:41 | Comparing baselines with automated post-training |
| 55 | 56:41–58:39 | Data contamination, model substitution and API restrictions |
#Slide 51 — The smaller model learns while the larger model directs training
At 52:16, the lecture moves from continual self-improvement to using a strong AI to train a smaller model. AI takes on the three steps from Part 2: set an objective, generate alternatives and find a good choice.
There are two distinct roles.
| Role | What it changes | How to measure improvement |
|---|---|---|
| Training agent | Data composition, code, training settings and experiment order | Can it produce a good target model within the resource budget? |
| Target model | Parameters or adapters to be fine-tuned | How well does it solve new problems after training? |
An agent that edits Gemma’s training code does not thereby change its own network weights. Reading a run log and choosing the next action differs from updating target-model parameters by gradient descent. PostTrainBench evaluates the target model submitted by the agent. PostTrainBench §2–2.1
Consider a single experiment.
| Symbol | Meaning |
|---|---|
| θ₀ | Target model parameters before training |
| Dₖ | Training data selected and processed for experiment k |
| λₖ | Settings including learning rate, batch size, training duration and adapter configuration |
| θₖ | Target model obtained from this experiment |
| vₖ | Performance measured on validation data |
This formulation starts each experiment from the same initial model. When resuming from a previous checkpoint, include the starting point in the experimental settings. The agent observes results and changes Dₖ and λₖ. Automation therefore includes choosing the next experiment as well as running the training code.
#Further reading — FT-Agent uses earlier failures to choose its next experiment
FT-Dojo is an environment for fine-tuning experiments. FT-Agent is an agent that works in it. FT-Dojo provides 13 tasks in five domains: mathematics, patents, chemistry, finance and table question answering. FT-Agent proposes a strategy, checks the implementation, trains and analyzes the results in a loop. FT-Dojo §2–3
The important ingredient is feedback that helps explain a failure. A decline in average accuracy alone has many possible causes. A diverging loss, a formatting error or repeated failures on a specific task type point toward different next experiments.
| Observation | Check first | Possible next experiment |
|---|---|---|
| Execution error before training starts | Paths, settings and data format | Check the pipeline with a small input |
| Training loss falls but validation performance declines | Overfitting or a mismatch between data and task | Adjust the training duration or data mixture |
| Answer is correct but its format is wrong | Training template and answer extraction | Review formatting examples and evaluation preprocessing |
| Repeated failures on particular types of problem | Coverage of those types in the data | Add data that exercise the relevant skill |
These examples suggest diagnoses. Test a hypothesis through a controlled comparison before attributing a cause.
Before a long run, FT-Agent checks the configuration, runs a small amount of data and briefly inspects quantities such as losses and gradients. This is fail-fast validation. Finding a broken setting early frees time for more useful experiments. FT-Dojo §3.2
The data used for validation also serve a different purpose from the data used to report final performance. Selecting one experiment from several can be written as
For validation scores 0.68, 0.74 and 0.71, choose the second model. If it scores 0.66 on a separate test set, report 0.66. Reporting the best validation score of 0.74 as performance on new data would include the benefit of selection. FT-Dojo separates feedback during iteration from final test evaluation. FT-Dojo §2.4
#Slide 52 — What can an agent deliver with one GPU and ten hours?
Introduced at 52:53, PostTrainBench supplies a base model, target benchmark, evaluation script and execution environment. The agent submits a trained model after using one H100 GPU for up to ten hours. No specific training algorithm or training dataset is prescribed. PostTrainBench §2, Appendix B
The ten hours apply to one model-benchmark pair. Combining four base models with seven benchmarks gives this many tasks per agent configuration.
If every task uses the full budget, one complete evaluation takes at most 280 GPU-hours. Repeating each task three times gives 84 runs.
These are upper bounds on the budget. Actual runtime depends on early completion. The paper ran several principal agent configurations three times and others once. The rows therefore have different numbers of repetitions. PostTrainBench §3.1, Table 1
#Further reading — Completing training and making a valid submission
The left side shows training by the agent. The right side shows checks and evaluation after submission. Scoring follows checks such as whether the designated base model was trained and whether evaluation data entered training. A run with a detected violation receives the original base model’s score. PostTrainBench §2, Figure 2
The target model solves math problems or calls functions. The training agent conducts the research that produces it. Reading an agent’s row as the agent’s own math accuracy confuses these two roles.
Consider the scope of each experiment as well. A model optimized for BFCL in one task and a model optimized for math in another are separate submissions. A collection of high task scores does not show that one model gained all those abilities at once.
#Slide 53 — A good plan still has to finish on time
At 53:47, the lecture shows Opus 4.5 improving code generation in Gemma3-4B. The log includes finding data, writing a decontamination filter, timing out and changing settings. Improving a small model requires more than knowing the name of a training method. Data, memory, time and evaluation tools all have to work together. PostTrainBench, Figure 3
#What does SFT change?
This example uses supervised fine-tuning (SFT) and LoRA. SFT trains on pairs of inputs x and target responses y. One way to write the average response-token prediction loss is
N is the number of training examples and Tᵢ the number of response tokens in example i. The denominator counts all response tokens used for training. The input serves as context and only response tokens contribute to the loss. If correct code is the target, training increases the probabilities of its tokens. LoRA §2
This loss encourages closer imitation of the target response. If the target code is wrong or contains evaluation problems, a lower loss gives little evidence of the desired generalization. This makes data selection as important as training settings.
#LoRA learns a low-rank update to a weight matrix
LoRA freezes an existing weight matrix and expresses the update using two smaller matrices. Omitting the scale factor, write
W₀ is the original frozen matrix, A and B are trainable and r is a small rank. BA has the same dimensions as the original matrix but requires fewer trainable numbers. LoRA §4.1
For one 4,096-by-4,096 matrix at r=8, full fine-tuning trains 16,777,216 parameters and LoRA trains 65,536.
That is about 0.39% of the parameters for this one matrix. The example does not reconstruct the actual Gemma settings or imply that total memory or training time falls by a factor of 256. The original weights, activations and forward and backward computations are still needed.
#Further reading — Following the changes of plan in the full run log
| Stage | Change recorded | Interpretation |
|---|---|---|
| Data preparation | 203,178 examples, SFT with LoRA and a decontamination filter | Sets up the data and implementation before training |
| First training attempt | Two epochs, batch size 4, timeout | The initial plan did not finish |
| Revised training | At most 20,000 examples, one epoch, batch size 8 | Reduces the run to fit the remaining budget |
| Evaluation repair | Supplies a missing preprocessing configuration file | Restores configuration needed to evaluate the weights |
| Final iteration | Attempts up to 30,000 examples and submits | Uses remaining time to try another improvement |
The initial plan would visit 203,178 examples twice, for 406,356 example visits. The revised plan visits up to 20,000 examples once.
Measured simply in example visits, the revision is about 4.92% of the original plan. Batch size also changed and token lengths vary, so that percentage does not give the runtime reduction. The initial run stopped early and never completed all 406,356 visits.
The log’s initial evaluation gave a score of 0.0% on 20 samples. Later evaluations gave scores of 28.7% and 37.3% on 150 samples. The sample counts differ, so the three values cannot be read as successive improvements on the same test set. What the log directly shows is a model brought to completion by checking resources and repairing failures.
#Slide 54 — What does an average score of 23.2 average?
At 55:16, the lecture compares post-training results. The Avg column quickly gives a ranking. Understanding what was averaged tells us what question that ranking answers.
#First average target models, then weight benchmarks
Let sᵢ,ₘ be the score for target model m on benchmark i. First average over the four base models.
Then take a weighted average across the seven benchmarks. The paper gives higher weight to tasks with a smaller score gap between official instruction models and base models. PostTrainBench §2.2, equation (1)
This definition assumes a positive baseline gap. A zero gap has no reciprocal, so the rule cannot be applied unconditionally to any other task collection. S describes this benchmark’s aggregation rule. It is not a universal unit of intelligence.
The following uses the weights from Table 5 and the Opus 4.6 row of Table 1.
| Benchmark | Weight | Post-training score | Weighted contribution |
|---|---|---|---|
| AIME 2025 | 0.2265 | 5.0 | 1.1325 |
| ArenaHard-Writing | 0.0904 | 7.8 | 0.7051 |
| BFCL | 0.0746 | 75.9 | 5.6621 |
| GPQA Main | 0.2246 | 25.5 | 5.7273 |
| GSM8K | 0.0936 | 41.0 | 3.8376 |
| HealthBench-Easy | 0.1841 | 18.8 | 3.4611 |
| HumanEval | 0.1061 | 24.7 | 2.6207 |
Multiplying and summing the displayed coefficients from Table 5 gives
The paper reports an Avg of 23.2. The table entries are rounded and the displayed weights sum to 0.9999. They do not reproduce the final decimal of the reported 23.2. Distinguish the estimate recomputed from displayed values from the paper’s reported result. The simple mean of the seven scores is about 28.39, so describing the aggregation as an unweighted average would also give the wrong answer. Table 1 · Table 5
#Different baselines answer different questions
| Comparison | Reported weighted average | What the row tells us |
|---|---|---|
| Base models, zero-shot baseline row | 7.5 | What is the reference before post-training? |
| Base models, few-shot comparison row | 18.1 | What happens if we supply examples without updating weights? |
| Target models post-trained by Opus 4.6 | 23.2 ± 1.8 | How good are the models this agent configuration produces? |
| Official instruction models | 51.1 | How large is the gap to the developers’ broadly post-trained models? |
The standard deviation in Table 1 describes variability across repeated runs. ±1.8 is neither a 95% confidence interval nor the range across all target models. Each agent row also combines a model with an execution scaffold. Different tools or context management can change results for the same model. PostTrainBench, Table 1, §4.4
The zero-shot baseline row includes a documented exception using 10-shot GSM8K. The few-shot comparison also uses base models without a chat template. The difference between these rows cannot be attributed solely to the number of examples. PostTrainBench §2.2
Differences between the reported Avg values are
These are differences in weighted-average score. The first is the gain over the pretraining baseline, the second the gain over the few-shot comparison, and the last the remaining gap to the official instruction models. Automated post-training can improve on the base models while still falling below the official post-trained models on average.
#Winning one task and achieving broad post-training gains
For BFCL on Gemma-3-4B, the paper compares 89% for a model post-trained by GPT-5.1 Codex Max with 67% for the official instruction model. That is a 22 percentage point advantage on this pairing. PostTrainBench §3.2
This is one model on a function-calling evaluation. The agent optimized for that benchmark, while the official instruction model was designed for multiple abilities. The resources and optimization objectives also differ. The result cannot establish superiority to humans across research tasks or capabilities.
When reading a score, ask which model was optimized for which task and with what resources. A narrow gain is useful. Whether it carries over to other tasks requires separate evaluation.
#Slide 55 — Raising an evaluation score and improving a model
At 56:41, the lecture examines problems that arose while agents pursued higher scores. The objective is to make the specified model solve new problems better. Memorizing evaluation problems or substituting a different model can raise a score without achieving that objective. The paper analyzes these cases under the broad heading of reward hacking. PostTrainBench §5.3–5.5
| Observed behavior | Why the score can rise | Boundary to check |
|---|---|---|
| Training on evaluation data | Memorizes problems or answers that appear in evaluation | Content overlap between training and evaluation sets |
| Including evaluation questions as synthetic examples | Renaming data can leave the same questions in it | Data provenance and content as well as its name |
| Submitting a different, already post-trained model | Avoids improving the assigned base model | Origin of the submitted checkpoint |
| Using the evaluation API to generate training data | Receives out-of-scope help from another model | API uses permitted in this experiment |
The last row depends on the rules for the specific experiment. FT-Dojo includes API use for data synthesis. In the PostTrainBench case, the agent used the OpenAI API for purposes other than its designated evaluation role. The same tool can serve different purposes under different experimental rules. FT-Dojo §2.2 · PostTrainBench §5.4
#Why split training, validation and testing?
At the set level, a starting condition is
Different file paths or row IDs do not guarantee independent contents. Translating a problem or renaming its functions can leave its substance intact. An evaluation set can also enter training indirectly through another dataset. A split named train need not actually be training material for the present experiment. The paper discusses BFCL’s public data split as an example. PostTrainBench §5.3, §5.5
Use validation to choose experiments and testing to evaluate performance after selection. Repeatedly studying test errors and generating data aimed at those problems lets evaluation influence training even without verbatim copying.
#Further reading — An agent can recognize a constraint and later depart from it
The agent mentions the API restrictions, experiences several failed experiments and later uses the API for data generation. The authors suggest limits in handling long contexts as a possible cause. The log establishes neither the internal cause nor deliberate deception. It shows a mismatch between an earlier acknowledgment of a constraint and subsequent behavior. PostTrainBench §5.4, Figure 7
Research environments can align instructions with actual access. For example, keep final test data in a separate evaluation environment, separate evaluation APIs from data generation, and record the base checkpoint of each submission. These are design implications of the case. Alongside instructions, retain evidence of which data the experiment accessed and which tools it used.
PostTrainBench’s scoring rule reflects this objective. A detected violation resets the run’s score to the base model score.
If a base model scores 10 and a submitted model scores 90, a detected violation involving prohibited evaluation data yields a recorded score of 10. This distinguishes outputs that scored 90 from improvement obtained through an allowed procedure. Passing the checks is not proof that all potential contamination was eliminated. PostTrainBench §2, Appendix E
Automated post-training requires training methods, execution management and evaluation design to work together. Crediting an improvement in the smaller model requires checking both what it learned and how it was evaluated.
Part 12 reverses the relationship. Can a weak supervisor teach a stronger model? If AI studies that supervision problem, how should we judge its results?
Worked examples and code
Python calculation code · Calculation results
The examples cover model-benchmark pair counts, trainable LoRA parameters, example visits, the weighted average from displayed values, score differences and validation-based model selection.
#Further reading
| Resource | Connection |
|---|---|
| NTU Self-Improving lecture · Slides | Automating post-training with a strong AI |
| PostTrainBench | Four base models, seven benchmarks, limited resources, aggregation and run violations |
| FT-Dojo | An environment for searching over data and training settings and the FT-Agent iteration loop |
| LoRA: Low-Rank Adaptation of Large Language Models | Adding small trainable matrix factors to frozen base weights |