Skip to main content
#self-improvement#knowledge-distillation#self-training#ntu#study

Self-Improving AI 02 — Who sets the learning target?

Loss functions, knowledge distillation and self-generated pseudo-labels illustrate when a model’s own outputs provide a new learning signal.

Self-Improving AI 01 — What Changes When AI Gets Better? distinguished the target of change, the scope of automation, and the evaluation criteria. We now follow the process that trains model parameters.

#Slide 6 — How does AI learn?

The slide points to a 2025 introductory machine learning lecture that reviews the basics of training.
NTU slide 6 — The basic structure of machine learning.Hung-yi Lee · NTU · Original source

Original slide 6 · Video at approximately 03:37–04:01

This transition slide reviews machine learning before introducing self-improvement. The central thumbnail points to the instructor’s fifth lecture in the 2025 introductory course.

To understand self-improvement, first identify what changes in ordinary training. For illustration, write a model as follows.

y=fθ(x).y=f_\theta(x).
  • xx: the input.
  • fθf_\theta: the model specified by parameters θ\theta.
  • yy: the model’s output.
  • θ\theta: trainable values, such as weights and biases.

In estimating human joint positions from an image, for example, xx is the image and yy gives the joint coordinates. A language model takes an input context, computes a distribution over the next token, and builds a sentence by choosing successive tokens from these distributions.

Inference computes an output with a given model. Parameter learning changes θ\theta according to a specified criterion. From this slide onward we focus on the latter.

The question about self-improvement now becomes more precise.

Can AI produce the training data or targets that change a model? If so, how can we judge the quality of those targets?

Slide 7 breaks the question into three parts.

#Slide 7 — The objective, candidates, and optimization

Three boxes distinguish the goal, the possible choices, and the method for selecting the best choice. The goal is highlighted.
NTU slide 7 — The goal, possible choices, and optimization procedure.Hung-yi Lee · NTU · Original source

Original slide 7 · Video at approximately 04:01–04:56

The diagram distinguishes what we seek, the available choices, and the procedure for choosing a good one. A red border around the first blue box marks the particular focus of this lecture.

Here is that distinction in machine learning terms.

Question on the slideMachine learning conceptJoint position estimation example
What are we looking for?Problem definition and criteria for a good outputOutput coordinates close to the ground-truth joints in the image
What choices are available?Hypothesis space, the set of functions the model can expressThe selected network architecture and possible parameter values
How do we select a good choice?OptimizationCompute a loss and update parameters using gradient descent or another optimizer

#Model architecture and training objectives play different roles

Even with the same network architecture, different values of θ\theta produce different functions. Write the candidate set as follows.

H={fθ:θ∈Θ}.\mathcal H=\{f_\theta:\theta\in\Theta\}.

Here Θ\Theta is the set of allowed parameter values, and H\mathcal H is the set of functions those values can express.

A loss function L(θ)L(\theta) evaluates how well a candidate model performs. The ideal objective of finding the best parameters under this criterion can be written as follows.

θ⋆∈arg⁡min⁡θ∈ΘL(θ).\theta^\star\in\arg\min_{\theta\in\Theta}L(\theta).

A training algorithm searches candidates in pursuit of this objective. The basic gradient descent update is

θt+1=θt−η∇θL(θt).\theta_{t+1}=\theta_t-\eta\nabla_\theta L(\theta_t).
  • η\eta: the learning rate, which sets the step size.
  • ∇θL\nabla_\theta L: the gradient, pointing in the direction of increasing loss at the current position.
  • The minus sign specifies a step in the opposite direction to reduce the loss.

This equation does not guarantee a global minimum for every neural network. Initialization, data, model architecture, and optimizer settings all influence the solution. A good candidate on training data also requires separate evaluation for generalization to new data. Supplement: Stanford CS229 notes on supervised learning and gradient descent

#Supplementary lecture · Stanford CS229 — What value does optimization change?

The gradient descent equation updates parameter theta using learning rate alpha and the gradient to reduce cost J.
Stanford CS229, PDF page 9 — Updating parameters by gradient descent.Nandita Bhaskhar / Stanford CS229 · Original source

On the Stanford slide, J(θ)J(\theta) corresponds to our total loss L(θ)L(\theta), and learning rate α\alpha to η\eta. The update changes parameter θ\theta. The discussion of stochastic gradient descent (SGD) below it distinguishes which training examples contribute to a single update.

Compare the equation with the first box on the NTU slide. If a different teacher supplies different target labels, the same gradient descent algorithm may receive a different loss and gradient. Examine both the optimizer and the target it receives.

#Why highlight the first box?

In self-improvement, the model architecture is only one possible target of change. AI can also generate or propose the data, answers, rewards, and losses that define a good answer.

For example, even with the same model and optimizer, the following training procedures differ.

  • Use human-reviewed joint coordinates as targets.
  • Use joint coordinates predicted by an existing model as targets.

The search procedure may stay the same while changing what the model learns to approximate. This distinction runs through slides 8–10.

#Slide 8 — The difference between the output and the target provides a learning signal

Given input x, compute output y and compare it with training target y-hat to obtain the overall loss L.
NTU slide 8 — A loss comparing the model output with the training target.Hung-yi Lee · NTU · Original source

Original slide 8 · Video at approximately 04:56–07:02

In this figure, yy is the model output, and y^\hat y is the correct answer or training target. Other texts use y^\hat y for a prediction, so check the role of each symbol here.

#Reading the diagram from left to right

The same model maps inputs x1,…,xNx^1,\ldots,x^N to outputs y1,…,yNy^1,\ldots,y^N. Superscript nn indexes examples rather than raising a value to a power.

yn=fθ(xn).y^n=f_\theta(x^n).

Compute an individual loss from the discrepancy between each output and its corresponding target.

ℓn=d(yn,y^n).\ell^n=d(y^n,\hat y^n).

Here dd evaluates the discrepancy. It need not be a mathematical metric satisfying symmetry and the triangle inequality.

Denote the overall loss aggregating individual losses by LL.

L(θ)=∑n=1Nℓn=∑n=1Nd ⁣(fθ(xn),y^n).L(\theta)=\sum_{n=1}^{N}\ell^n = \sum_{n=1}^{N}d\!\left(f_\theta(x^n),\hat y^n\right).

Define a good output for an individual example, then adjust parameters to satisfy that criterion across the data.

#A numerical regression loss example

Consider a hypothetical distance prediction task measured in meters, with squared error as the individual loss.

ℓ=(y−y^)2.\ell=(y-\hat y)^2.
ExampleTarget y^\hat yModel A outputA squared errorModel B outputB squared error
11.01.20.041.10.01
22.01.90.012.50.25
Total0.050.26

B is more accurate on the first example. A does better under the criterion aggregated across both.

For mean squared error (MSE), divide the sum by NN. A’s MSE is 0.025 and B’s is 0.13. On the same fixed data, sums and means rank models identically but give gradients of different magnitudes. Read the learning rate alongside the loss definition in implementations.

#Cross-entropy for classification

If pθ(y^∣x)p_\theta(\hat y\mid x) denotes the probability assigned to the correct class, a common loss is

ℓCE=−log⁡pθ(y^∣x).\ell_{\mathrm{CE}}=-\log p_\theta(\hat y\mid x).

In a hypothetical example using natural logarithms, a probability of 0.8 for the correct class produces a loss of approximately 0.223, while 0.2 yields about 1.609. Unlike accuracy, which counts correct final choices, the loss responds to changes in the probability assigned to the correct class. Supplement: classification and logistic regression in CS229

So far the target has been given. Next we change who fills in the answer box at the bottom of the diagram.

#Slide 9 — Knowledge distillation from a strong AI’s answers

The slide introduces knowledge distillation, in which another strong AI supplies the training targets.
NTU slide 9 — Knowledge distillation using another AI's answers as targets.Hung-yi Lee · NTU · Original source

Original slide 9 · Video at approximately 07:02–08:03

Compared with the previous slide, a strong AI with a different appearance now stands beside the lower answer box. It supplies the training targets. The lecture introduces this as knowledge distillation.

The model that teaches is the teacher, and the one that learns is the student. Training moves the student’s outputs toward the teacher’s targets.

#Transferring an answer or a probability distribution

Start with the answer box on the slide. The teacher produces answer y~\tilde y to question xx. We introduce y~\tilde y to distinguish the AI-generated target. It occupies the position labeled y^\hat y on the slide.

One loss for a language model learning from that answer is

Lanswer(θ)=−1N∑n=1Nlog⁡pθ(y~n∣xn).L_{\mathrm{answer}}(\theta) = -\frac1N\sum_{n=1}^{N}\log p_\theta(\tilde y^n\mid x^n).

The probability of an entire sentence is a product of token-level conditional probabilities. Taking its negative logarithm gives a sum of token losses.

Another approach uses the teacher’s output distribution as the target. For class cc, let qϕ(c∣x)q_\phi(c\mid x) be the teacher distribution and pθ(c∣x)p_\theta(c\mid x) the student distribution.

Lsoft(θ)=−∑cqϕ(c∣x)log⁡pθ(c∣x).L_{\mathrm{soft}}(\theta) = -\sum_c q_\phi(c\mid x)\log p_\theta(c\mid x).

The teacher distribution is a fixed target during the student update. Both approaches use the teacher’s output, but they communicate different information. Hinton and colleagues’ 2015 paper describes distillation using output distributions and temperature. Original paper, section 2

#Supplementary lecture · Lambert — Fix the teacher distribution and update the student

A KL loss compares the distributions of a frozen teacher and a trainable student on the same input. Gradients flow only to the student.
Nathan Lambert, Lecture 7, PDF page 14 — Knowledge distillation with a frozen teacher and a trainable student.Nathan Lambert / RLHF Book · Original source

The upper teacher is labeled Frozen and the lower student Trained. Both receive the same input, but the gradient along the dashed arrow changes only the student. Updating the teacher’s target would move what the student is trying to match, so the diagram holds the teacher distribution fixed.

Temperature τ\tau in the middle adjusts how sharp the distribution is when logits become probabilities. The setting τ>1\tau>1 reveals information beyond the largest probability. This differs from supplying a completed answer as a target. The student matches the teacher’s distribution across possibilities.

#What extra information does the distribution retain?

Consider a hypothetical three-class example.

Target representationClass AClass BClass C
Target retaining only the most probable class1.00.00.0
Teacher’s soft target0.70.20.1

Both targets identify A as the most likely class. The soft target also records that the teacher considered B a more plausible alternative than C. The student can learn these relative relationships.

The teacher distribution can contain errors. Receiving a detailed distribution does not by itself make it a correct target.

#Must a “strong teacher” be a larger model?

Distillation extends beyond compressing a large model into a small one. What matters is what supplies a better learning signal. Noisy Student studies students of equal or greater size and introduces noise during training. Distinguish model size from the quality of the teacher’s signal. Noisy Student, CVPR 2020

A different teacher supplies the target on this slide. Slide 10 puts the model being trained in that teacher’s place.

#Slide 10 — Can a model teach itself with its own answer?

The same model as the one being trained supplies pseudo-answers.
NTU slide 10 — Using self-generated answers as training targets.Hung-yi Lee · NTU · Original source

Original slide 10 · Video at approximately 08:03–08:31

The most visible change is that the teacher robot below now looks like the model being trained above.

The slide asks whether self-generated pseudo-answers can serve as training targets. A pseudo-answer is a target assigned for training. Its correctness has yet to be established.

When a model learns from its own output, first ask which form of output becomes the target.

#Case A. Matching the model’s own probability distribution directly

Let θ0\theta_0 be the model’s current weights. Use the distribution obtained on the same input as a fixed target.

q(c∣x)=stopgrad⁡ ⁣[pθ0(c∣x)].q(c\mid x) = \operatorname{stopgrad}\!\left[p_{\theta_0}(c\mid x)\right].

Stop-gradient means derivatives do not flow through the target-producing branch. Suppose the student starts at θ=θ0\theta=\theta_0 and that identical inputs, temperatures, and execution conditions produce exactly identical distributions.

At temperature 1, the gradient of cross-entropy with respect to student logit zcz_c is

∂Lsoft∂zc=pθ(c∣x)−q(c∣x).\frac{\partial L_{\mathrm{soft}}}{\partial z_c} = p_\theta(c\mid x)-q(c\mid x).

When both distributions agree,

∇θLsoft∣θ=θ0=0.\nabla_\theta L_{\mathrm{soft}}\big|_{\theta=\theta_0}=0.

This distillation loss itself supplies no learning direction. The result follows by substituting identical distributions into the logit gradient. Hinton et al., 2015, section 2

All assumptions matter. Dropout or input augmentation may make the distributions differ. A separate ground-truth loss or regularization term can also change the result. Optimizer momentum or weight decay may still move parameters, so the claim here concerns only the gradient of this distillation loss.

Also, zero gradient and zero loss are different conditions.

CE⁡(q,q)=H(q),DKL(q∥q)=0.\operatorname{CE}(q,q)=H(q), \qquad D_{\mathrm{KL}}(q\|q)=0.

With identical distributions, their cross-entropy still equals the distribution’s entropy. For example, q=(0.8,0.2)q=(0.8,0.2) gives cross-entropy of approximately 0.5004. That remaining value does not give the student a further direction to move.

#Supplementary lecture · Lambert — What decreases in cross-entropy?

Cross-entropy between teacher q and student p expands into the teacher's entropy H(q) plus KL divergence.
Nathan Lambert, Lecture 7, PDF page 21 — Decomposing cross-entropy into entropy and KL divergence.Nathan Lambert / RLHF Book · Original source

The slide’s H(q,p)H(q,p) is our CE⁡(q,p)\operatorname{CE}(q,p). The last line separates it into the teacher’s uncertainty H(q)H(q) and KL divergence measuring the difference between the distributions. With a fixed teacher, only the KL term can change through student updates.

The residual H(q)H(q) after the distributions agree is therefore not unresolved student error. Treating KL as the only reducible term requires a fixed teacher. Recomputing the teacher with the same parameters and differentiating through that branch would optimize a different objective.

#Case B. Turning a prediction into one hard label

Consider a hypothetical defect classifier that predicts a defect probability of 0.8. Based on this prediction, assign the pseudo-label “defective,” or y~=1\tilde y=1.

In binary cross-entropy, the gradient with respect to defect logit zz is

∂Lhard∂z=p−y~=0.8−1=−0.2.\frac{\partial L_{\mathrm{hard}}}{\partial z} = p-\tilde y = 0.8-1 = -0.2.

Gradient descent increases the logit. In a calculation directly updating one logit, with z=log⁡4z=\log 4 and learning rate η=0.1\eta=0.1,

znew=log⁡4+0.02,σ(znew)≈0.8032.z_{\mathrm{new}}=\log4+0.02, \qquad \sigma(z_{\mathrm{new}})\approx0.8032.

A learning signal arises even though it originated from the same model’s prediction. The procedure changed the probability target 0.80.8 into hard label 11.

If the item is actually normal, the model has become more confident in an incorrect judgment. An increase in predicted probability and an increase in actual accuracy require separate checks. A model score of 0.8 does not guarantee 80% empirical correctness.

This calculation illustrates the movement of one logit. Real networks share weights across inputs, so we cannot infer that outputs for all examples change the same way.

#In depth. What if we train on every randomly generated answer?

Making the most likely answer a hard label differs from randomly sampling an answer from the model distribution. Suppose we fit all sampled answers by maximum likelihood without selection.

y~∼pθ0(⋅∣x)\tilde y\sim p_{\theta_0}(\cdot\mid x)

Under that sampling distribution, the expected gradient at the starting point is

Ey~∼pθ0[−∇θlog⁡pθ(y~∣x)]θ=θ0=0\mathbb E_{\tilde y\sim p_{\theta_0}} \left[ -\nabla_\theta\log p_\theta(\tilde y\mid x) \right]_{\theta=\theta_0} =0

This follows from probabilities summing to one when the output space is fixed and differentiation can be exchanged with summation.

A finite sample can give a nonzero gradient. Movement due to sampling alone, however, is insufficient evidence of a signal pointing toward the correct answer. This observation does not rule out all effects over multiple actual training rounds.

#What changes in actual self-training?

Research using self-generated data changes the target-generation procedure, the data, or the conditions under which the student learns.

What changesWhy the learning signal changes
Filter generations using answers, tests, or rewardsThe distribution of outputs retained for training changes
Augment student inputsThe student learns the teacher’s target across different observations of the same content
Add noise to the studentThe student runs under different conditions from the teacher that produced the targets
Add new unlabeled inputsTraining constrains input regions previously unseen
Change the student’s initialization, capacity, or training procedureThe same signal can be represented by different functions and optimization paths

Three original studies illustrate these possibilities.

Noisy Student, CVPR 2020. A teacher assigns pseudo-labels to unlabeled images, and a student trains on both labeled and pseudo-labeled data. The student uses data augmentation and model noise and may be as large as or larger than the teacher. Original paper

Supplementary paper · Noisy Student
A teacher trains on labeled data, creates pseudo-labels for unlabeled data, and trains a noisy student that becomes the next teacher.
Noisy Student, Figure 1 — Generating pseudo-labels and training a noisy student.Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le (2020). Example images: ImageNet. · Original source

The upper-left teacher starts with labeled data and then labels unlabeled inputs. The student below uses both kinds of data. Data augmentation, dropout, and stochastic depth, marked in blue, apply during student training. The teacher turns these sources of noise off when generating pseudo-labels.

The figure differs from matching the same distribution on identical inputs under identical execution conditions: it adds new inputs and changes the student’s training conditions. The final arrow reuses the trained student as the next teacher. That arrow alone does not establish improvement, which requires separate evaluation.

STaR, NeurIPS 2022. STaR trains on generated rationales that reach a correct answer. After a failure, it regenerates a rationale conditioned on the correct answer. Problems, answers, and initial worked examples enter the procedure. Original paper

ReST, August 2023. ReST separates Grow, which generates data, from Improve, which trains on data filtered by reward. The paper considers negative log-likelihood and several offline reinforcement learning losses, so the method extends beyond a single loss. Original paper, section 3

None of these ingredients guarantees improvement. Incorrect pseudo-labels, inaccurate selection criteria, or augmentations that change a label’s meaning can cause problems. Check the direction of change and the outcome on new data.

The question on slide 10 can now be phrased more precisely.

What new learning signal was constructed from the model’s outputs, and how did the experiment establish that the signal improves the desired capability?

Next, Self-Improving AI 03 — When Does Self-Correction Become Learning? covers self-correction beginning on slide 11. It distinguishes revising an answer from using the revised answer as training data to update parameters.

#Further reading

Lecture PPTX · PDF · Lecture video

  1. Hung-yi Lee, official NTU ML 2026 course. Self-Improving, Part 1, PDF pages 6–10: learning targets and AI-generated answers.
  2. Stanford CS229 lecture notes. Foundations of supervised learning, gradient descent, and classification losses.
  3. Nandita Bhaskhar, Stanford CS229 Midterm Review. May 6, 2022. PDF page 9: parameter updates in gradient descent.
  4. Hinton, Vinyals, Dean, Distilling the Knowledge in a Neural Network. March 9, 2015. Knowledge distillation using probability distributions.
  5. Xie et al., Self-training with Noisy Student improves ImageNet classification. CVPR 2020, arXiv v4 (June 19, 2020). Figure 1 on PDF page 2: iterative teacher–student training.
  6. Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022. Filtering and learning from generated rationales.
  7. Gulcehre et al., Reinforced Self-Training (ReST) for Language Modeling. arXiv v2 (August 21, 2023). Iterating generation, selection, and training.
  8. Nathan Lambert, Post-training Course, Lecture 7 · Official PDF. PDF page 14: teacher–student distillation. Page 21: the relationship between cross-entropy and KL divergence.

Connect