Skip to main content
#self-improvement#test-time-training#test-time-adaptation#fine-tuning#ntu

Self-Improving AI 19 — What should a model keep after adapting to one input?

Pages 26–30 of NTU’s second lecture compare resetting a model for each input with carrying its updates forward. Rotation prediction, noisy speech recognition, and experience replay show how to measure both adaptation and forgetting.

The lights dim above a factory inspection camera. You adjust the vision model on the first frame, and it can pick out the part again. Which model should read the second frame? The recent adjustment could help if the lighting stays the same. If the next frame comes from a brightly lit camera on the adjacent line, that same adjustment could put the model at a disadvantage.

Episode 18 ended just as the goal changed. Pages 26–30 of the second lecture ask what to keep at that moment. Episode 7 introduced test-time training (TTT). Here we follow an update from one input to the next, then look at a speech recognition experiment that measured the consequences.

Worked examples Gradient conflict · Rotation gradient · Brightness correction · Speech shift · Forgetting

Find each topic in the lecture
Original slideNTU lecture, May 22, 2026What to listen for
p. 2626:56When a tank must become an aircraft, is the radar worth keeping?
p. 2727:55Observe the input, update the model, then make the prediction.
p. 2828:46The updated parameters become the starting point for the next input.
p. 2929:07Lee points to a previous lecture and a study that combines the two approaches.
p. 3029:37What happens to an old skill after an update to the weights or the harness?

The timestamps mark the start of the relevant discussion.

#Part II, p. 26 — Could an aircraft use the tank’s radar?

A simple robot develops into a tank. The upper branch returns to the original robot, while the lower branch carries all the tank equipment forward.
Part II, p. 26 · Two choices when the goal changes. Discard all the equipment or take it all along.Hung-yi Lee / NTU ML 2026 · Original source

The robot on the left changes to meet goal H. In the middle, a new goal H′ splits its path. The robot above starts over with its original body. The robot below keeps the heavy equipment. Lee asks us to consider the radar. It can still serve the aircraft, even though the tracks would only weigh it down. Lee’s radar example, 27:29

The lecture moves from this analogy to speech recognition. It helps to ask what changed in each case. The robot’s criterion for success changes from traveling on the ground to flying. For speech recognition, success still means transcribing the words correctly. The background noise reaching the microphone changes the input distribution. We can therefore compare performance before and after adaptation using the same metric.

#Worked example · When does learning a new task raise the old task’s loss?

Translate the radar example into two weights. Keep a few examples of the earlier task A and call the gradient of their mean loss g_A. Call the gradient for the new task B g_B. A gradient points toward increasing loss, so an optimizer subtracts a chosen direction d scaled by the learning rate η. For a small step, a dot product approximates the change in A’s loss.

ΔLA≈gA⊤(−ηd)=−η gA⊤d\Delta L_A\approx g_A^\top(-\eta d)=-\eta\,g_A^\top d

Take g_A = (1, 0), g_B = (−1, 2), and η = 0.1. Following B’s gradient increases the first weight by 0.1. That is precisely the weight whose increase raises A’s loss.

gA⊤gB=1(−1)+0(2)=−1,ΔLA≈−0.1(−1)=+0.1g_A^\top g_B=1(-1)+0(2)=-1,\qquad \Delta L_A\approx-0.1(-1)=+0.1

For A’s loss to stay level or fall to first order, the first coordinate of d must be at least zero. The closest vector to (−1, 2) satisfying that constraint is (0, 2). It removes the conflicting coordinate and keeps the second one.

gA⊤(0,2)=0,ΔLB≈−0.1 [(−1)(0)+2(2)]=−0.4g_A^\top(0,2)=0,\qquad \Delta L_B\approx-0.1\,[(-1)(0)+2(2)]=-0.4

A’s estimated change is zero, while B’s loss falls by about 0.4. This is a two-coordinate example of gradient projection in Gradient Episodic Memory (GEM). GEM checks a constraint for each earlier task using examples stored in memory. Those losses and full-task accuracy measure different things. Evaluate on held-out examples to see which skills survived. Lopez-Paz and Ranzato, NeurIPS 2017, Eqs. 7–8

#Part II, p. 27 — All three rows start with the same blue robot

The model on the left is blue for all three inputs. Each row updates that model on the current input and produces its own prediction.
Part II, p. 27 · Episodic adaptation starts each input from the same reference model.Hung-yi Lee / NTU ML 2026 · Original source

Read down the left side first. The inputs differ, but the model starts blue every time. Now read across any row: input → reference model → update → adapted model → output. The weights adjusted for the first input do not pass into the second row. Let θ₀ denote the reference weights, xₜ the current input, and A the adaptation procedure.

θ~t=A(θ0,xt),y^t=fθ~t(xt)\tilde\theta_t=A(\theta_0,x_t),\qquad \hat y_t=f_{\tilde\theta_t}(x_t)

What can the model learn without the correct class label? In the vision experiments by Sun and colleagues, the program rotates an image by 0°, 90°, 180°, or 270°. It knows the angle it applied even when it does not know what object is in the photo. During training, object classification and rotation prediction share a feature extractor. At test time, the rotation loss updates the shared features. The object classifier then uses those features to read the original image. Imagine photographing a circular washer. If the image looks nearly identical under rotation, the angle provides little information. Which features the auxiliary task shares with the task we care about matters. Sun et al., ICML 2020, §2

#Worked example · Differentiate a rotation loss without knowing the object class

The program rotates the image by 90°. The model assigns one score to each of four angles. For a calculation we can do by hand, let z be the score (logit) of the known angle and hold the other three scores at zero. The 3 in the softmax denominator comes from three contributions of exp(0) = 1. Let p be the probability of the known rotation and let L be its negative natural log. Stanford CS231n, Softmax classifier

p(z)=ezez+3,L(z)=−ln⁡p(z)=ln⁡(ez+3)−zp(z)=\frac{e^z}{e^z+3},\qquad L(z)=-\ln p(z)=\ln(e^z+3)-z

Initially z = 0. All four angles are equally likely, so p = 1/4 and L = ln 4 ≈ 1.3863. Differentiate the logarithm and the final term −z separately.

dLdz=ezez+3−1=p(z)−1\frac{dL}{dz}=\frac{e^z}{e^z+3}-1=p(z)-1

The slope is 0.25 − 1 = −0.75. With a learning rate of 0.4, subtracting that negative slope increases the score of the known angle.

z1=0−0.4 (0.25−1)=0.3,p(z1)=e0.3e0.3+3≈0.3103z_1=0-0.4\,(0.25-1)=0.3,\qquad p(z_1)=\frac{e^{0.3}}{e^{0.3}+3}\approx0.3103

The new loss is −ln p(z₁) ≈ 1.1701. This verifies a decrease in the loss for the known rotation. In TTT the chain rule carries this gradient back into the shared feature extractor. The calculation used no object label. Whether defect classification improved still requires a separate measurement. Sun et al., ICML 2020, Eqs. 2–3

#Supplementary lecture · What changes when a model spends longer on an answer?

Two graphs from Lambert’s lecture relate math performance to compute spent during training and compute spent on each answer.
Lambert, Lecture 5, p. 7 · The horizontal axes distinguish when compute is spent.Nathan Lambert / Post-Training Course · plots from OpenAI (2024) · Original source

Nathan Lambert’s two plots have different horizontal axes. The left varies the compute spent training the policy. The right gives an already trained reasoning model more compute to answer a problem. In the recording, Lambert discusses the right-hand plot first, then returns to the left between 5:51 and 6:33. On the next slide, a sum-of-primes example shows the model using tokens to check candidate numbers and its arithmetic. The weights can stay the same for the next request. By comparison, implementing the update arrow on NTU’s page 27 requires a signal from the current input to change some parameters. Tracking generated tokens and weight updates separately tells us where computation went. Lambert, Lecture 5, slides 7–8 · Lecture video, 5:51

#Part II, p. 28 — Look at the model that starts the next row

The orange result of row one starts row two. The purple result of row two starts row three.
Part II, p. 28 · Continual adaptation passes the updated weights on to the next input.Hung-yi Lee / NTU ML 2026 · Original source

Compare the colors with page 27. The orange robot at the end of the first row reappears at the start of the second. The purple robot makes the same transition one row later. The adaptation procedure A is unchanged. Its first argument now holds the previous weights.

θt=A(θt−1,xt),y^t=fθt(xt)\theta_t=A(\theta_{t-1},x_t),\qquad \hat y_t=f_{\theta_t}(x_t)

Let’s put numbers on the camera example. Imagine there is a brightness reference chart in view, and it gives us a target correction zₜ. The baseline correction is zero. On each frame, we move the current correction b halfway to the target. These are illustrative values.

bnew=bstart+12(zt−bstart)b_{\mathrm{new}}=b_{\mathrm{start}}+\tfrac12(z_t-b_{\mathrm{start}})
FrameTarget correctionValue after starting from zeroValue after starting from the previous correction
First frame from the dim camera422
Second frame from the same camera423
First frame from the bright camera001.5

On the second frame, the remaining errors are 2 and 1, respectively. Keeping the previous correction helped. On the third, the errors are 0 and 1.5, and the ranking reverses. Both procedures took one update per frame. The starting state explains the difference.

#Worked example · When does the correction error halve on each frame?

Replace the factor 1/2 in the table with a learning rate η. On the carry-forward path, bₜ = bₜ₋₁ + η(z − bₜ₋₁). Subtract the same target z from both sides and factor out the common term. As long as the lighting stays constant for t frames, we can substitute this relation t times.

bt−z=(1−η)(bt−1−z),bt−z=(1−η)t(b0−z)b_t-z=(1-\eta)(b_{t-1}-z),\qquad b_t-z=(1-\eta)^t(b_0-z)

Use target 4, initial value 0, and η = 0.5. The correction reaches 3 after two frames. On the third frame the target changes, so we stop using the closed form derived for target 4 and substitute the new target 0 into the one-step update.

b2=4+(0.5)2(0−4)=3,b3=3+0.5(0−3)=1.5b_2=4+(0.5)^2(0-4)=3,\qquad b_3=3+0.5(0-3)=1.5

With a fresh start on every frame, the first relation is applied only once each time. At the switch to the brighter camera, the reference value happens to equal the new target, giving zero error. If the darker scene continues instead, carry-forward halves its remaining error each time. That accounts for the change in ranking in the table.

#Supplementary paper · Which bars show the effect of carrying an image update forward?

For each corruption on CIFAR-10-C, Sun and colleagues plot classification error for standard training, joint training, TTT, and online TTT.
Sun et al., Figure 1 · Compare standard and online TTT within each corruption type.Sun et al. / ICML 2020 · Original source

Choose one corruption on the horizontal axis and read the four bars in its group. The vertical axis shows classification error, so lower is better. The first two models were trained on object classification alone or jointly on classification and rotation prediction. The final two adjust features using test images. For the highest corruption severity, level 5, standard TTT resets and takes ten gradient steps per image. TTT-Online carries the updated state forward and takes one step per new image. The two bars differ in update budget as well as starting state. Comparing them tells us how the two procedures performed under the same type of corruption. To isolate the effect of retaining state, we would also need a comparison with matched step counts. To measure residual error immediately after switching cameras, we would need the order of corruptions and measurements taken just after the switch. Sun et al., Figure 1 and §3.1

#Part II, p. 29 — One model for this utterance, another to prepare for the next

Portraits of Wei-Ping Huang and Guan-Ting Lin appear beside a link to the noisy-speech paper and a diagram of standard TTT from the 2025 lecture.
Part II, p. 29 · The earlier lecture’s per-input TTT diagram leads into the speech recognition study.Hung-yi Lee / NTU ML 2026 · Original source

The diagram on the right starts each input from the same model. In the paper linked on the left, Lin, Huang, and Lee ask a follow-up question: How much of a quick adjustment made to hear one sentence should the next sentence inherit? In Lecture 8 from 2025, Lee distinguishes fast adaptation to one utterance from slower learning across several utterances. The time around 1 hour and 54 minutes printed on the slide belongs to that earlier recording. Lecture 8, 1:54:32 · Fast and slow updates, 1:56:16

#Supplementary paper · Two update speeds in DSUTA

Figure 2 compares resetting for each utterance, carrying every update forward, and separating fast adaptation from slow updates.
Lin et al., Figure 2 · Trace which model each of the three methods passes to the next utterance.Lin, Huang & Lee / EMNLP 2024 · Original source

Dynamic SUTA (DSUTA) starts each utterance from a copy of the current reference weights. The fast update adjusts that copy several times on the current audio and uses it to produce a transcript. The slow update collects recent utterances in a buffer and updates the reference weights using their average adaptation loss. The next utterance starts from those reference weights. A final adjustment made for one unusual pronunciation can thus remain local to the utterance. The experiment uses reference transcripts to evaluate performance. Adaptation uses unlabeled speech and a loss computed from model predictions. Paper, §§3.1–3.2 and Algorithm 1

How does the model detect a change in noise? The dynamic reset procedure tracks how well the recent speech fits the model adapted to the earlier noise environment. Its indicator compares losses with those of the pretrained model. The procedure checks for a departure from the earlier range across multiple utterances before resetting the reference weights. It receives neither reference transcripts nor the true times at which the noise changes. Paper, §3.3

#Worked example · Detect a possible change without a reference transcript

In the DSUTA paper, L is a speech adaptation loss computed without labels. Evaluate the same utterance xᵢ with a reference model from the earlier environment, φ_D, and with the pretrained model, φ_pre. The paper calls the difference the Loss Improvement Index (LII). The order of subtraction matters. In the example below, −0.06 means the earlier environment’s model achieved a loss lower by 0.06.

Ii=L(ϕD,xi)−L(ϕpre,xi),I1=0.64−0.70=−0.06I_i=L(\phi_D,x_i)-L(\phi_{\rm pre},x_i),\qquad I_1=0.64-0.70=-0.06

Suppose the four latest utterances give indices −0.06, −0.02, 0.00, and +0.04. The mean is still negative. To tell whether conditions have deteriorated, compare it with the mean recorded in the previous environment.

Iˉ=−0.06−0.02+0.00+0.044=−0.01\bar I=\frac{-0.06-0.02+0.00+0.04}{4}=-0.01

Let the previous mean μ be −0.10 and its standard deviation σ be 0.08. For M = 4 utterances, the calculation uses a standard error of 0.08/√4. Substituting into the paper’s buffer-average rule gives the following value.

Z=Iˉ−μσ/M=−0.01−(−0.10)0.08/4=0.090.04=2.25Z=\frac{\bar I-\mu}{\sigma/\sqrt M}=\frac{-0.01-(-0.10)}{0.08/\sqrt4}=\frac{0.09}{0.04}=2.25

The statistic 2.25 exceeds the paper’s threshold of 2. The adapted reference still has a small average advantage over the pretrained model, but much of the advantage it used to have has disappeared. This establishes one threshold crossing. The implementation also uses a patience condition to handle transient changes. Strong correlations between successive utterances can make the standard error of their mean larger than expected. The statistic itself gives no probability that the noise changed. Lin et al., §3.3 and Appendix Algorithm 2

Word error rate (WER) counts the substitutions S, deletions D, and insertions I needed to match a reference transcript, then divides by the number of reference words N. Two substitutions, one deletion, and one insertion against a 20-word reference give a WER of 20%.

WER=100 S+D+IN  %\mathrm{WER}=100\,\frac{S+D+I}{N}\;\%

The denominator counts the 20 reference words. A word added by the model increases only the numerator.

100 2+1+120=20  %100\,\frac{2+1+1}{20}=20\;\%
Method in Table 2State retained for the next utteranceMD-Hard WER ↓
Source modelFixed pretrained weights74.6%
SUTAReturn to pretrained weights for each utterance60.4%
CSUTAWeights adapted on the preceding utterance83.6%
DSUTAReference weights updated on batches of recent utterances45.6%
DSUTA + Fixed resetReset reference weights at fixed intervals49.4%
DSUTA + Dynamic resetReset when the loss indicator signals an environment change39.8%

These are means over three runs on MD-Hard, a stream with changing noise conditions. Adding dynamic reset to DSUTA reduced WER from 45.6% to 39.8%, a 5.8 percentage-point drop. Resetting the same method every 50 utterances raised WER by 3.8 points to 49.4%. Carrying every update forward (CSUTA) put WER 9.0 points above the fixed model. These results describe transcription on the changing stream. Measuring retention in a previous environment requires evaluating the model again on a separate set from that environment. Lin et al., EMNLP 2024, Table 2

The change from 45.6% to 39.8% can also be reported as a relative error reduction. The denominator is the WER of DSUTA before adding dynamic reset, 45.6%.

45.6−39.845.6×100≈12.7  %\frac{45.6-39.8}{45.6}\times100\approx12.7\;\%

Using the rounded entries in the table, the absolute difference is 5.8 percentage points and the relative reduction is about 12.7%. The two calculations use different denominators.

#Part II, p. 30 — After teaching a new skill, ask for the old one again

A puzzled robot sits beside the title of the 2025 lecture on post-training and forgetting.
Part II, p. 30 · The slide points back to the 2025 lecture on forgetting.Hung-yi Lee / NTU ML 2026 · Original source

Lee recalls that repeated weight updates can erode a previously learned skill. He then asks what happens if only the harness changes, for example through a new system instruction or a change to external memory. The diagnostic differs. Suppose an agent stops reporting units after an instruction edit. Give it the same input and weights, and restore the previous instruction. If the units return, the instruction change explains the behavioral difference in this case. NTU lecture, 30:18

#Supplementary lecture · Rehearse on old examples and evaluate on held-out ones

Some training examples from an earlier task are mixed with training examples for the new task before fitting the next model.
NTU’s 2025 post-training lecture, p. 29 · Show examples from the first task while teaching the second.Hung-yi Lee / NTU ML 2025 · Original source

Pages 28–30 of the 2025 post-training lecture compare sequential fine-tuning, a regularization method that limits changes to important weights (MAS), and experience replay. Follow the plus sign on the middle slide: add a sample of old training examples to the training set for the new task. Revisiting those original inputs and targets can help preserve a mapping the model might otherwise lose. Lee’s example adds about five old examples per hundred new training examples. A suitable replay ratio depends on how much data can be retained and how closely the tasks resemble each other. Lecture 6, 42:56 · Slides 28–30

How would we check for forgetting in a vision system? Set aside separate training and evaluation images for each of two cameras, A and B. The hypothetical results below compare predictions on the same held-out images before and after adaptation on camera B’s training data. Only A’s training images can go into the replay buffer. Its evaluation images remain held out throughout.

Same held-out evaluation setBefore adapting to BAfter adapting to BAccuracy change
Original camera A90%82%−8 percentage points
New camera B60%78%+18 percentage points
ΔB=78−60=+18  pp,ΔA=82−90=−8  pp\Delta_B=78-60=+18\;\mathrm{pp},\qquad \Delta_A=82-90=-8\;\mathrm{pp}

The new camera gained 18 points and the original camera lost 8. To measure the effect of replay, start two runs from the same model. Use only B’s training images in the first. Mix in some of A’s training images in the second. Give both runs the same update budget, then measure both models on the same held-out A and B sets. This gives separate readings for retention and adaptation.

#Worked example · How much forgetting does a final accuracy of 80.3% conceal?

Read the table above as a history of learning two tasks. The GEM paper’s backward transfer (BWT) metric compares accuracy just after a task was learned with accuracy on the same evaluation set after the final task. With just one earlier task A, BWT is 82 − 90 = −8 percentage points. The +18 points on B measure adaptation to the new task. GEM, Eqs. 2–4

Now introduce a third camera C. Each row marks the point just after learning from that camera. Each column uses a fixed test set from one camera. The diagonal gives each camera’s accuracy immediately after learning it. The row for C is hypothetical.

Training completed throughAccuracy on AAccuracy on BAccuracy on C
A90%——
B82%78%—
C86%71%84%

Read the last row across for the final performance. If each camera gets equal weight, average accuracy (ACC) is as follows.

ACC=(86+71+84)/3=241/3≈80.3  %\mathrm{ACC}=(86+71+84)/3=241/3\approx80.3\;\%

To measure forgetting, read down each column. A fell from 90% when learned to 86% at the end. B fell from 78% to 71%. Average these two changes. C has experienced no subsequent task, so it does not enter the denominator.

BWT=(86−90)+(71−78)2=−4−72=−5.5  pp\mathrm{BWT}=\frac{(86-90)+(71-78)}{2}=\frac{-4-7}{2}=-5.5\;\mathrm{pp}

On A, accuracy recovered by 4 percentage points since the preceding checkpoint. It still sits 4 points below its value just after learning A. Recovery and residual forgetting can coexist because their reference times differ. The final average of 80.3% does not expose that history.

The blue starting models on page 27, the colors passed between rows on page 28, and the slowly updated reference model on page 29 each answer the same question: Which changes made for this input should the next input inherit? Episode 20 continues on page 31 of the second lecture.

Connect