Skip to main content
#self-improvement#entropy-minimization#policy-gradient#test-time-adaptation#ntu

Self-Improving AI 08 — The missing gradient in entropy minimization

Token-wise estimates of sequence entropy connect to the policy gradient. A probability tree and automatic differentiation show which term disappears when samples are held fixed.

Getting the right loss on average and updating parameters in the right direction are different things. If the model determines the distribution that supplies the samples, its derivative also matters.

Part 7 — When should self-training stop, and when should a model adapt? distinguished the scope of self-training from test-time training. Here we examine the calculation behind entropy minimization. Does the expression we actually backpropagate give the gradient of the entropy we want to reduce?

Video timestamps by slide
SlideVideo segment and topicRelated concept
3636:00–36:33 — Introducing the math and the studyLoss values and gradients, notation and assumptions
3736:33–39:55 — Sequence and token entropy and their relationship in expectationChain rule and a two-token probability tree
3839:55–41:11 — Computing a gradient with the sample held fixedChanges to the sampling distribution versus automatic differentiation
3941:11–42:44 — The missing term in the derivative of an expectationProduct rule, log-derivative identity and a correction to the slide
4042:44–43:24 — Reducing entropy along the selected pathGradient sign, numerical counterexample and a surrogate loss for implementation

#Slide 36 — Does equality of values imply equality of gradients?

Slide introducing a study of entropy minimization in autoregressive models.
P036 — Entropy minimization in autoregressive models.Hung-yi Lee / NTU ML 2026. · Original source

In the video · 36:00–36:33. The lecturer asks how entropy can serve as a loss when enumerating every possible answer is impractical. He introduces research from his lab and turns to the mathematics.

Rethinking Entropy Minimization in Test-Time Adaptation for Autoregressive Models explains entropy minimization for autoregressive models in terms of two gradient components. Huang et al., arXiv v1

#Further reading A — Why do two losses appear together?

Diagram distinguishing the roles of token entropy loss and policy gradient loss.
Huang et al., Figure 1 — Roles of the token entropy loss and policy gradient loss (PDF p. 1).Wei-Ping Huang et al., arXiv:2605.08186v1 (2026). · Original source

The upper half reads a given token sequence as context and sharpens the distribution at each position. The lower half generates several output paths and trains the model to select paths with lower costs more often. The arrows and multiplication signs are schematic. Use the exact derivatives in the paper when implementing them. Figure 1 and equations (8)–(9)

Keep four questions distinct as you read these five slides.

QuantityQuestion
Objective valueWhat is the entropy over all answers the model can produce?
Sample estimateHow can one generated answer estimate that quantity?
Fixed-sample gradientWhat do we get if we fix the sampled tokens and differentiate the loss?
Full objective gradientHave we included the effect of sampling different answers as the parameters change?

We use xx for the input, Y=(Y1,…,YT)Y=(Y_1,\ldots,Y_T) for the random token sequence and yy for one sampled sequence. pθ(y∣x)p_\theta(y\mid x) is the probability that the model generates that sequence. We use natural logarithms and measure entropy in nats.

We start with a fixed finite length and a finite vocabulary. We can then sum over all outputs and check the derivatives directly if the probabilities are positive and differentiable. Variable-length sequences with an end-of-sequence token (EOS) require additional conditions, discussed at the end of slide 37.

#Slide 37 — Computing sequence entropy one token at a time

Original slide 37 relating full sequence entropy to the expected sum of next-token entropies over the prefixes visited.
P037 — Sequence entropy and the sum of token entropies.Hung-yi Lee / NTU ML 2026. · Original source

In the video · 36:33–39:55. The lecturer contrasts a classifier’s finite outputs with the large output space of text generation. He calculates the next-token distribution and sums the token entropies along a generated path. The relationship between the expectation of this sum and full entropy is our starting point.

#What exactly is infeasible to compute?

With vocabulary size V and length T, there are V to the power T possible sequences of fixed length. Entropy is well defined. What becomes impractical for a large model is enumerating all sequences exactly. In a small model, we can list every output and calculate entropy directly.

The entropy of the full output is

L(θ;x)=Hθ(Y∣x)=−∑ypθ(y∣x)log⁡pθ(y∣x)=EY∼pθ(⋅∣x)[−log⁡pθ(Y∣x)].L(\theta;x)=H_\theta(Y\mid x) =-\sum_y p_\theta(y\mid x)\log p_\theta(y\mid x) =\mathbb E_{Y\sim p_\theta(\cdot\mid x)}[-\log p_\theta(Y\mid x)].

An autoregressive model assigns a probability equal to the product of token probabilities conditioned on the preceding tokens.

pθ(y∣x)=∏t=1Tpθ(yt∣x,y<t),−log⁡pθ(y∣x)=∑t=1T−log⁡pθ(yt∣x,y<t).p_\theta(y\mid x)=\prod_{t=1}^{T}p_\theta(y_t\mid x,y_{<t}),\qquad -\log p_\theta(y\mid x)=\sum_{t=1}^{T}-\log p_\theta(y_t\mid x,y_{<t}).

Turning this product into a sum does not imply that the tokens are independent. It follows from the chain rule for conditional probabilities. Different preceding tokens can give different distributions for later tokens.

#H on the slide describes a distribution before the token is selected

Given a prefix s, calculate entropy over the whole next-token distribution.

hθ(s;x)=−∑v∈Vpθ(v∣x,s)log⁡pθ(v∣x,s).h_\theta(s;x)= -\sum_{v\in\mathcal V}p_\theta(v\mid x,s)\log p_\theta(v\mid x,s).

The slide’s H(ϕ)H(\phi) corresponds to the initial empty prefix. The H terms below it use prefixes containing previously sampled tokens. Here we denote the empty prefix by ∅\varnothing. Having already selected A does not set this entropy to zero. We are evaluating the distribution before choosing the next token in that context.

Define the sum of entropies over the prefixes visited on one path as follows. It corresponds to the slide’s surrogate loss, with a subscript to show its dependence on the parameters.

gθ(y;x)=∑t=1Thθ(y<t;x).g_\theta(y;x)=\sum_{t=1}^{T}h_\theta(y_{<t};x).

This value generally differs from the negative log probability of that path. Their expectations coincide when we sample paths from the model’s own distribution.

L(θ;x)=∑t=1TEY<t ⁣[hθ(Y<t;x)]=EY∼pθ ⁣[gθ(Y;x)].L(\theta;x) =\sum_{t=1}^{T}\mathbb E_{Y_{<t}}\!\left[h_\theta(Y_{<t};x)\right] =\mathbb E_{Y\sim p_\theta}\!\left[g_\theta(Y;x)\right].

This is unbiasedness of the value estimate. Averaging over many paths drawn from the same distribution estimates the full entropy. Neither the value on a single path nor its derivative must equal the full entropy or its gradient, respectively. Paper §3.1, equations (3)–(7)

#Further reading B — The chain rule uses conditional entropy

Lecture notes giving the entropy chain rule H(X,Y)=H(Y)+H(X|Y) in terms of conditional entropy.
Stanford EE376A — The chain rule linking joint and conditional entropy (PDF p. 5).Tsachy Weissman / Stanford EE376A, Lecture 3 (2018-01-16). · Original source

The Stanford notes decompose the joint entropy of two random variables into the entropy of one and the conditional entropy of the other. Conditional entropy also averages over the possible values of the conditioning variable. The notes use base-2 logarithms by default. The identity is the same with natural logarithms, but converting the numerical values requires a change of units. Definitions and logarithm base · Chain rule

#Worked example — A small model with four possible answers

Consider a model that generates two tokens. The first is A or B with probability 0.5 each. After A it generates 0 or 1 with probabilities 0.9 and 0.1. After B the two probabilities are 0.5 each. After two tokens it terminates with probability 1, so termination contributes no further entropy.

Output pathPath probabilitySum g of entropies of the token distributions visited
A00.451.018230 nat
A10.051.018230 nat
B00.251.386294 nat
B10.251.386294 nat

Writing h for binary entropy gives

h(p)=−plog⁡p−(1−p)log⁡(1−p),h(0.5)=0.693147,h(0.9)=0.325083.h(p)=-p\log p-(1-p)\log(1-p),\qquad h(0.5)=0.693147,\quad h(0.9)=0.325083.
g(A0)=g(A1)=h(0.5)+h(0.9)=1.018230,g(B0)=g(B1)=2h(0.5)=1.386294.g(A0)=g(A1)=h(0.5)+h(0.9)=1.018230,\qquad g(B0)=g(B1)=2h(0.5)=1.386294.
E[g(Y)]=0.5(1.018230)+0.5(1.386294)=1.202262 nat.\mathbb E[g(Y)] =0.5(1.018230)+0.5(1.386294) =1.202262\ \mathrm{nat}.

Computing the full entropy directly from the probabilities of the four paths gives the same result.

H(0.45,0.05,0.25,0.25)=1.202262 nat≈1.734498 bit.H(0.45,0.05,0.25,0.25)=1.202262\ \mathrm{nat} \approx1.734498\ \mathrm{bit}.

For the single path A0, g is 1.018230, while the full entropy is 1.202262. Greedy decoding always picks the most likely path A0 here and therefore cannot stand in for the expectation under the original distribution.

Conditions for variable length. The paper proves unbiasedness of the token estimator for finite sequences ending in EOS when the entropy is finite. An implementation should include EOS and exclude subsequent padding. Truncation at a maximum length or averaging by token count can change the objective. Also, the conditions for an unbiased value do not by themselves justify exchanging differentiation and an infinite sum. We first derive the gradient for a fixed, finite output space. Theorem 1 and proof

#Slide 38 — What does differentiating with a fixed sample leave out?

Original slide 38 sampling an answer and differentiating token entropy while holding its prefixes fixed.
P038 — Computing the token entropy gradient with a fixed sample.Hung-yi Lee / NTU ML 2026. · Original source

In the video · 39:55–41:11. The lecturer first samples a token sequence and feeds its prefixes back into the model to compute the token entropy gradients. A different path is sampled each time, so the average update is expressed as an expectation.

The bottom of the figure feeds previously sampled tokens in as input. Automatic differentiation treats these token IDs as fixed. It differentiates the logits, probabilities and entropy with respect to the parameters given those tokens as context.

The resulting gradient for one sample is

∇θgθ(y;x)with sampled y held fixed.\nabla_\theta g_\theta(y;x)\qquad\text{with sampled }y\text{ held fixed}.

Let A be the average of this gradient over samples.

A(θ)=EY∼pθ ⁣[∇θgθ(Y;x)].A(\theta)=\mathbb E_{Y\sim p_\theta}\!\left[\nabla_\theta g_\theta(Y;x)\right].

The gradient of the actual objective, however, is

∇θL(θ;x)=∇θEY∼pθ ⁣[gθ(Y;x)].\nabla_\theta L(\theta;x) =\nabla_\theta\mathbb E_{Y\sim p_\theta}\!\left[g_\theta(Y;x)\right].

The two expressions appear to differ only in the position of the gradient. The important difference is that the distribution defining the expectation also depends on the same parameters. A weight change alters both the entropy at a given prefix and the probability of reaching it.

The expectation contains the product of two parameter-dependent terms.

L(θ;x)=∑ypθ(y∣x)⏟sampling probabilitygθ(y;x)⏟cost on this path.L(\theta;x)=\sum_y \underbrace{p_\theta(y\mid x)}_{\text{sampling probability}} \underbrace{g_\theta(y;x)}_{\text{cost on this path}}.

More samples can reduce the sampling error in estimating A. Even a perfect estimate of A still omits part of the full gradient. Repeating the calculation cannot remove the bias from the missing term.

Discrete sampling does not preclude computing the gradient. Instead of differentiating the sampled ID directly, we can use the gradient of its selection probability. The log-derivative identity in the next section gives the connection.

#Slide 39 — The product rule gives two terms

Original slide 39 comparing the objective loss, the surrogate loss and the two gradient terms.
P039 — The objective, the surrogate loss and the two gradient terms.Hung-yi Lee / NTU ML 2026. · Original source

In the video · 41:11–42:44. The lecturer emphasizes that although the computable surrogate loss equals the objective in expectation, moving the derivative inside the expectation can drop a term.

#Deriving the gradient line by line

In our finite output space, assume positive, differentiable probabilities and fix the input x. Start with the product rule.

∇θL=∇θ∑ypθ(y∣x)gθ(y;x)=∑ypθ(y∣x)∇θgθ(y;x)+∑ygθ(y;x)∇θpθ(y∣x).\begin{aligned} \nabla_\theta L &=\nabla_\theta\sum_y p_\theta(y\mid x)g_\theta(y;x)\\ &=\sum_y p_\theta(y\mid x)\nabla_\theta g_\theta(y;x) +\sum_y g_\theta(y;x)\nabla_\theta p_\theta(y\mid x). \end{aligned}

The first sum already weights each term by its probability and is therefore an expectation. To turn the second sum into an expectation under the same distribution, multiply and divide by the probability.

∇θpθ(y∣x)=pθ(y∣x)∇θlog⁡pθ(y∣x).\nabla_\theta p_\theta(y\mid x) =p_\theta(y\mid x)\nabla_\theta\log p_\theta(y\mid x).

The exact decomposition is

∇θL=EY∼pθ[∇θgθ(Y;x)]⏟A: direct entropy term+EY∼pθ[gθ(Y;x)∇θlog⁡pθ(Y∣x)]⏟B: sampling-distribution term.\boxed{\begin{aligned} \nabla_\theta L &=\underbrace{\mathbb E_{Y\sim p_\theta}[\nabla_\theta g_\theta(Y;x)]}_{A:\ \text{direct entropy term}}\\ &\quad+\underbrace{\mathbb E_{Y\sim p_\theta}[g_\theta(Y;x)\nabla_\theta\log p_\theta(Y\mid x)]}_{B:\ \text{sampling-distribution term}}. \end{aligned}}

A directly changes the cost of a visited path. B changes how often different paths are selected. Interpreting cost as negative reward connects B to the policy gradient. The decomposition follows by applying the product rule to the equality of values. Equation (8f) of the paper likewise contains the gradient of the log probability. Paper, PDF p. 3

#A notation trap — ∇p in the sum, ∇log p in the expectation

The final term in the original image takes the following form.

EY∼pθ[gθ(Y;x)∇θpθ(Y∣x)](as printed on the slide).\mathbb E_{Y\sim p_\theta} [g_\theta(Y;x)\nabla_\theta p_\theta(Y\mid x)]\qquad\text{(as printed on the slide)}.

This form is incorrect in general. The gradient of the probability belongs in the sum without probability weights. To express the same term as an expectation under p, use the gradient of the log probability.

∑ygθ(y;x)∇θpθ(y∣x)=EY∼pθ[gθ(Y;x)∇θlog⁡pθ(Y∣x)].\sum_y g_\theta(y;x)\nabla_\theta p_\theta(y\mid x) =\mathbb E_{Y\sim p_\theta}[g_\theta(Y;x)\nabla_\theta\log p_\theta(Y\mid x)].

The expectations on slides 39 and 40 omit log. Use the log-probability gradient as on the right side of the identity above.

#Further reading C — Berkeley’s log-derivative identity

Berkeley lecture slide explaining how the derivative of a probability equals the probability times the derivative of its logarithm.
UC Berkeley CS 185/285 — Rewriting a probability gradient using the log-probability gradient (PDF p. 5).Sergey Levine / UC Berkeley CS 185/285, Policy Gradients. · Original source

The equation in the blue box is the log-derivative trick we just used. Berkeley maximizes reward, whereas here we minimize an entropy cost. Mixing the signs for reward and cost reverses the update direction. Also, unlike the reward in the diagram, g here depends directly on the parameters. We must retain A as well as B. Original PDF, p. 5

#A probability tree with a correct value estimate and a zero gradient

Return to the two-token model on slide 37. This time only q, the probability of A as the first token, can change. Keep 0.9 after A and 0.5 after B fixed. Let q be the sigmoid of a logit z.

q=σ(z),L(z)=h(q)+q h(0.9)+(1−q)h(0.5).q=\sigma(z),\qquad L(z)=h(q)+q\,h(0.9)+(1-q)h(0.5).

The exact derivative is

dLdz=q(1−q)[log⁡1−qq+h(0.9)−h(0.5)].\frac{dL}{dz} =q(1-q)\left[\log\frac{1-q}{q}+h(0.9)-h(0.5)\right].

At q = 0.5 the entropy gradient of the first token itself is zero. But the distribution after A has lower entropy than the one after B. This difference contributes to the full gradient.

Az=0,Bz=0.25 [0.325083−0.693147]≈−0.092016.A_z=0,\qquad B_z=0.25\,[0.325083-0.693147] \approx-0.092016.
CalculationGradient with respect to zEffect of gradient descent
Differentiate full entropy over all four paths−0.092016Increases z, making A more likely
Use only the fixed-sample token entropy gradient A0Cannot change the first branch probability
Include both A and the correct B−0.092016Matches the derivative of full entropy
Use the slide’s expectation term without the logarithm+0.017725Moves the first branch in the opposite direction in this example

These are exact calculations over all four paths. The positive sign in the last row is not a rounding artifact. Using the probability gradient inside an expectation already weighted by p changes the weights. A finite-difference check of the full objective also gives −0.092016.

Limit of this interpretation. The model does not specify whether answers on path A are factually correct. This calculation determines the entropy gradient’s direction. Improving accuracy is a separate question. As Part 6 noted, a model can become more confident in an incorrect answer.

#Slide 40 — Making the selected path more certain

Original slide 40 highlighting the term that lowers token entropy along the sampled path.
P040 — The term that lowers token entropy along the selected path.Hung-yi Lee / NTU ML 2026. · Original source

In the video · 42:44–43:24. With a diagram of branching token paths, the lecturer explains the term that reduces entropy on the selected path. Slide 41 turns to changes in the probability of selecting other paths.

The red arrows mark the path used for the calculation. They do not identify a correct answer. A gives a gradient that sharpens distributions at fixed prefixes. On its own it does not account for the selection probabilities of all paths.

The slide speaks of the gradient as an update direction. When minimizing, parameters actually move in the negative gradient direction.

θk+1=θk−η ∇θL^,η>0.\theta_{k+1}=\theta_k-\eta\,\widehat{\nabla_\theta L},\qquad \eta>0.

#How does token entropy change the logits?

For a given context, write z for the logit vector and p for the softmax probabilities. The derivative of token entropy with respect to each logit is

∂H(p)∂zj=−pj [log⁡pj+H(p)].\frac{\partial H(p)}{\partial z_j} =-p_j\,[\log p_j+H(p)].

Because probabilities are normalized, each logit affects the other probabilities too. The gradient is zero at a uniform distribution. Reducing entropy therefore need not strengthen the best answer. This also explains why A cannot change the first branch in our probability tree when the initial distribution is exactly 0.5 versus 0.5.

#Further reading D — Building a surrogate loss for automatic differentiation

Original Berkeley slide constructing a surrogate loss so automatic differentiation gives the desired policy gradient.
UC Berkeley CS 185/285 — A surrogate loss for computing policy gradients with automatic differentiation (PDF p. 20).Sergey Levine / UC Berkeley CS 185/285, Policy Gradients. · Original source

The slide builds the computation graph to produce the gradient we want. The surrogate loss passed to backpropagation can have a different numerical value from the original evaluation metric. Its purpose is to give the required gradient estimate when differentiated. Berkeley PDF, p. 20

Hold the sample y fixed during differentiation. Write sg for stop-gradient, which preserves a value but blocks differentiation along that route. The implementation loss corresponding to equation (9) is

ℓsur(θ;y)=gθ(y;x)+sg⁡ ⁣(gθ(y;x)−b)log⁡pθ(y∣x).\ell_{\mathrm{sur}}(\theta;y) =g_\theta(y;x)+\operatorname{sg}\!\left(g_\theta(y;x)-b\right)\log p_\theta(y\mid x).

Here b is a fixed baseline independent of the sampled answer. Start with zero. Differentiating one sample gives

∇θℓsur=∇θgθ(y;x)+(gθ(y;x)−b)∇θlog⁡pθ(y∣x).\nabla_\theta\ell_{\mathrm{sur}} =\nabla_\theta g_\theta(y;x) +(g_\theta(y;x)-b)\nabla_\theta\log p_\theta(y\mid x).

The baseline leaves the expected gradient unchanged because the probabilities sum to one.

EY∼pθ[∇θlog⁡pθ(Y∣x)]=∑y∇θpθ(y∣x)=∇θ1=0.\mathbb E_{Y\sim p_\theta}[\nabla_\theta\log p_\theta(Y\mid x)] =\sum_y\nabla_\theta p_\theta(y\mid x)=\nabla_\theta 1=0.

Substituting a batch mean that includes the current sample, however, can make the baseline depend on that sample. Claiming unbiasedness then requires a separate design or correction. The code below uses a fixed scalar baseline.

#The essential PyTorch calculation

The excerpt below is the core of a function that validates input shapes, data types and masks. logits contains logits recomputed at each sampled prefix, sampled_tokens contains the sampled token IDs, and valid_mask is true through EOS and false for padding.

# logits: [batch, time, vocabulary]
# sampled_tokens: [batch, time], dtype=torch.long
# valid_mask: [batch, time], dtype=torch.bool
logp = torch.nn.functional.log_softmax(logits, dim=-1)
token_entropy = -(logp.exp() * logp).sum(dim=-1)
token_logp = logp.gather(-1, sampled_tokens.unsqueeze(-1)).squeeze(-1)

entropy_sum = (token_entropy * valid_mask).sum(dim=1)
sequence_logp = (token_logp * valid_mask).sum(dim=1)
loss = (entropy_sum + (entropy_sum.detach() - baseline) * sequence_logp).mean()

detach() makes the cost in the second term act only as a weight. Omitting detach there introduces an extra derivative of the cost multiplied by the log probability. Detaching the first entropy_sum as well would remove A. The signs here assume cost minimization. Equation (9) · PyTorch detach documentation · log_softmax documentation

If the function returns one loss per sequence, the caller should take the mean before backpropagating.

Implementation conditions. The policy that generates samples must match the distribution used in the loss. Arbitrary top-k or top-p filters, different temperatures, or samples from a stale model can invalidate the unbiasedness claim. Keep the model fixed between sampling and recomputation and handle stochastic behavior such as dropout consistently. model.eval() disables dropout. no_grad() separately disables gradient tracking.

Keep input prompt tokens separate from generated answer tokens and sum only over the generated portion, including EOS. Averaging by length can change the objective. Low-precision probabilities can be numerically unstable, so the full function performs the calculation in float32 for half and bfloat16 logits. These checks clarify the conditions of the calculation. Accuracy and stable adaptation still require evaluation.

#What these five slides establish

Under appropriate sampling conditions, the sum of token entropies can be an unbiased estimate of sequence entropy. Differentiating only that sum with the sample held fixed can omit changes in the distribution that selects the path.

Both terms on slide 39 follow from one objective. Differentiating the expectation accounts for changes to the cost and to the probabilities. Even the missing logarithm on the original slide can reverse the gradient direction, as the worked example shows. Checking the equations, the implementation and a small numerical example together catches this error.

Part 9 — The gradient that selects paths and a model that makes its own training problems continues with slides 41–45. We examine the path-selection term, comparison conditions in a speech recognition results table, AI evaluation and input generation.

#Further reading

ResourceWhere to lookConnection
NTU lecture video, first half36:00–43:24Entropy estimation and the two gradient terms
NTU Self-Improving PDFSlides 36–40Entropy values and gradients
Huang et al., arXiv:2605.08186v1Figure 1, §3.1–3.2, equations (3)–(9)Estimator conditions, exact gradient and stop-gradient
Stanford EE376A lecture notesLecture 3, January 16, 2018, pp. 1, 5Entropy units and the chain rule
Berkeley CS 185/285 lecture slidesPDF pp. 5, 20Log-derivative identity and an autodiff surrogate loss

Official course materials

Code and numerical checks

The 30 numerical checks and eight PyTorch autograd checks enumerate all four paths to compare entropy, gradients and baselines. The full function with shape, data type and mask handling documents its input requirements.

Connect