Skip to main content
#self-improvement#reinforcement-learning#reward-shaping#robotics#ntu

Self-Improving AI 04 — What should we reward when correct answers are rare?

A counterexample shows how intermediate rewards can change the objective. Potential-based shaping, generated reward code and policy evaluation offer ways to check the effect.

A reward that is easy to learn from can differ from the goal we want to achieve. Even when AI designs the reward, evaluate the policy trained on it using a separate success criterion.

Part 3 distinguished self-correction, which revises an answer, from training, which changes model parameters. It also compared supervised imitation of a target with reinforcement learning from rewards.

That leads to the next question. If we can give a score instead of a correct answer, who should design that score, and how?

#Slide 16 — Success is clearly defined but rarely encountered

Original slide 16

A robot opening a door illustrates how sparse real-world rewards can make learning difficult.
NTU slide 16 — A robot door-opening task with rare success rewards.Hung-yi Lee · NTU · Original source

The slide raises the problem of sparse rewards in practice, using a robot opening a door as an example.

Suppose the task gives one point only when the door opens and zero otherwise. The following is a simplified illustration, separate from actual robot reward code.

rttask={1,the door-opening task succeeds at step t,0,otherwise.r_t^{\mathrm{task}} = \begin{cases} 1, & \text{the door-opening task succeeds at step }t,\\ 0, & \text{otherwise.} \end{cases}

The success criterion is clear. But if an unskilled policy repeatedly fails to open the door, every observed task reward is zero. A trial that approaches the handle can receive the same score as one moving in the opposite direction.

What is missing is a learning signal indicating which attempts came closer to success. When exploration rarely encounters useful experience, there are fewer opportunities to learn from it.

A zero reward does not imply that all learning must stop. Depending on the algorithm, value estimates, entropy terms, pretraining, or separate exploration signals may contribute. The difficulty is that the task’s success reward alone provides little basis for distinguishing better actions.

#Calculating sparsity with a 0.1% chance of success

For a toy calculation, assume that training has yet to change the policy, that trials are independent, and that each has a fixed success probability of 0.001.

The probability of at least one success in 100 trials is

P(at least one success in N trials)=1−(1−p)N.P(\text{at least one success in }N\text{ trials})=1-(1-p)^N.
1−(1−0.001)100≈0.0952.1-(1-0.001)^{100}\approx 0.0952.

Even after 100 attempts, the chance of experiencing success at least once is only about 9.52%. The expected number of trials to first success is

E[Nfirst]=1p=1000.\mathbb{E}[N_{\mathrm{first}}]=\frac{1}{p}=1000.

In actual RL the policy changes, and trials may be correlated. These figures give neither a prediction nor a lower bound for robot training sample requirements. They illustrate how obtaining successful experience can itself become the bottleneck when reward events are rare.

#Further reading A — Sparse and dense rewards at UMass

Scott Niekum, CS 690 Reward Specification, PDF page 4

A slide compares a grid rewarding only the goal with a grid assigning rewards at multiple states.
UMass CS 690, PDF page 4 — Sparse and dense rewards.Scott Niekum / UMass Amherst CS 690, Spring 2024 · Original source

The lecture slide compares a grid with a large value only at the goal against one with values at multiple locations. The left grid represents a simple success condition. The right distinguishes more situations along the way.

Separate observing informative values more frequently from the claim that actions maximizing them best serve the original goal. Dense rewards provide more learning signals, but adding signals alone does not preserve the objective.

Sparse and delayed rewards are also distinct. Sparsity concerns how infrequently a reward event occurs. Delay concerns time between an important action and its consequence. Door opening can exhibit both.

#Slide 17 — Break door opening into steps and keep the ultimate goal in view

Original slide 17

The original reward-shaping diagram assigns zero far from the handle, one for approaching, and five for contact.
NTU slide 17 — Intermediate rewards for approaching and touching the handle.Hung-yi Lee · NTU · Original source

Slide 17 assigns zero far from the handle, +1 when approaching, and +5 for contact. These are different illustrative reward settings from the +0.5 for contact and +1 for opening on slide 18.

Given the goal “open the door,” a researcher might consider the following intermediate events.

StageObservable changeWhat it leaves uncertain
ApproachFingertip-to-handle distance decreases.Suitable hand orientation and collision avoidance need separate checks.
Contact and graspThe robot touches and grasps the handle.The handle may not have turned and the door may still be closed.
Manipulate the handleHandle angle or latch state changes.Door motion and eventual passage are separate outcomes.
Move the doorThe opening angle increases.The target angle, duration, and safety conditions need verification.

The table helps identify measurable signals before final success. It does not prescribe one sequence for all doors. A handleless door, a push door, and an automatic door have different intermediate steps.

#State, action, and reward have different roles

State can include task information such as hand position, handle angle, and door angle. Actions depend on the control interface, such as target joint positions or force commands. A reward is a number evaluating the state, action, and next state.

For simplicity, the expression below assumes a fully observable Markov decision process (MDP).

at∼πθ(⋅∣st),st+1∼P(⋅∣st,at),rt=r(st,at,st+1).a_t\sim\pi_\theta(\cdot\mid s_t),\qquad s_{t+1}\sim P(\cdot\mid s_t,a_t),\qquad r_t=r(s_t,a_t,s_{t+1}).

An actual camera-based robot uses observations without direct access to the full state. Estimating these state variables from video is a separate problem. Reward design does not automatically resolve perception error or sensor delay.

#What changes when intermediate events receive rewards?

Adding a term for proximity to the handle distinguishes actions even before the first successful opening. Designing auxiliary rewards to aid learning is the basic idea of reward shaping. Ng, Harada, and Russell, section 2.2 also describes adding an auxiliary term to the original reward.

If approaching becomes the effective objective, the robot may stop at the door. A large reward for grasping may encourage holding the handle indefinitely. A good intermediate signal helps the policy learn actions leading to final success within a limited training budget.

#Slide 18 — Even an AI-designed reward can change the objective

Original slide 18

The slide compares the true reward for opening the door with a proxy rewarding intermediate handle contact.
NTU slide 18 — The door-opening objective and a handle-contact proxy.Hung-yi Lee · NTU · Original source

The slide distinguishes the original reward function from a proxy reward. The example scores successful door opening under the original goal and adds an intermediate signal such as handle contact to the proxy.

Reinforcement learning generally concerns return accumulated across time steps. The difference between an instantaneous score and return creates a reward design pitfall.

G=∑t=0T−1γtrt,0≤γ≤1.G=\sum_{t=0}^{T-1}\gamma^t r_t,\qquad 0\leq\gamma\leq1.

The discount factor γ\gamma determines the weight of future rewards relative to current ones. TT is the maximum trajectory length considered here.

#A counterexample — Collecting contact points without opening the door

Consider a hypothetical thought experiment in which the +0.5 contact reward can be collected repeatedly.

Start in a state where the door can be opened immediately. Opening it earns one point and ends the episode on that step. If the door stays closed but the robot touches the handle, it earns 0.5 per step. The horizon is 20 steps and the discount factor is 0.99.

Policy A maintains contact for 20 steps without opening the door. Policy B opens it on the first step and terminates. The example does not award contact and success together.

GA=0.5∑t=0190.99t=0.51−0.99201−0.99≈9.1047.G_A=0.5\sum_{t=0}^{19}0.99^t =0.5\frac{1-0.99^{20}}{1-0.99} \approx9.1047.
GB=1.G_B=1.
PolicyActual openingReturn under the designed reward
A: maintain contactFailureAbout 9.10
B: open immediately and terminateSuccess1.00

The reward ranks sustained contact above the intended success. The agent has calculated the score correctly. Behavior that faithfully maximizes the designed score has diverged from the behavior we wanted.

Paying once on first contact can reduce this particular form of repeated collection. Other shortcuts may still change the effective objective, for example rushing to an intermediate milestone and then abandoning the task.

#Further reading B — Potential-based reward shaping

Scott Niekum, CS 690 Reward Specification, PDF page 5

A potential-based reward shaping slide adds the discounted next-state potential minus the current-state potential to the reward.
UMass CS 690, PDF page 5 — Shaping rewards with differences in potential.Scott Niekum / UMass Amherst CS 690, Spring 2024 · Original source

We can add a term whose structure permits analysis of policy preservation. Potential-based reward shaping (PBRS) uses differences in a potential function Φ\Phi defined on states.

rt′=rt+F(st,st+1),F(st,st+1)=γΦ(st+1)−Φ(st).r'_t=r_t+F(s_t,s_{t+1}),\qquad F(s_t,s_{t+1})=\gamma\Phi(s_{t+1})-\Phi(s_t).

Theorem 1 in Ng, Harada, and Russell (1999) studies how this transformation relates to preservation of optimal policies. The potential need not equal the true value function. The discount, state definition, and treatment of termination must meet the mathematical conditions before drawing a preservation claim.

Summing reward over a finite trajectory makes the intermediate potential terms cancel.

G′=∑t=0T−1γt[rt+γΦ(st+1)−Φ(st)]=G−Φ(s0)+γTΦ(sT).\begin{aligned} G' &=\sum_{t=0}^{T-1}\gamma^t \left[r_t+\gamma\Phi(s_{t+1})-\Phi(s_t)\right]\\ &=G-\Phi(s_0)+\gamma^T\Phi(s_T). \end{aligned}

With the same initial state and terminal potential set to zero, the only added value is −Φ(s0)-\Phi(s_0), independent of policy. The ranking of policies by return is preserved under these conditions. For an infinite discounted horizon, the same intuition follows as the boundary term vanishes when the potential is bounded and γ<1\gamma<1.

Conversely, truncating at a time limit while leaving an arbitrary final potential can make γTΦ(sT)\gamma^T\Phi(s_T) policy dependent. Distinguish actual termination from truncation by a time limit. For a finite horizon, also check the time-augmented state and potential and the final boundary condition.

#Numerical check — Intermediate rewards without changing the ranking

Assign the following potentials and original rewards to four illustrative states. The potentials are chosen to check the derivation and are not experimental estimates.

[Φ(s0),Φ(s1),Φ(s2),Φ(s3)]=[−1.0,−0.6,−0.2,0],γ=0.9.[\Phi(s_0),\Phi(s_1),\Phi(s_2),\Phi(s_3)] =[-1.0,-0.6,-0.2,0],\qquad \gamma=0.9.
[r0,r1,r2]=[0,0,1].[r_0,r_1,r_2]=[0,0,1].

The added rewards are 0.46, 0.42, and 0.20, respectively. The training rewards therefore become

[r0′,r1′,r2′]=[0.46,0.42,1.20].[r'_0,r'_1,r'_2]=[0.46,0.42,1.20].
G=0.92=0.81,G′=0.46+0.9(0.42)+0.92(1.20)=1.81.G=0.9^2=0.81,\qquad G'=0.46+0.9(0.42)+0.9^2(1.20)=1.81.

The difference is exactly −Φ(s0)=1-\Phi(s_0)=1. This single trajectory does not verify every policy. Rank preservation for all policies follows from the structure and boundary conditions above.

Moreover, preserving optimal policies does not guarantee faster learning. With finite samples, function approximation, exploration, and optimizer choices, learning outcomes require measurement. Check that an LLM-written reward actually has the required form before claiming PBRS guarantees.

#Slide 19 — Separate scores for training from criteria for evaluation

Original slide 19

The diagram distinguishes the proxy loss used to optimize the output from the real loss used to evaluate performance.
NTU slide 19 — Proxy loss for training and real loss for performance evaluation.Hung-yi Lee · NTU · Original source

This slide distinguishes “For evaluation” from “For optimization.” Applied to the counterexample:

Training proxy reward: Provide useful signals during learning, such as approach, contact, and handle manipulation.

Separate task evaluation: Check whether the door actually opens, measure the success rate, and characterize failures.

The original also calls these quantities real loss and proxy loss. Rewards are maximized and losses minimized, but an implemented RL policy loss need not simply negate the numeric reward. Policy gradients, value estimates, and regularization may enter the training equation.

#Why raw scores from different reward candidates are incomparable

A new reward function may change units and scale. If one candidate awards 0.1 for approaching the handle and another awards 100 for the same event, their training-curve heights cannot be compared directly.

Multiplying the same reward by a positive constant of 100 multiplies raw return by 100 without multiplying actual success rate by 100. Even when the mathematical ranking is preserved, scale may affect training stability in practice. Include it among experimental conditions.

Compare reward candidates by what policies trained with them achieve under a shared evaluation criterion. A large number emitted by the reward conveys little on its own.

#Further reading C — Rewards can overfit to the training environment

Scott Niekum, CS 690 Reward Specification, PDF page 11

The slide asks whether a correct task specification suffices and discusses overfitting during trial-and-error reward design.
UMass CS 690, PDF page 11 — Overfitting from repeated reward design.Scott Niekum / UMass Amherst CS 690, Spring 2024 · Original source

This supplementary slide considers overfitting as people repeatedly adjust rewards after observing training. The associated AAAI 2023 paper by Booth and colleagues investigates how trial-and-error reward design can overfit a learning algorithm and its settings or misspecify the intended task.

In our example, a reward working on one door and one initial pose cannot yet be generalized to door opening. Other handle positions, starting poses, or training settings may produce different results. The following design can reduce that risk.

StageData or settingsAppropriate useMisleading use
Policy trainingTraining environments and initial statesTrain a policy under each candidate reward.Report the training score as independent final performance.
Reward candidate selectionSeparate selection environments and seedsCompare rewards and decide the next change.Call a repeatedly consulted selection score an untouched test score.
Final evaluationConditions unused for candidate selectionEvaluate generalization after selection ends.Revise the reward after inspecting the test and report that same result as final.

Control training steps, compute, policy architecture, initialization, and evaluation seeds as far as possible. Report variation across seeds. The appropriate statistic and sample size depend on the experiment. No one run count is universally sufficient.

On real robots, monitor collisions, excessive contact forces, and joint-limit violations in addition to success. Penalizing an event does not establish that safety requirements are met. The experimental suggestion here is to validate with independent constraints and stopping mechanisms in place.

#Slide 20 — The LLM edits reward code, and evaluation measures the trained policy

Original slide 20

Two loops show an LLM revising a proxy reward and evaluation of a policy trained with it. The policy need not be an LLM.
NTU slide 20 — The reward-design and policy-training loops.Hung-yi Lee · NTU · Original source

This slide asks how AI can discover which proxy rewards help learning. It also points out that the policy need not be an LLM.

There are two roles. The reward designer produces a reward function or code. The policy learner learns how to act in an environment using that reward. An LLM designer does not imply that the robot’s low-level control policy uses an LLM.

#Inner and outer loops

The inner loop holds a reward fixed and trains a policy. The outer loop revises candidate rewards based on the trained policy’s performance and failures.

We can express the two roles schematically as follows.

θϕ=Train⁡(θ0,rϕ,B,Dtrain).\theta_{\phi} =\operatorname{Train}(\theta_0,r_{\phi},B,\mathcal D_{\mathrm{train}}).
ϕ∗∈arg max⁡ϕEval⁡(πθϕ;Dselect).\phi^*\in\operatorname*{arg\,max}_{\phi} \operatorname{Eval}(\pi_{\theta_{\phi}};\mathcal D_{\mathrm{select}}).

ϕ\phi denotes a reward candidate’s code or settings, θ\theta the policy parameters, and BB a fixed training budget. The outer objective is a good evaluation outcome for the policy trained with that reward. The size of the reward’s raw output is incidental.

The notation does not require differentiating the outer loop. An LLM can search by proposing code candidates, receiving textual feedback on actual training, and producing another candidate. Modifying code and training the LLM’s weights are different operations.

#Further reading D — Reward generation and policy evaluation in Eureka

Ma and colleagues, Eureka paper, project PDF page 2, Figure 2

Eureka Figure 2 shows generation of reward code from environment code and a task description, followed by policy training, evaluation, and reward reflection.
Eureka, Figure 2 — Reward code generation, policy training and evaluation, and reward reflection.Yecheng Jason Ma et al. / Eureka · Original source

The Eureka diagram starts from environment code and a task description, generates reward code, trains and evaluates a policy with it, then uses the results to revise the reward. The authors’ project description of the components and paper, sections 2–3 clarify the division of roles.

Note three features. The inputs include the task and available environment variables. Evaluation includes actual policy training. Feedback guides the next edit to the reward code. This goes beyond asking an LLM once to write a good reward.

The paper describes reward reflection using individual reward components and policy-training statistics. These records reveal whether a term dominates, barely changes, or encourages unintended behavior. Giving that feedback in the LLM’s context differs from fine-tuning its parameters.

#Executable code alone does not establish reward quality

Evaluate a generated reward for computability, behavioral preferences, and actual learning effects.

First, can it be computed? Check input variables and units, batch dimensions, output format, and finite values. A candidate referencing a missing state variable or producing NaN fails before a performance comparison.

Second, does it prefer the intended behavior? Test simple counterexamples such as repeating in place, moving back and forth, maintaining contact, or terminating early. Passing a few counterexamples offers no guarantee across the entire action space.

Third, does it help training? Train policies under equal budgets and compare common success and failure criteria. Reserve test conditions from candidate revision.

Separate the door-opening success check from the code that generates reward candidates. If a candidate writer can edit the success check, it can change the evaluation rule instead of improving behavior. Structure the experiment so that it cannot modify that check.

#Three questions to check understanding

1. Is 0.5 points for handle contact always a good reward?

No. Return depends on when, how often, and under what conditions the reward is paid. A one-time milestone and a per-step payment can define different objectives.

2. Does potential-based reward shaping resolve every failure?

No. Optimal policy preservation under specified assumptions differs from efficiency, stability, and safety in finite training. Mishandling the terminal boundary can invalidate even the rank-preservation premise.

3. If the LLM produces good reward code, has the robot finished learning?

No. Code proposal, policy learning, and independent evaluation are distinct. Code that looks good still needs performance validation on the actual policy.

#How the five slides fit together

Slide 16 introduces rare success rewards. Slide 17 uses door opening to identify intermediate signals. Slide 18 distinguishes true goals from proxies, and slide 19 separates optimization from evaluation. Slide 20 connects reward design and policy learning as two distinct loops.

The central question is “Does a policy trained on the reward achieve the original goal more often?” It applies whether a human or an AI designed the reward.

Next, Self-Improving AI 05 — AI That Designs Rewards and AI That Evaluates Answers covers slides 21–25, comparing reward-design research and distinguishing the training signals in RLHF and RLAIF.

#Further reading

NTU Machine Learning 2026 course information

SourceWhere to readConnection to this article
Hung-yi Lee, NTU Self-ImprovingPDF pages 16–20Main lecture sequence and notation
Scott Niekum, UMass CS 690 Reward SpecificationPDF pages 4, 5, and 11Sparse and dense rewards, potential-based shaping, overfitting
Ng, Harada, Russell, ICML 1999Section 2.2 and section 3, Theorem 1, PDF pages 3–4Reward transformations and conditions for policy preservation
Booth et al., AAAI 2023Paper and authors’ research materialsOverfitting and task misspecification in trial-and-error reward design
Ma et al., Eureka project paperPDF page 2, Figure 2, and sections 2–3Reward generation, policy training and evaluation, reward reflection
Eureka authors’ projectComponents and public task examplesDistinguishing reward generation from policy learning
Reproducing the calculations in Python

The following reproduces the three illustrative calculations without external packages.

from math import isclose


def discounted_return(rewards: list[float], gamma: float) -> float:
    """Finite-horizon discounted return for the illustrative examples."""
    if not 0.0 <= gamma <= 1.0:
        raise ValueError("gamma must be in [0, 1]")
    return sum((gamma ** t) * reward for t, reward in enumerate(rewards))


def main() -> None:
    # Independent fixed-probability trials, not an RL sample-complexity bound.
    p, trials = 0.001, 100
    chance = 1.0 - (1.0 - p) ** trials
    print(f"Success within 100 trials: {100.0 * chance:.4f}%")
    print(f"Expected trials to first success: {1.0 / p:.0f}")

    # A stays in contact; B succeeds immediately and terminates.
    contact_return = discounted_return([0.5] * 20, gamma=0.99)
    success_return = discounted_return([1.0], gamma=0.99)
    assert contact_return > success_return
    print(f"Repeated contact / immediate success: {contact_return:.6f} / {success_return:.6f}")

    gamma = 0.9
    rewards = [0.0, 0.0, 1.0]
    potentials = [-1.0, -0.6, -0.2, 0.0]
    if len(potentials) != len(rewards) + 1:
        raise ValueError("One potential is required for every visited state")
    shaped = [
        reward + gamma * potentials[t + 1] - potentials[t]
        for t, reward in enumerate(rewards)
    ]
    original_return = discounted_return(rewards, gamma)
    shaped_return = discounted_return(shaped, gamma)
    boundary_identity = (
        original_return - potentials[0]
        + gamma ** len(rewards) * potentials[-1]
    )
    assert isclose(shaped_return, boundary_identity, rel_tol=0.0, abs_tol=1e-12)
    assert isclose(shaped_return, 1.81, rel_tol=0.0, abs_tol=1e-12)
    print("Shaped rewards:", [round(value, 2) for value in shaped])
    print(f"Original / shaped return: {original_return:.2f} / {shaped_return:.2f}")


if __name__ == "__main__":
    main()

The first calculation finds the chance of a rare success under independent trials. The second checks that repeated intermediate rewards can outscore final success. The third verifies cancellation of potential terms at the boundaries. We use isclose to avoid exact equality comparisons between floating-point results as though they were identical strings.

Before applying the code in an experiment, define termination rules, time limits, and evaluation environments. An identity verified in a numerical example does not establish convergence for neural network training.

Connect