Skip to main content
#self-improvement#reinforcement-learning#reward-shaping#robotics#ntu

Self-Improving AI 05 β€” Designing rewards and evaluating answers

Eureka, REvolve and RF-Agent introduce reward search. The lesson connects dopamine prediction errors, preference-based reward models, RLHF and RLAIF.

An AI that writes reward code and an AI that judges answer quality serve different roles. Both require a separately defined and validated standard for a good outcome.

Part 4 β€” When Correct Answers Are Rare, What Should We Reward? distinguished a training proxy reward from the actual task evaluation. We now compare reward-design search strategies and turn to how a model can learn criteria that are difficult to express in code.

Topics by slide
SlideDistinctions to watch for
21Refining one reward candidate versus maintaining a population or tree of candidates
22The outer loop designing reward code and the inner loop training a policy
23Reward shaping as an analogy and research on dopamine reward prediction errors
24Direct human scores and a reward model trained on human preferences
25An AI emitting a score and a system training on AI feedback

#Slide 21 β€” Finding a good reward is itself a search problem

A comparison from the RF-Agent paper diagrams how Eureka, REvolve, and RF-Agent search reward candidates.
P021 β€” Reward candidate search in Eureka, REvolve, and RF-Agent.Hung-yi Lee / NTU. Β· Original source

The slide presents Eureka, REvolve, and RF-Agent side by side. Its figure comes from the RF-Agent paper. Read it as the RF-Agent authors’ abstraction for comparing their method with earlier work, rather than a new measurement under common experimental conditions. RF-Agent, Figure 1 and section 1

#Reward-generation methods retain different things

Eureka generates reward code from environment code and a task description, then uses policy-training outcomes as feedback. It carries promising candidates and failure records into the next iteration. The original paper calls this evolutionary optimization. Comparisons in REvolve and RF-Agent emphasize iteration from top candidates and simplify it as greedy search. The descriptions use different levels of abstraction. Eureka, section 3 Β· REvolve, section 1

REvolve maintains a population of reward functions and applies genetic programming operations such as mutation, crossover, and migration. People compare the resulting policy behavior and provide natural-language feedback. The LLM writes Python reward code, but people continue to judge the goal. REvolve, sections 2–3

RF-Agent organizes histories of reward modifications in a tree and chooses which path to expand with Monte Carlo tree search (MCTS). Its branches represent reward-design decisions and their histories, distinct from individual joint actions by a robot. RF-Agent, section 4

ComparisonTypical Eureka iterationREvolveRF-Agent
Starting point for improvementSelected promising candidates and training feedbackPopulation of reward candidates and genetic operationsSelected tree path and accumulated history
Reward formExecutable codeExecutable codeExecutable code
Important evaluation evidenceOutcomes of the policy trained with the rewardHuman policy comparisons and natural-language feedbackPolicy-training results and evaluation metrics
Distinction to keep in mindImproving LLM context versus training LLM weightsAutomation still incorporates human feedbackRobot action search versus reward-design search

The table compares structures. It does not rank performance for all tasks and implementations. The best scores from unrelated experiments cannot establish a ranking under common conditions.

#Further reading A β€” Figure 1 from REvolve

REvolve Figure 1 shows an LLM generating a population of rewards, policies trained with each, human comparison of their behavior, and population updates using preferences and natural-language feedback.
REvolve, Figure 1 β€” Reward search incorporating human policy preferences and natural-language feedback (PDF page 3).Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, Pedro Zuidberg Dos Martires. Β· Original source

Start at the left. The reward designer receives the task and environment variables. Each resulting reward trains a policy. People then compare rollouts of the trained policies, rather than the wording of reward code. Their preferences and natural-language feedback help form the next reward population. REvolve v4, Figure 1, PDF page 3

Imagine two driving policies that both reach the destination. One changes lanes abruptly while the other moves smoothly. A human preference for the latter communicates more than a binary arrival outcome.

How should preferences be aggregated when evaluators disagree? If one evaluator changes their judgment after repeated comparisons, how is that recorded? Human involvement does not ensure complete and consistent evaluation. For reproduction, check the guidelines, presentation order, evaluator count, and treatment of disagreements.

#Slide 22 β€” Separate the reward-design tree from the policy-training loop

RF-Agent uses candidate reward histories and tree search in reward design.
P022 β€” RF-Agent tree search using reward-design history.Hung-yi Lee / NTU. Β· Original source

This slide illustrates the components and search paths of RF-Agent in more detail. Each reward-design decision entails training and evaluating a policy under that reward. Counting LLM calls alone can miss much of the total cost. RF-Agent, Figure 2 and sections 3–4

#Two kinds of action and two kinds of feedback

An outer-loop action creates or edits reward code. An inner-loop action is a control taken by the robot policy in the environment. The outer loop observes trained-policy performance and failures. The inner loop uses rewards to update policy parameters.

We can write the two loops in terms of policy training and evaluation.

ΞΈk=Train⁑(ΞΈ0,rΟ•k,B),qk=Eval⁑(πθk;Dselect).\theta_k=\operatorname{Train}(\theta_0,r_{\phi_k},B),\qquad q_k=\operatorname{Eval}(\pi_{\theta_k};\mathcal D_{\mathrm{select}}).

Here Ο•k\phi_k is the reward candidate’s code or settings, ΞΈk\theta_k the policy trained with it, BB the training budget, and qkq_k the evaluation value under a common criterion. A record for the LLM can include candidate code, intended edits, execution errors, changes in individual reward terms, and policy evaluation.

RF-Agent combines history-conditioned reasoning with MCTS. It describes procedures for aligning design reasoning with code and for self-verification. Self-verification is not an infallible substitute for evaluating policy behavior. The paper still uses inner-loop training and environmental feedback. RF-Agent, sections 4.2–4.3 and Appendix B

#Why maintain a search tree, and what does it cost?

Imagine that candidate A emphasizes approaching a handle, B the cost of maintaining contact, and C the door angle. Even if A is best now, editing only A may miss an improvement accessible from B. A tree records which changes produced which outcomes and lets the procedure choose between exploiting promising paths and exploring underexamined ones.

Keeping candidates does not eliminate computational costs. Training eight candidates from scratch with three independent seeds each requires 24 policy-training runs. With BB per run, the total inner-loop budget is 24B24B, plus LLM generation and evaluation costs.

Did experiments compare success rates at the same number of complete training runs? Did they report LLM tokens and time alongside policy-training time? Were candidate-selection conditions held apart from the final test? Unequal search budgets make it difficult to attribute a difference solely to the algorithm.

Experiments in RF-Agent arXiv v1 (February 27, 2026) evaluated 17 simulated low-level control tasks. They do not establish performance guarantees across all physical robots, sensor errors, or human–robot collaboration settings. Version information Β· Experiments, section 5.1

#Slide 23 β€” How far does the dopamine analogy go?

A title slide introduces the relationship between dopamine and reward shaping.
P023 β€” Dopamine and reward shaping.Hung-yi Lee / NTU. Β· Original source

Consider intermediate goals and rewards. The speaker notes draw an analogy with an intermediate cue guiding behavior before a final outcome.

The analogy illustrates how cues that predict an outcome can guide behavior before that outcome arrives. It would be a substantial overreach to conclude that dopamine is a pleasure score and the brain operates exactly like a reward function we wrote.

#Further reading B β€” Reward and reward prediction error

Original plots compare dopamine neuron responses to an unexpected reward, a cue-predicted reward, and an expected reward that was omitted.
Schultz, Dayan, and Montague, Figure 1 β€” Dopamine responses with reward prediction and omission (PDF page 2).Wolfram Schultz, Peter Dayan, P. Read Montague / Science 275 (1997). Β· Original source

Figure 1 of the 1997 paper by Schultz, Dayan, and Montague compares dopamine neuron responses recorded under specific experimental conditions. The top panel shows an unexpected reward, the middle a cue-predicted reward, and the bottom an expected reward that did not arrive. CS marks the conditioned stimulus and R the reward time. Original paper, PDF page 2, printed page 1594

The paper connects responses to reward with the difference between prediction and actual outcome. As the prediction is learned, the response can shift from the reward to an earlier cue. When an expected reward is omitted, activity decreases around the anticipated time. This links a particular experiment and computational model, rather than completely defining all dopamine activity.

In a simple state-value model, the temporal-difference error in reinforcement learning is

Ξ΄t=rt+Ξ³V(st+1)βˆ’V(st).\delta_t=r_t+\gamma V(s_{t+1})-V(s_t).

rtr_t is the observed reward, and V(st)V(s_t) represents expected future reward from the current state. Ξ΄t\delta_t is an update signal from the comparison. Reward and prediction error are different variables.

Suppose a hypothetical episode ends after one last step and V(st+1)=0V(s_{t+1})=0. If a reward of 1 was unexpected and V(st)=0V(s_t)=0, the error is 1. If a reward of 1 was expected and received, the error is zero. Expecting 1 and receiving nothing gives βˆ’1. These are illustrative numbers, not measured firing rates.

The resemblance to the potential-based shaping term Ξ³Ξ¦(st+1)βˆ’Ξ¦(st)\gamma\Phi(s_{t+1})-\Phi(s_t) in Part 4 is interesting. PBRS changes reward, while a TD error updates a value prediction. Similar mathematical terms do not establish biological identity or guarantee learning.

Read the speaker notes’ analogies involving evolution, food, and intermediate goals within these limits. They do not by themselves explain modern eating habits or addiction or support clinical judgments. The point here is to distinguish learning signals from ultimate objectives.

#Slide 24 β€” Learning evaluation criteria that are hard to write down

An RLHF diagram shows how human preferences train a reward model, which in turn evaluates a generative policy.
P024 β€” A reward model trained on human preferences and a generative policy.Hung-yi Lee / NTU. Β· Original source

Door-opening success can be defined by an angle or a state. A few simple conditionals struggle to express whether a long summary is helpful, whether an answer addresses the question faithfully, or whether it includes unnecessarily risky content. The RLHF diagram introduces a reward model that learns such judgments.

In a typical preference-based RLHF workflow, people compare candidate answers. Their comparisons train a reward model, which then scores answers produced by the policy. People need not grade every token or training step in real time. Stanford CS224N Lecture 10, PDF pages 58–62

#Further reading C β€” From preference comparisons to reward-model loss

A slide trains a reward model using preferences over three summaries and a logistic loss on the score difference between winners and losers.
Stanford CS224N β€” Training a reward model on pairwise answer preferences (PDF page 60).Archit Sharma / Stanford CS224N; based on Jesse Mu slides. Β· Original source

The Stanford slide compares answers to the same question and gives an expression for training the reward model to score the chosen answer higher. Its printed slide number is 63 and its PDF page number is 60.

Let xx be the question, ywy_w the preferred answer, yly_l the less preferred answer, and rψ(x,y)r_\psi(x,y) the reward-model score. A Bradley–Terry pairwise model gives the following preference probability.

Pψ(yw≻yl∣x)=σ ⁣(rψ(x,yw)βˆ’rψ(x,yl)),Οƒ(z)=11+eβˆ’z.P_\psi(y_w\succ y_l\mid x) =\sigma\!\left(r_\psi(x,y_w)-r_\psi(x,y_l)\right), \qquad \sigma(z)=\frac{1}{1+e^{-z}}.

A common loss for fitting the reward model to observed preferences is

LRM(ψ)=βˆ’E(x,yw,yl)∼Dlog⁑σ ⁣(rψ(x,yw)βˆ’rψ(x,yl)).\mathcal L_{\mathrm{RM}}(\psi) =-\mathbb E_{(x,y_w,y_l)\sim\mathcal D} \log\sigma\!\left(r_\psi(x,y_w)-r_\psi(x,y_l)\right).

This loss trains reward-model parameters ψ\psi. Distinguish them from generative-policy parameters θ\theta. The learned preferences depend on the criteria used to create the comparisons.

#A numerical example with a score difference of 1.5

Suppose the preferred answer scores 1.2 and the other βˆ’0.3. The difference is 1.5, yielding the following model preference probability.

Οƒ(1.5)β‰ˆ0.8176,βˆ’log⁑σ(1.5)β‰ˆ0.2014.\sigma(1.5)\approx0.8176,\qquad -\log\sigma(1.5)\approx0.2014.

This does not say there is an 81.76% chance that the answer is factually true. It is the probability the specified comparison model assigns to the observed preference. Adding 100 to both scores leaves their difference at 1.5 and the preference probability unchanged. Pairwise preferences do not identify an absolute zero for scores.

A model trained on human judgments can reflect disagreement and noise in those judgments. Preference learning is a different task from determining truth. Examine how factuality, helpfulness, and safety were combined.

#Fitting a reward model and training a generative policy

Once trained, the reward model can guide a generative policy toward producing high-scoring answers more often. The following representative objective simplifies a completed answer to one choice.

J(ΞΈ)=Ex, yβˆΌΟ€ΞΈ(β‹…βˆ£x)[rψ(x,y)βˆ’Ξ²log⁑πθ(y∣x)Ο€ref(y∣x)].J(\theta)=\mathbb E_{x,\,y\sim\pi_\theta(\cdot\mid x)} \left[r_\psi(x,y)-\beta \log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\right].

Ο€ref\pi_{\mathrm{ref}} is the reference policy, and Ξ²\beta controls the strength of the term discouraging departure from it. The expected log ratio connects to KL divergence. This expression omits token-level rewards, value estimation, and optimization details and is not the full PPO implementation. Stanford PDF page 62, printed slide 66

A KL constraint does not remove reward-model errors. A policy may find weaknesses in the scorer, so evaluate final answers against independent criteria. The proxy-reward problem from Part 4 remains.

#Slide 25 β€” RLHF already uses AI. What changes in RLAIF?

An RLAIF diagram changes the source of training judgments to AI feedback.
P025 β€” RLAIF uses AI feedback as a training signal.Hung-yi Lee / NTU. Β· Original source

The lecture moves from RLHF to RLAIF. Saying that AI scores outputs instead of people can be misleading. In RLHF, an AI reward model can already provide the scores during policy training. Who emits the final score alone therefore does little to distinguish RLHF and RLAIF.

In a preference-based setup, ask where the comparisons and judgments that train the reward model came from. RLHF uses human feedback and RLAIF uses AI-generated feedback as its main supervision. Some methods train a preference model while others use AI evaluation directly, so RLAIF does not imply a single architecture.

#Further reading D β€” The two training stages of Constitutional AI

The original two-stage diagram places supervised learning on critiques and revisions above AI preference data, a preference model, and reinforcement learning.
Constitutional AI, Figure 1 β€” Supervised learning from critiques and revisions, then reinforcement learning with AI preferences (PDF page 2).Yuntao Bai et al. / Anthropic. Β· Original source

The upper part of Constitutional AI Figure 1 shows supervised learning on critiques and revisions. Below, AI comparisons of answer pairs train a preference model that supports reinforcement learning. Bai et al., arXiv v1, Figure 1, PDF page 2

The upper stage connects to training on revised answers in Part 3. In the lower stage, AI preference judgments produce data for a preference model, whose scores then train a generative policy. Generating critiques, generating preference data, training a reward model, and training a generative policy are different stages.

Humans still contribute. People set the principles in the constitution, and a base model and initial training data already exist. The paper describes combining automated harmlessness feedback with human helpfulness feedback. Saying the AI invented its goals from scratch without any human supervision would misrepresent the work. Constitutional AI, section 4 and discussion of combining human and AI feedback

#Three questions remain when AI evaluates

Which criteria apply? A preference can change depending on the weight given to truthfulness, politeness, brevity, or risk avoidance. Make criteria explicit in the prompt, principles, or labeling instructions.

What evidence informs the decision? Distinguish a judgment based on fluent phrasing from one using executed calculations, source checks, or inspection tools. If the same model generates and grades answers, check for repeated errors. Changing evaluator models may help, but distinct model names do not guarantee independent errors.

Who performs final verification and with what evidence? A higher AI evaluation score alone does not establish better user outcomes or actual accuracy. Use task-appropriate independent evidence such as separately human-rated data, executable tests, unseen questions, or real-world success criteria.

Suppose a hypothetical summary evaluator tends to prefer longer answers. A generative policy could increase its score by becoming verbose without improving accuracy. Alongside refining the evaluator prompt, compare answers at controlled lengths, check facts, and use new evaluation questions to establish what improved.

#Two types of self-improvement compared

QuestionAutomated reward-code designAutomated preference or evaluation feedback
What AI primarily producesCode calculating scores from environment statesPreferences, critiques, and evaluation data for answers
ExamplesEureka, REvolve, RF-AgentRLAIF stage of Constitutional AI
What subsequently learnsA policy acting in the environmentA preference model and the policy generating answers
Easily overlooked human rolesTask, environment, success criteria, feedback designPrinciples, rating instructions, data, final evaluation
Shared riskA policy can increase proxy reward while missing the goalA policy can satisfy the evaluator’s preferences while missing the goal

Look past the name of the automation to ask which artifact changed, which parameters were trained, and what additional information informed the judgment of quality. Editing code does not in itself train LLM parameters. A higher AI score does not in itself establish a more capable policy.

#Three questions to check understanding

1. Does REvolve find good rewards without people? No. It uses human policy preferences and natural-language feedback to evolve rewards.

2. Does applying a sigmoid to a difference in reward-model scores give the probability an answer is true? No. It gives a preference probability under the specified pairwise model. A probability of factual correctness requires its own definition and validation.

3. Does RLHF become RLAIF whenever a neural network assigns the score? No. A typical RLHF setup uses a reward model trained on human comparisons. Identify the feedback source and training stage.

Next are slides 26–30. They examine how AI self-evaluation can become a signal through verbal scores, agreement across answers, confidence, and entropy. Continue to Self-Improving AI 06 β€” Does a More Confident AI Become More Accurate?.

#Original sources for further reading

SourceVersion and locationTopic
NTU Self-ImprovingPDF pages 21–25Main lecture
EurekaAuthors’ project PDF, section 3Reward code generation and training feedback
REvolvev4, May 23, 2025, Figure 1 on PDF page 3Population search and human feedback
RF-Agentv1, February 27, 2026, Figures 1–2 and sections 4–5Reward design with history and MCTS
Schultz, Dayan, and MontagueScience, March 14, 1997, Figure 1 on PDF page 2Connecting reward prediction error and dopamine research
Stanford CS224N Lecture 102024 lecture, PDF page 60 (printed 63) and PDF page 62 (printed 66)Preference model and KL-constrained objective
Constitutional AIv1, December 15, 2022, Figure 1 and section 4Principle-guided AI feedback and RLAIF

Connect