An AI that writes reward code and an AI that judges answer quality serve different roles. Both require a separately defined and validated standard for a good outcome.
Part 4 β When Correct Answers Are Rare, What Should We Reward? distinguished a training proxy reward from the actual task evaluation. We now compare reward-design search strategies and turn to how a model can learn criteria that are difficult to express in code.
Topics by slide
| Slide | Distinctions to watch for |
|---|---|
| 21 | Refining one reward candidate versus maintaining a population or tree of candidates |
| 22 | The outer loop designing reward code and the inner loop training a policy |
| 23 | Reward shaping as an analogy and research on dopamine reward prediction errors |
| 24 | Direct human scores and a reward model trained on human preferences |
| 25 | An AI emitting a score and a system training on AI feedback |
#Slide 21 β Finding a good reward is itself a search problem
The slide presents Eureka, REvolve, and RF-Agent side by side. Its figure comes from the RF-Agent paper. Read it as the RF-Agent authorsβ abstraction for comparing their method with earlier work, rather than a new measurement under common experimental conditions. RF-Agent, Figure 1 and section 1
#Reward-generation methods retain different things
Eureka generates reward code from environment code and a task description, then uses policy-training outcomes as feedback. It carries promising candidates and failure records into the next iteration. The original paper calls this evolutionary optimization. Comparisons in REvolve and RF-Agent emphasize iteration from top candidates and simplify it as greedy search. The descriptions use different levels of abstraction. Eureka, section 3 Β· REvolve, section 1
REvolve maintains a population of reward functions and applies genetic programming operations such as mutation, crossover, and migration. People compare the resulting policy behavior and provide natural-language feedback. The LLM writes Python reward code, but people continue to judge the goal. REvolve, sections 2β3
RF-Agent organizes histories of reward modifications in a tree and chooses which path to expand with Monte Carlo tree search (MCTS). Its branches represent reward-design decisions and their histories, distinct from individual joint actions by a robot. RF-Agent, section 4
| Comparison | Typical Eureka iteration | REvolve | RF-Agent |
|---|---|---|---|
| Starting point for improvement | Selected promising candidates and training feedback | Population of reward candidates and genetic operations | Selected tree path and accumulated history |
| Reward form | Executable code | Executable code | Executable code |
| Important evaluation evidence | Outcomes of the policy trained with the reward | Human policy comparisons and natural-language feedback | Policy-training results and evaluation metrics |
| Distinction to keep in mind | Improving LLM context versus training LLM weights | Automation still incorporates human feedback | Robot action search versus reward-design search |
The table compares structures. It does not rank performance for all tasks and implementations. The best scores from unrelated experiments cannot establish a ranking under common conditions.
#Further reading A β Figure 1 from REvolve
Start at the left. The reward designer receives the task and environment variables. Each resulting reward trains a policy. People then compare rollouts of the trained policies, rather than the wording of reward code. Their preferences and natural-language feedback help form the next reward population. REvolve v4, Figure 1, PDF page 3
Imagine two driving policies that both reach the destination. One changes lanes abruptly while the other moves smoothly. A human preference for the latter communicates more than a binary arrival outcome.
How should preferences be aggregated when evaluators disagree? If one evaluator changes their judgment after repeated comparisons, how is that recorded? Human involvement does not ensure complete and consistent evaluation. For reproduction, check the guidelines, presentation order, evaluator count, and treatment of disagreements.
#Slide 22 β Separate the reward-design tree from the policy-training loop
This slide illustrates the components and search paths of RF-Agent in more detail. Each reward-design decision entails training and evaluating a policy under that reward. Counting LLM calls alone can miss much of the total cost. RF-Agent, Figure 2 and sections 3β4
#Two kinds of action and two kinds of feedback
An outer-loop action creates or edits reward code. An inner-loop action is a control taken by the robot policy in the environment. The outer loop observes trained-policy performance and failures. The inner loop uses rewards to update policy parameters.
We can write the two loops in terms of policy training and evaluation.
Here is the reward candidateβs code or settings, the policy trained with it, the training budget, and the evaluation value under a common criterion. A record for the LLM can include candidate code, intended edits, execution errors, changes in individual reward terms, and policy evaluation.
RF-Agent combines history-conditioned reasoning with MCTS. It describes procedures for aligning design reasoning with code and for self-verification. Self-verification is not an infallible substitute for evaluating policy behavior. The paper still uses inner-loop training and environmental feedback. RF-Agent, sections 4.2β4.3 and Appendix B
#Why maintain a search tree, and what does it cost?
Imagine that candidate A emphasizes approaching a handle, B the cost of maintaining contact, and C the door angle. Even if A is best now, editing only A may miss an improvement accessible from B. A tree records which changes produced which outcomes and lets the procedure choose between exploiting promising paths and exploring underexamined ones.
Keeping candidates does not eliminate computational costs. Training eight candidates from scratch with three independent seeds each requires 24 policy-training runs. With per run, the total inner-loop budget is , plus LLM generation and evaluation costs.
Did experiments compare success rates at the same number of complete training runs? Did they report LLM tokens and time alongside policy-training time? Were candidate-selection conditions held apart from the final test? Unequal search budgets make it difficult to attribute a difference solely to the algorithm.
Experiments in RF-Agent arXiv v1 (February 27, 2026) evaluated 17 simulated low-level control tasks. They do not establish performance guarantees across all physical robots, sensor errors, or humanβrobot collaboration settings. Version information Β· Experiments, section 5.1
#Slide 23 β How far does the dopamine analogy go?
Consider intermediate goals and rewards. The speaker notes draw an analogy with an intermediate cue guiding behavior before a final outcome.
The analogy illustrates how cues that predict an outcome can guide behavior before that outcome arrives. It would be a substantial overreach to conclude that dopamine is a pleasure score and the brain operates exactly like a reward function we wrote.
#Further reading B β Reward and reward prediction error
Figure 1 of the 1997 paper by Schultz, Dayan, and Montague compares dopamine neuron responses recorded under specific experimental conditions. The top panel shows an unexpected reward, the middle a cue-predicted reward, and the bottom an expected reward that did not arrive. CS marks the conditioned stimulus and R the reward time. Original paper, PDF page 2, printed page 1594
The paper connects responses to reward with the difference between prediction and actual outcome. As the prediction is learned, the response can shift from the reward to an earlier cue. When an expected reward is omitted, activity decreases around the anticipated time. This links a particular experiment and computational model, rather than completely defining all dopamine activity.
In a simple state-value model, the temporal-difference error in reinforcement learning is
is the observed reward, and represents expected future reward from the current state. is an update signal from the comparison. Reward and prediction error are different variables.
Suppose a hypothetical episode ends after one last step and . If a reward of 1 was unexpected and , the error is 1. If a reward of 1 was expected and received, the error is zero. Expecting 1 and receiving nothing gives β1. These are illustrative numbers, not measured firing rates.
The resemblance to the potential-based shaping term in Part 4 is interesting. PBRS changes reward, while a TD error updates a value prediction. Similar mathematical terms do not establish biological identity or guarantee learning.
Read the speaker notesβ analogies involving evolution, food, and intermediate goals within these limits. They do not by themselves explain modern eating habits or addiction or support clinical judgments. The point here is to distinguish learning signals from ultimate objectives.
#Slide 24 β Learning evaluation criteria that are hard to write down
Door-opening success can be defined by an angle or a state. A few simple conditionals struggle to express whether a long summary is helpful, whether an answer addresses the question faithfully, or whether it includes unnecessarily risky content. The RLHF diagram introduces a reward model that learns such judgments.
In a typical preference-based RLHF workflow, people compare candidate answers. Their comparisons train a reward model, which then scores answers produced by the policy. People need not grade every token or training step in real time. Stanford CS224N Lecture 10, PDF pages 58β62
#Further reading C β From preference comparisons to reward-model loss
The Stanford slide compares answers to the same question and gives an expression for training the reward model to score the chosen answer higher. Its printed slide number is 63 and its PDF page number is 60.
Let be the question, the preferred answer, the less preferred answer, and the reward-model score. A BradleyβTerry pairwise model gives the following preference probability.
A common loss for fitting the reward model to observed preferences is
This loss trains reward-model parameters . Distinguish them from generative-policy parameters . The learned preferences depend on the criteria used to create the comparisons.
#A numerical example with a score difference of 1.5
Suppose the preferred answer scores 1.2 and the other β0.3. The difference is 1.5, yielding the following model preference probability.
This does not say there is an 81.76% chance that the answer is factually true. It is the probability the specified comparison model assigns to the observed preference. Adding 100 to both scores leaves their difference at 1.5 and the preference probability unchanged. Pairwise preferences do not identify an absolute zero for scores.
A model trained on human judgments can reflect disagreement and noise in those judgments. Preference learning is a different task from determining truth. Examine how factuality, helpfulness, and safety were combined.
#Fitting a reward model and training a generative policy
Once trained, the reward model can guide a generative policy toward producing high-scoring answers more often. The following representative objective simplifies a completed answer to one choice.
is the reference policy, and controls the strength of the term discouraging departure from it. The expected log ratio connects to KL divergence. This expression omits token-level rewards, value estimation, and optimization details and is not the full PPO implementation. Stanford PDF page 62, printed slide 66
A KL constraint does not remove reward-model errors. A policy may find weaknesses in the scorer, so evaluate final answers against independent criteria. The proxy-reward problem from Part 4 remains.
#Slide 25 β RLHF already uses AI. What changes in RLAIF?
The lecture moves from RLHF to RLAIF. Saying that AI scores outputs instead of people can be misleading. In RLHF, an AI reward model can already provide the scores during policy training. Who emits the final score alone therefore does little to distinguish RLHF and RLAIF.
In a preference-based setup, ask where the comparisons and judgments that train the reward model came from. RLHF uses human feedback and RLAIF uses AI-generated feedback as its main supervision. Some methods train a preference model while others use AI evaluation directly, so RLAIF does not imply a single architecture.
#Further reading D β The two training stages of Constitutional AI
The upper part of Constitutional AI Figure 1 shows supervised learning on critiques and revisions. Below, AI comparisons of answer pairs train a preference model that supports reinforcement learning. Bai et al., arXiv v1, Figure 1, PDF page 2
The upper stage connects to training on revised answers in Part 3. In the lower stage, AI preference judgments produce data for a preference model, whose scores then train a generative policy. Generating critiques, generating preference data, training a reward model, and training a generative policy are different stages.
Humans still contribute. People set the principles in the constitution, and a base model and initial training data already exist. The paper describes combining automated harmlessness feedback with human helpfulness feedback. Saying the AI invented its goals from scratch without any human supervision would misrepresent the work. Constitutional AI, section 4 and discussion of combining human and AI feedback
#Three questions remain when AI evaluates
Which criteria apply? A preference can change depending on the weight given to truthfulness, politeness, brevity, or risk avoidance. Make criteria explicit in the prompt, principles, or labeling instructions.
What evidence informs the decision? Distinguish a judgment based on fluent phrasing from one using executed calculations, source checks, or inspection tools. If the same model generates and grades answers, check for repeated errors. Changing evaluator models may help, but distinct model names do not guarantee independent errors.
Who performs final verification and with what evidence? A higher AI evaluation score alone does not establish better user outcomes or actual accuracy. Use task-appropriate independent evidence such as separately human-rated data, executable tests, unseen questions, or real-world success criteria.
Suppose a hypothetical summary evaluator tends to prefer longer answers. A generative policy could increase its score by becoming verbose without improving accuracy. Alongside refining the evaluator prompt, compare answers at controlled lengths, check facts, and use new evaluation questions to establish what improved.
#Two types of self-improvement compared
| Question | Automated reward-code design | Automated preference or evaluation feedback |
|---|---|---|
| What AI primarily produces | Code calculating scores from environment states | Preferences, critiques, and evaluation data for answers |
| Examples | Eureka, REvolve, RF-Agent | RLAIF stage of Constitutional AI |
| What subsequently learns | A policy acting in the environment | A preference model and the policy generating answers |
| Easily overlooked human roles | Task, environment, success criteria, feedback design | Principles, rating instructions, data, final evaluation |
| Shared risk | A policy can increase proxy reward while missing the goal | A policy can satisfy the evaluatorβs preferences while missing the goal |
Look past the name of the automation to ask which artifact changed, which parameters were trained, and what additional information informed the judgment of quality. Editing code does not in itself train LLM parameters. A higher AI score does not in itself establish a more capable policy.
#Three questions to check understanding
1. Does REvolve find good rewards without people? No. It uses human policy preferences and natural-language feedback to evolve rewards.
2. Does applying a sigmoid to a difference in reward-model scores give the probability an answer is true? No. It gives a preference probability under the specified pairwise model. A probability of factual correctness requires its own definition and validation.
3. Does RLHF become RLAIF whenever a neural network assigns the score? No. A typical RLHF setup uses a reward model trained on human comparisons. Identify the feedback source and training stage.
Next are slides 26β30. They examine how AI self-evaluation can become a signal through verbal scores, agreement across answers, confidence, and entropy. Continue to Self-Improving AI 06 β Does a More Confident AI Become More Accurate?.
#Original sources for further reading
| Source | Version and location | Topic |
|---|---|---|
| NTU Self-Improving | PDF pages 21β25 | Main lecture |
| Eureka | Authorsβ project PDF, section 3 | Reward code generation and training feedback |
| REvolve | v4, May 23, 2025, Figure 1 on PDF page 3 | Population search and human feedback |
| RF-Agent | v1, February 27, 2026, Figures 1β2 and sections 4β5 | Reward design with history and MCTS |
| Schultz, Dayan, and Montague | Science, March 14, 1997, Figure 1 on PDF page 2 | Connecting reward prediction error and dopamine research |
| Stanford CS224N Lecture 10 | 2024 lecture, PDF page 60 (printed 63) and PDF page 62 (printed 66) | Preference model and KL-constrained objective |
| Constitutional AI | v1, December 15, 2022, Figure 1 and section 4 | Principle-guided AI feedback and RLAIF |