The same model can act as a different agent depending on what it reads, which tools it uses, and how past failures inform its next action.
Part 12 distinguished an AI designing a student’s learning procedure from evidence that the researcher itself improved. Here the question extends beyond the model: Can an agent do its job better while its model weights stay fixed?
The length of a retrieved document, the retry chosen after a tool fails, and the records loaded from earlier tasks all change the model’s next input. The execution system that makes these decisions is itself a target for improvement.
The final two slides introduce this perspective and lead into the next lecture. ReAct, Generative Agents, Reflexion, and SWE-agent are supplementary papers that develop the idea. Their designs help distinguish learning weights, updating memory, and revising a harness.
Video locations by slide
| Slide | Video | Topic |
|---|---|---|
| 61 | From 1:03:02 | The language model, harness, and files created during interaction |
| 62 | From 1:03:40 | The question leading into the next lecture |
#Slide 61 — An agent includes components outside the model
At 1:03:14, the lecturer explains that an agent encompasses more than the parameters of its language model. The harness region of the slide lists OpenClaw, Cowork, Claude Code, and Hermes. The language model region lists Claude, GPT, and Gemini. These examples illustrate the distinction the lecturer makes.
The harness calls the model, carries out tool requests, and packages results into subsequent inputs. It may determine which information and actions the model can access, how to maintain the task history, and when to stop. Its precise scope varies by system. This is why the lecturer says that improving the model alone improves just one part of the agent. Lecture at 1:03:20
It helps to distinguish the model, the harness, and the state accumulated during execution.
| Component | What it contains | Example change |
|---|---|---|
| Model parameters θ | Weights obtained through pretraining or fine-tuning | Training changes the conditional distribution of the next token |
| Harness H | Input assembly, tool interfaces, and rules for logging, retrying, and stopping | Modify the search result format or error handling |
| Execution state s | Current conversation, plans, observations, stored memories, and files | Read new search results and append to the task history |
A file’s format alone does not tell you its role. An observation log is execution state. A rules file read on the next run can become part of the harness configuration. Generated code that has yet to run is an output artifact. To explain how improvement happened, check which content actually informed the next action.
#Tracing the inputs to the model
The following notation illustrates the architecture without reproducing a particular paper’s algorithm. Given task x and execution state sₜ, suppose the harness uses a context builder C to assemble cₜ for the model.
Here aₜ is the next action proposed by the model. It could be a tool call or a final answer. Even with the same θ, the distribution over actions can change depending on which search results, failure messages, or relevant memories appear in cₜ.
Executing a tool action in environment state eₜ yields a new state and an observation oₜ₊₁. The harness incorporates this observation into the execution state.
P represents interaction with the environment through the tool interface. U is the rule for updating or summarizing the record. The environment can return different results on different runs. The model weights θ remain fixed throughout this loop.
A new observation can improve an answer. That improvement alone does not establish that θ learned anything or that the rules in H were automatically redesigned. Changing state and changing the rules for processing state are distinct operations.
#Supplement A — ReAct connects observations to the next action
ReAct interleaves reasoning and acting. In the question-answering example at the top, the agent chooses a search query, reads the tool’s response, and decides whether to search again or answer. In the environment task below, it checks whether it actually found an object as it carries out its plan. ReAct, section 2 and Figure 1
The purpose is to connect model output with external feedback so that an observation can change the next action when an initial assumption fails. Longer reasoning traces are incidental. Some tasks in the paper use reasoning only occasionally, rather than before every action. ReAct, sections 2–3
For example, imagine an agent looking for a function in a repository. An empty search result might prompt it to conclude that the function is absent. Alternatively, it could check the name and search scope and try again. The latter requires retaining the observation in the input and providing actions that can change the search strategy. This example is illustrative.
Here we focus on the paper’s prompted ReAct setting, which supplies demonstrations in the prompt. It shows how to design the sequence of interactions while keeping weights fixed. The original paper also reports separate fine-tuning experiments. ReAct, sections 3.2–3.3
#Supplement B — Generative Agents separates storage from retrieval
Generative Agents centers on a memory stream that records experiences in natural language. The agent retrieves relevant memories to choose an action, then stores reflections drawn from several memories as well as plans. The study concerns plausible human behavior in a virtual environment. Its findings do not establish gains on every kind of task. Generative Agents, section 4 and Figure 5
Logging more is insufficient on its own. Records left in storage cannot be read directly by a model call unless they enter its context. Conversely, repeatedly retrieving an old conjecture because it appears relevant can carry a mistaken assumption into later actions.
The paper combines recency, importance, and relevance. Recency uses elapsed game time since the last retrieval, importance uses a model-assigned rating, and relevance uses cosine similarity between query and memory embeddings. Each component undergoes min–max normalization to the 0–1 range before combination. Generative Agents, section 4.1
#Worked example 1 — A retrieval rule can change the ranking of the same memories
These hypothetical scores illustrate checking the deployment procedure for a repository. The three records are the entire candidate set for this example. R is a previously computed recency value, I is importance, and Q is relevance to the current question.
| Memory | R | I | Q |
|---|---|---|---|
| A: General task note just recorded | 0.98 | 2 | 0.1 |
| B: Previously verified deployment rule | 0.80 | 9 | 0.9 |
| C: Recent deployment checklist | 0.89 | 5 | 0.7 |
Normalize each column across candidates before combining raw scores that use different scales. For a component j whose maximum and minimum differ, the calculation is:
For C, normalized importance is (5−2)/(9−2)=3/7, and normalized relevance is (0.7−0.1)/(0.9−0.1)=0.75. If every candidate has the same value in a column, the denominator is zero. The verification code below gives such a column zero contribution because it cannot distinguish candidates. That convention is specific to this example.
Apply weights to the normalized components. The original paper’s implementation gives all three a weight of 1.
| Memory | Normalized (R, I, Q) | Score with weights (1, 1, 1) | Score emphasizing recency (0.8, 0.1, 0.1) |
|---|---|---|---|
| A | (1, 0, 0) | 1.0000 | 0.8000 |
| B | (0, 1, 1) | 2.0000 | 0.2000 |
| C | (0.5, 3/7, 0.75) | 1.6786 | 0.5179 |
B ranks first with equal weights, and A ranks first with the illustrative recency-heavy weights. The memories and model weights are identical: the retrieval rule changes which information the model sees. These scores can sum to more than 1 because they are not probabilities.
Retrieving B does not guarantee a correct answer. Looking important, resembling the question, and being factually correct are different properties. The example demonstrates the ranking calculation. Determining which weights work on real tasks requires a separate evaluation.
#Supplement C — Reflexion carries an explanation of failure into the next attempt
Reflexion connects the actor’s trial, the evaluator’s feedback, and the explanation produced by self-reflection. The current trajectory is a short-term record. An explanation of the failure enters the experience memory available to later trials. Feedback may come from external outcomes, rules, or a model’s own evaluation. Reflexion, section 3 and Figure 2(a)
The following notation simplifies the architecture. j indexes the trial, τ records its actions and observations, r is feedback, and m is memory retained between trials. As before, θ refers only to the language model weights.
Here ∥ denotes concatenating text records, and Keep is the rule for retaining records for the next trial. This expression illustrates the roles rather than reproducing the full algorithm. Appending a reflection involves no gradient update to the weights. The authors call this feedback path verbal reinforcement. Reflexion, introduction and section 3
After a failed test, an agent might record, “I missed a boundary condition. Next time I will try empty input first.” If the next trial reads that sentence and changes its behavior, the agent has reused experience. A reflection that misdiagnoses the cause can lead to the same error under a different description.
Fluent reflection alone is insufficient evidence of improvement. Check whether the revised behavior passed verification and whether the failure recurred less often. Tests written by a model can contain errors, so distinguish passing those tests from passing an independent final evaluation.
If H still uses the same rule for appending memories, the quantity that changed is m. A change to H would also be at issue if the agent altered the memory format or retention rule and actually applied it on the next run. Both changes can help, through different mechanisms.
#Supplement D — SWE-agent shows why tool interfaces affect performance
SWE-agent focuses on the agent–computer interface (ACI) between a model and a computer. Its design covers commands for navigating a repository and reading and editing files, as well as the format of command results. The paper explicitly studies improvements to that interface with a fixed language model. SWE-agent, section 2
Repeatedly printing an entire file produces a different context from displaying the current location, line numbers, and relevant excerpt. Immediate feedback from a syntax check also changes the agent’s opportunity to detect an error and choose its next action. The paper treats navigation, editing, and feedback as parts of ACI design. SWE-agent, sections 2–3
Saying an agent “can use tools” is only the starting point. Execution depends on the granularity of available actions, the specificity of failure messages, and how much old output remains in context. Harness changes shape how a model’s capabilities translate into work.
Gains from extra calls need to be distinguished from the effect of the interface itself. The calculation below shows why both success rate and budget matter.
#Worked example 2 — Retries change both the probability of success and the cost
Consider a tool request subject to transient failures. Assume the probability p of success is constant across independent attempts, and a success is detected perfectly and stops the process immediately. With at most K calls, the probability of at least one success is:
This subtracts the probability of K consecutive failures from 1. K counts all attempts, including the first call. The example models the success of a single tool request. It does not estimate whole-task agent success or a model’s empirical pass@k.
Let N be the number of calls actually made. Reaching call j+1 requires that all of the preceding j attempts failed. The mean number of calls is therefore:
With p=0.6 for illustration and equal cost per call, the results are:
| Maximum calls K | Probability of at least one success | Mean calls E[N] | Residual failure probability |
|---|---|---|---|
| 1 | 60.0% | 1.00 | 40.0% |
| 2 | 84.0% | 1.40 | 16.0% |
| 3 | 93.6% | 1.56 | 6.4% |
A cap of three calls does not mean every request uses all three. Stopping at the first success brings the average to 1.56. Single-call success probability remains 0.6. The increase in overall success comes from allowing further attempts.
If p=0, any finite K yields zero probability of success and K calls. A persistent cause, such as an incorrect address or a lasting permissions error, also undermines the independence assumption. An actual agent might revise its request after each observation, changing the success probability. The expression above should not be applied directly as a performance prediction.
#What should stay the same when comparing improvements?
Compare a new harness H₁ with the original H₀ using the same model, tasks, resources, and access to information. If repeated inspection of evaluation scores influenced the choice of H₁, that evaluation set was part of the design process. As in the earlier discussion of automated research, reserve separate tasks for the final comparison.
| Condition to fix or record | Practical meaning |
|---|---|
| Model and generation settings | Use identical weights, versions, and sampling settings |
| Tasks and initial state | Match questions, repository state, initial memory, and tool access |
| Repeated runs | Use a shared, predetermined set of seeds and record external tool variability |
| Budget B | Set model-call, tool-call, token, and time limits in advance |
| Success criterion | Use the same independent check and include timeouts and failed runs in the denominator |
Even a shared seed cannot ensure that remote services and tools behave identically. Repeat runs and report variability. To test memory, match the initial contents, then apply each condition’s update rule. Otherwise answers from earlier evaluations could be available to only one condition.
Index matched tasks and run conditions by i. The following expression summarizes the difference in success between two harnesses under budget B. This is notation for a comparison design and contains no measurements from a paper.
Δ is the mean difference in success across all n runs. Even under the same cap B, actual usage can differ. Record cost and elapsed time for all runs. Comparisons at several values of B can reveal whether a harness helps under tight resource constraints or requires many attempts.
Averaging costs only across successes leaves out resources spent on failures. Likewise, a gain under one budget does not establish an advantage on all tasks and models. An improvement is easier to reuse when the evaluation identifies which component helped in which environment.
ReAct, Generative Agents, and SWE-agent illustrate components of human-designed harnesses. In Reflexion, the specific change examined here is an update to text memory for the next trial. These architectures alone cannot establish that an AI repeatedly improved by designing and replacing its own harness.
To evaluate a claim that AI improved its harness, identify the revisions it proposed. Track the tasks used to verify the candidates, the adoption criterion, and whether an adopted rule actually ran in the next execution. The chain proposal → verification → adoption → new execution, together with an independent performance comparison, establishes who made the change and what it achieved.
#Slide 62 — The target of self-improvement expands to the system
At 1:03:28 the lecturer says that harness adjustment will be covered in the next lecture, and closes at 1:03:40. These two slides introduce a question. They supply neither an automatic harness improvement algorithm nor experimental results.
The supplementary papers give that question concrete form. ReAct connects observations to actions. Generative Agents handles experience as retrievable memories. Reflexion preserves feedback for the next attempt, and SWE-agent designs the interface for actions and observations. Each exposes a component that can change outside the model.
When a system reportedly improved itself, ask three questions: Did the model weights change? Did execution state or memory change? Did the rules of the harness change? Was the change applied on the next run, and did it help on new tasks under comparable conditions?
These distinctions connect a specific change to the capability that improved. The next subject is both the learning signal that helps an agent choose better actions and the architecture that remembers, verifies, and reuses it.
The distinctions from the first half carry into Part 14, the opening of the second half. It examines the self-learning loop formed by a problem setter, solver, and verifier, and distinguishes decreasing training loss from improved ability on new tasks.
Calculations and code
These two educational examples verify memory score normalization and ranking, changes in weights, handling constant columns, and the success probability and expected call count for bounded independent retries. The code illustrates the calculations without running an actual agent or reproducing the papers’ benchmarks.
#Further reading
| Resource | Relevant topic |
|---|---|
| NTU Self-Improving lecture · Slides 61–62 | Agent improvement beyond model parameters |
| ReAct: Synergizing Reasoning and Acting in Language Models | Prompts and execution structure connecting reasoning, actions, and observations |
| Generative Agents: Interactive Simulacra of Human Behavior | Memory streams, retrieval, reflection, and planning |
| Reflexion: Language Agents with Verbal Reinforcement Learning | Using experience on a subsequent trial without a weight update |
| SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering | Designing model commands and environmental feedback |