Reinforcement Learning 11 — Problems Robots Must Solve Beyond the Lab

A robot grasps a cup in the lab. How does the same policy fare the next day under different lighting, a shifted camera, and a thin glass cup? Changing all three at once makes failures difficult to diagnose. This final chapter turns questions behind impressive demonstrations into concrete research problems. Previous chapter: Embodied Reasoning

This is week eleven on ETH’s official schedule and Oier Mees’s final main lecture. He introduces it as Lecture 12 and begins with information about a policy-testing PC for demo day. The Korean transcript was compared with English captions; P numbers below follow copied slides including the cover. ETH schedule · Official PDF · Slide-copy index

#From impressive demonstrations to generalization experiments

#P002 · What evidence supports weekly claims that robotics is solved?

The lecture begins with social-media claims of having solved robotics. Rather than dismissing announcements, it asks what videos actually establish. Their meaning depends on the robot and objects, trial counts including failures, and whether testing was repeated in a new room.

#P003 · Which changes degrade performance outside the lab?

The lecturer shows older humanoid and mobile-kitchen-manipulation videos, not a ranking of current state-of-the-art performance. They introduce how easily long everyday tasks go wrong under lighting, objects, and viewpoints differing from training. Slow dinner preparation can include failures in navigation, recognition, grasping, and recovery.

#P004 · How to make an impressive demo—and what it evaluates

Mees’s satirical recipe is to collect many demonstrations on the target robot, repeat behavior cloning until it works, and film under the same day’s lighting and camera setup. Adding previous failures to training can turn them into interpolation problems among familiar examples. Handling another room and unfamiliar tools together requires separate tests. People recombine knowledge of drawers and utensils in new kitchens; this is the compositional generalization the lecture seeks. Original lecture, 4:06 · Slide P004 · Slide P005

#P005 · Generalization across one object, one room, one building, and the world

Collecting more target-platform data can bring previous failures inside training coverage and improve performance. This is useful engineering, but differs from recombining knowledge in an unseen kitchen. Before calling a policy general-purpose, specify which axes changed: objects, rooms, buildings, embodiments, or instructions.

Use a baseline with the same camera and objects, then change only the camera and only the object in separate conditions, and finally both. The equation below is an educational way to report each condition’s success rate and decline from baseline.

Record trial counts and success definitions too. Holding a lifted cup for two seconds and transporting it to a specified location upright can give different results.

#Choosing backbones and data combinations separately

#P006 · Combining broad semantic knowledge with body-specific skills

A backbone is the model’s central component converting observations or language into representations for action prediction. The lecture compares three starting points. VLMs help represent names and spatial relations; video models learn changes across scenes; training from robot data directly addresses deployed actions and contacts but carries collection costs. Recognizing a glass or generating plausible footage of it falling does not establish how it actually tips under fingertip contact. Original lecture, 6:50 · Slide P006 · Slide P008

#P007 · Embodied experience connects sensation, space, and time

Describing a glass differs from judging which push makes it fall or what force breaks it. Plausible generated video does not guarantee consistent mass, contact, or object permanence. The lecture proposes strengthening physical grounding through experience linking multiple senses with action outcomes.

#P008 · Comparing VLMs, video models, and robot-data training

These approaches differ in their starting information: semantic object/instruction knowledge, temporal visual change, or directly paired sensors and actions. None is established as always superior. Match final demonstration counts, compute budgets, and observations to separate backbone benefits from simply using more data.

#P009 · Three-stage and two-stage paths through web data, simulation, and reality

Data ordering offers choices: learn language and objects from web data, add simulated interactions, then adapt to reality; or spend simulation-building time collecting real robot data instead. A video-centered path first learns motion from Internet and human egocentric videos, then connects robot actions. Comparing backbones requires matching final robot demonstration counts and physical tests. Original lecture, 11:18 · Slide P009 · Slide P010 · Scott Reed’s invited talk

#P010 · From general video through human first-person views to robot actions

Web→simulation→reality provides many inexpensive interactions but retains world-building costs and the reality gap. Web→reality spends that effort on physical collection. Video-centered paths add human first-person tasks after Internet video before connecting robot actions. Robot-centered approaches directly learn sensors and actions without Internet pretraining. Compare the order and the stage at which action labels enter.

#P011 · Collection interfaces differ in costs and alignment

#P012 · The trade-off between data scalability and hardware alignment

Everyday first-person video scales across locations, but five-fingered motions are not gripper commands. Handheld grippers and gloves narrow the gap while supporting mobile collection. Direct teleoperation best matches the deployed body, yet resets, operating time, and maintenance become bottlenecks. The thousands-to-tens-of-millions-of-hours axis conceptually contrasts scales; it does not guarantee actual holdings for every project.

In leader–follower operation, the robot arm follows a human-moved leader arm. Mees notes its value for precise demonstrations alongside increased breakdown and maintenance costs as hardware grows. Video scales more easily but retains the human–robot hand gap. The slide’s time axis is conceptual. Original lecture, 13:50 · Slide P011 · Slide P012

#Why cameras alone can be insufficient for grasping a mug

#P013 · What dexterity and generality need in common

Dexterity can benefit from fast control and contact feedback; broad generalization from varied experience and large representations. Slow large models and scarce tactile data create tension. Mobility matters too: robots visiting several rooms encounter more variation than arms fixed to one table. The goal is to combine specialist dexterity with broad knowledge.

#P014 · A broken mug: seeing well versus grasping well

During his PhD, Mees estimated a mug’s 3D shape and pose from images to grasp its partly occluded handle. The robot missed the handle, grasped the body, and broke it with excessive force. The example concerns feedback after contact. Cameras may show apparent contact, while force and tactile sensors indicate actual loads or slipping. Proprioception measures the robot’s own body, such as joint angles. Original lecture, 19:24 · Slide P013 · Slide P014

#P015 · Will joint training across all senses solve it?

Recall native multimodal learning from Chapter 7. Vision, touch, force, and joint states can become tokens learned in one model. A common vector interface alone is insufficient: sensor semantics, temporal alignment, and available simultaneous measurements determine learnability.

#P016 · Separate scarcity, missing paired data, and sensor heterogeneity

First, tactile and force data are scarce because sensors are costly and damage is possible. Second, even if separate modality datasets exist, synchronized RGB, depth, touch, and force are rarer. Third, robot-specific sensors impede transfer of recorded values. These are distinct problems. More audio data do not automatically create paired touch and vision for the same contact.

#P017 · Can separately observed senses be connected?

Separate vision–language and touch–language data can share descriptions such as rough or soft. Language carries common meaning rather than perfectly reconstructing every sensory measurement. A few words cannot preserve every material or pressure difference, so connections need evaluation under relevant tasks and sensors.

#P018 · FuSe: language as a bridge between senses

One multimodal model needs sufficient sensor data aligned to common events, yet expensive tactile sensors have scarce, heterogeneous outputs. FuSe uses natural language to connect separately collected sensory data. Finding a brown object that feels like a pineapple requires visual color and tactile texture. Describing an object inside a bag shows information touch adds when cameras cannot see. This demonstrates cross-sensory behavior in particular experiments. Original lecture, 21:21 · Slide P016 · Slide P018 · Slide P019 · FuSe paper

#P019 · Prompts combining color, touch, and sound

An object that is brown and feels like a pineapple must satisfy both conditions. Vision alone may leave several brown candidates; touch alone may include differently colored objects with the same texture. Feeling and describing an object in a bag uses information unavailable to the camera. This motivates fusion without establishing universal handling of unfamiliar sensors from one demonstration.

#Keep moving, think longer, or stop?

#P020 · Familiar, uncertain, and out-of-scope situations

The lecture proposes fast policy execution for familiar tasks, Chapter 10’s additional reasoning for uncertain judgments, and stopping for human help beyond the system’s ability. The numerical example below illustrates this. A confidence band of 0.8 is calibrated if roughly 80% of such cases succeed. Contact risks and safety-limit violations require independent stopping conditions. Original lecture, 24:40 · Slide P020 · Slide P021

#P021 · How can a model detect what it does not know?

Adaptive computation assumes uncertainty can first be detected. High token probability cannot directly be read as task-success probability. A familiar-looking scene may have different friction, camera calibration, or embodiment. Even larger models can confidently complete the wrong pattern. The confidence-threshold example explains design principles rather than reports a completed out-of-distribution detector.

A familiar cup may still come with changed sensors or a shifted workbench. Independent changes in objects, locations, sensors, bodies, and instructions can produce confident errors. Alongside success, record continuations in dangerous situations, unnecessary stops during normal tasks, and recovery after human intervention to evaluate routing decisions.

#Making deployment experience improve the next policy

#P022 · Limits of policies that imitate demonstrations

If demonstrations always follow states 1→2→3, behavior cloning learns that sequence well. If separate episodes contain a path to an intermediate point and a path from there to the goal, offline reinforcement learning can evaluate stitching them together using rewards. Unobserved shortcuts may be overvalued, making transition coverage and reward definitions essential. Original lecture, 28:49 · Slide P022 · Slide P023

#P023 · Which experiences can reinforcement learning combine?

Behavior cloning trained only on 1→2→3 lacks evidence to reinforce a direct 1→3 action. Rewards and transition experience support comparing returns of alternative connections. Offline RL still cannot invent arbitrary physical transitions absent from data. Algorithms must handle exploration cost, failures, and overestimation outside data coverage.

#P024 · A cycle linking better policies and more experience

A data flywheel feeds deployment experience into training, then uses improved policies to collect broader experience. Expert teleoperation may be smooth and brief, while autonomous records contain hesitation, failures, and recovery. Treating all as correct demonstrations can teach pre-failure actions too. Record failures and interventions separately and reevaluate old tasks after learning. Losing existing skills while learning new ones is catastrophic forgetting. Original lecture, 31:00 · Slide P024 · Slide P025 · Slide P026 · Sutton’s The Bitter Lesson

#P025 · Can expert demonstrations and autonomous records be mixed unchanged?

Autonomous runs may spend many frames hesitating where experts act briefly. Uniform imitation can overrepresent slow stationary behavior and treat commands preceding failures as targets. Consider recording frequency, trajectory length, success labels, and intervention locations alongside quality. Failure data can still teach outcomes or values, motivating alternative learning signals rather than discarding experience.

#P026 · Learning after deployment without forgetting

Lifelong learning means more than storing experience. Capabilities must update, retain earlier skills, and improve over time, beyond the common train→freeze→deploy cycle. After learning a new refrigerator, reevaluate earlier drawer opening. Rapid local adaptation and long-term weight learning can operate on different timescales.

#Connecting autonomous collection to actual improvement

In the autonomous-improvement example, foundation models propose tasks based on scene affordances and evaluate success afterward. An affordance is an action possibility such as pulling a drawer or pressing a button. Repeatedly choosing easy tasks can accumulate successes without broadening experience. Task-selection criteria and success-evaluation quality must be examined together to establish policy improvement. Original lecture, 41:44 · Autonomous-improvement research

A person finishing in three seconds and a robot hesitating for fifteen seconds can reach the same final state with different movement distributions. Separately recording failures and human recovery lets data serve as imitation targets, outcome-prediction experience, or reward-ranked paths. Rechecking improvement with the same evaluation is essential to the flywheel.

#Memory needed to adapt to one refrigerator

#P027 · Putting a verbal correction or demonstration into context

If fetching a cola fails only because the handle estimate is wrong, “grasp slightly farther left” may suffice. If opening itself is unfamiliar, a demonstration may help more. In-context adaptation places such feedback or demonstrations in the current input to change behavior quickly. Its timescale differs from lifelong weight learning. PALO illustrates few-demonstration task decomposition and adaptation. Original lecture, 35:30 · Slide P027 · Slide P032 · PALO project

#P028 · Representations combining navigation and manipulation

A mobile robot working across rooms needs information exchange between simultaneous localization and mapping (SLAM) and manipulation. Topological room connections may suffice in corridors, while physical distances matter near doors or narrow shelves. VLMaps places vision–language features in 3D maps for language-based localization. LeLaN learns language-goal navigation from sources including human walkthrough videos without a prebuilt map. Suitability depends on distance, environmental change, and required arrival precision. Original lecture, 37:09 · Slide P028 · Slide P030 · VLMaps paper · LeLaN paper

Modular systems use SLAM, localization, and planning during navigation before switching to manipulation. An integrated model must choose between explicit location/map state and learned representations from image history. Remembering routes through a building differs in timescale from avoiding a new nearby obstacle. Maintaining maps or scene graphs while local control reacts to fresh observations is another option.

#P029 · Preserving seconds of sensor data and years of experience

Whole-body policies need not store all memory in network weights. Nearby-obstacle information and routes to rooms on another floor need different precision. Maps and walkthrough context require timestamps and update rules when furniture moves. What to record or forget, and how to retrieve sensations across time and space, are separate research questions.

Catching a slipping object needs immediate sensor history. Finding where a cup was moved weeks earlier may need longer-term place-and-event records. Calculating storage for every raw camera frame illustrates why memory must select and summarize events. Original lecture, 39:36 · Slide P029

#What should be remembered for longer?

Short-term memory can retain detailed hand–cup motion; long-term memory can store events such as placing a cup on the right shelf that morning. Events must link object identity, observation time, and subsequent movement. If the cup moves again, old positions must be distinguished from current ones. Such updates matter in prolonged home interaction.

The lecture considers explicit 3D maps, bird’s-eye views, topological room maps, object-relation scene graphs, and walkthrough videos. Compare retrieval accuracy, goal-reaching success, and update costs after environmental changes. Repeating an instruction after furniture moves can reveal whether old memory is blindly reused or corrected with observations.

#P030 · VLMaps and LeLaN represent environments differently

VLMaps links visual–language features with 3D positions to locate goals such as “near the chair.” LeLaN connects language with movement from sources including human walkthrough videos, learning navigation without a prebuilt map. Map-free approaches still require observations and experience supporting movement decisions. Compare travel distance, arrival precision, and scene-change conditions alongside map use.

#P031 · Foundation models proposing tasks and evaluating outcomes

Imagine a model seeing a drawer handle, proposing opening it, and checking afterward whether it opened. Task proposals determine exploration; success evaluation supplies learning signals. Both can be wrong. Autonomous improvement requires checking both expanded useful task coverage and whether automatic-scoring errors accumulate.

#P032 · PALO: recomposing existing skills from a few demonstrations

The slide compares PALO with roughly five demonstrations against fine-tuning using about a hundred. PALO combines task decomposition and language optimization with an existing generalist policy. These numbers describe data efficiency in that long-horizon evaluation, not a law that every in-context model solves any task after five examples. PALO team

#P033 · Reasoning, dexterous mobile manipulation, lifelong learning, and rapid adaptation

The four elements depend on one another. Reasoning can choose a subtask yet fail at contact without touch and fast control. Mobile collection offers new experience, but memory and learning systems are needed to reuse it. The aim goes beyond assembling good demonstrations of each element to making them work together in unfamiliar environments.

#P034 · Which boundary will these tools help extend?

Chapters 1–11 connect control, MDPs, imitation and reinforcement learning with generative models, sequences, world models, generalist policies, and reasoning. The final question is not which tool is newest, but which tool and experiment address the failure just identified.

#P035 · Turning open problems into research plans

#Research that reveals where failures begin

#P036 · Incremental research, failed ideas, and simple designs

Mees advises choosing an important problem together with a testable plan. Method-driven research may start with a new video model and seek applications. Problem-driven research can begin with camera-viewpoint failures and retain that question even if its first method fails. Before spending months, act as your own critical Reviewer 2 and find the quickest experiment that could disprove your claim. Original lecture, 44:14 · Slide P036 · Slide P037 · Slide P040

The example of ten ideas becoming five experiments, two interesting results, and one paper is an experiential analogy, not measured statistics on research success. Recording failures clarifies assumptions even when attempts do not immediately produce publications. If a method needs seventeen components, also ask whether a simpler version captures much of the benefit.

#P037 · Both an important problem and a testable approach are needed

The lecture contrasts important goals without approaches, such as curing cancer, with narrow benchmark gains lacking a clear substantive problem. Small improvements are not inherently meaningless. The point is that a number alone struggles to demonstrate importance without explaining the cost or failure reduced. First state what success would newly make possible.

For example, recovering from camera reinstallation with a small calibration set instead of rerecording dozens of demonstrations makes the saved cost concrete. Even a small gain under fixed conditions needs uncertainty and repeat counts. Explain both why the problem matters and how to approach it.

#P038 · Interest sustains long failures and repeated attempts

Mees notes that many ideas disappear in experiments and much research accumulates small results. Simple designs are easier to reproduce and scale. Choosing a problem that interests you helps sustain long cycles of failure and repetition. This advice comes from his research experience.

#P039 · Examine the likeliest failure reasons first

Finding data leakage or a weak baseline only after months is costly. List strong objections before starting: could gains come only from augmentation, disappear with changed objects, or exceed real-time compute budgets? Small early tests of those objections protect time for later stages.

#P040 · Method-driven and problem-driven research start differently

One seeks applications for a new video model; the other seeks tools to reduce viewpoint-change failures. Both can succeed, but the latter preserves an important question when a first method fails. This motivates the lecturer’s recommendation of problem-driven PhD research.

#P041 · Overfit a small sample and inspect inputs and outputs directly

For camera-viewpoint failures, deliberately overfit a few demonstrations first to verify the input-to-action connection. Visualize crops, coordinates, and actual commands. Then compare a baseline and new method with only the camera changed before extending to unseen rooms and objects. Share code and data where possible, make comparison conditions visible in figures, and limit claims to demonstrated coverage. Original lecture, 51:44 · Slide P041 · Slide P042 · Slide P043

#P042 · Discuss with colleagues and share code, data, and figures

Explaining conditions to colleagues or cited authors can reduce repetition of known failures. Publish code and data where possible and show inputs, outputs, and comparisons directly in figures. Readers should understand the tested claim before reading the entire article.

#P043 · Communicate research without exceeding demonstrated scope

Do not assume the audience shares your background; explain unfamiliar terms and conditions first. arXiv and social media make findings usable by others, but expanding one success into “robotics solved” erases evaluation scope. The opening critique returns as a standard for communicating one’s own research.

#P044 · Finding research questions in a failing complex pipeline

A week before a PhD deadline, Mees’s robot stalled during automatic calibration. After a lucky late recovery, experiments were rushed through the final night. He later spent about a year preparing a Franka, custom workbench, VR interface, and wrist camera. Slow physical iteration motivated a simulator with the same manipulation workflow; that side project became CALVIN for language-conditioned long-horizon manipulation. Later work trained one policy on more than 25 real tasks. The simulation benchmark and physical-robot study are separate results. Original lecture, 54:54 · Slide P044 · Slide P046 · Slide P047 · CALVIN paper · HULC++ paper

#P045 · A year spent building the Franka experimental setup

Before research, the setup required a custom metal workbench, VR teleoperation code, cameras, a 3D-printed wrist mount, and low-level controllers. This infrastructure rarely appears in algorithm plots but determines collection accuracy and iteration speed. His wish to simplify a complex ROS pipeline is personal experience, not a conclusion that every robot system should remove ROS.

#P046 · A tool for faster physical iteration becomes CALVIN

Simulation sharing the real manipulation workflow created a foundation for evaluating language-conditioned long-horizon tasks. Mees recalls releasing this side project, which became the widely used, award-winning CALVIN benchmark. Important research can emerge from removing experimental bottlenecks rather than only pursuing the original main idea. Benchmark achievements and subsequent physical-robot results remain separate evaluations.

#P047 · Beyond 25 real tasks toward unfamiliar task combinations

The final demonstration shows the goal of chaining language-directed manipulations and generalizing to unseen combinations. The Korean transcript says “about five years,” but the slide says “five papers” before the final demo. Either way, the point is accumulated environment, data, evaluation, and policy development rather than one completed training run. Playback labels such as 2× speed must also be distinguished from actual completion times.

#P048 · Student feedback helping improve a new course

#P049 · Thanks to the assistants who built exercises and projects

The lecture closes by thanking students for feedback, teams for data sharing, and teaching assistants for practical environments. The official schedule lists a separate invited lecture for the following week. Original lecture, 60:10 · Slide P048 · Slide P049 · Lucas Beyer’s invited talk

#Further reading

• Architectures connecting prediction and planning. LeCun, A Path Towards Autonomous Machine Intelligence
• Long-term advantages of methods exploiting more computation and experience. Sutton, The Bitter Lesson
• A robotics perspective centered on interaction with the real world. Brooks, Intelligence without Representation

First chapter: What Does a Robot Learn?