#Paper review
I came across this survey while choosing a paper to review for a class assignment. I wanted to understand where deep reinforcement learning (DRL) has worked on physical robots. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
#1) The paper at a glance
- Paper: Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
- Version: arXiv v3, September 16, 2024
- Two questions guide the survey:
- How far has DRL progressed on real-world robotics problems?
- Which areas have matured, and which remain difficult?
The survey classifies DRL research by what robots achieved in physical environments and uses that evidence to assess maturity. Its scope extends beyond comparing algorithms.
#2) The paper’s framework
The authors analyze robotics research along four dimensions.
| Dimension | What it describes |
|---|---|
| Robotic competency | The skill learned, including mobility, manipulation, and interaction with humans or other robots |
| Problem formulation | How states, observations, rewards, and the action space define an RL problem |
| Solution method | The training strategy, including simulation, sim-to-real transfer, and real-world learning |
| Level of real-world success | How closely the evaluation conditions resemble real deployment |
#Levels of real-world success
| Level | Evidence |
|---|---|
| L0 | Evaluation in simulation only |
| L1 | Evaluation in a limited set of laboratory conditions |
| L2 | Evaluation in diverse laboratory conditions |
| L3 | Evaluation in a limited set of real-world conditions |
| L4 | Evaluation in diverse real-world conditions |
| L5 | Deployment in a commercial product or service |
#3) How the presentation is organized
Each of the 30 slide images is accompanied by its main point and my interpretation. In sequence, they introduce the taxonomy, define levels of real-world success, and examine evidence and open questions for each robotic competency.
#4) Slide-by-slide notes
#Slide 01. Title and authors

The presentation introduces a survey of how successfully DRL has been applied to physical robots. The authors represent UT Austin, the University of Virginia, and Sony AI. The taxonomy in the next few slides helps organize the breadth of the review.
#Slide 02. Contents and version history

The presentation has 13 sections: contents and history, taxonomy, competency review, locomotion, navigation, manipulation, mobile manipulation (MoMa), human–robot interaction (HRI), multi-robot interaction, general trends, key future directions, additional tables, and an appendix. The version history records the initial draft (v0.1, November 10, 2025) and its revision (v0.2, December 1, 2025).
#Slide 03. Why this survey?

Earlier surveys often emphasized simulation or focused on a particular technique or task. This survey starts from real-world outcomes, introduces a taxonomy covering competency, formulation, solution, and success level, and places recent DRL advances in the context of transfer from simulation to reality. Together these choices make it possible to ask systematically where the technology has succeeded.
#Slide 04. Taxonomy of robot competencies

The hierarchy groups skills into mobility (locomotion and navigation), manipulation, and interaction with other agents (humans or multiple robots). Figure 1 shows how single-robot competencies combine in mobile manipulation and how the taxonomy extends to human and multi-robot interactions.
#Slide 05. Problem formulation

The survey distinguishes low-level joint commands, mid-level task-space actions, and high-level temporally extended actions. It also distinguishes high- and low-dimensional sensor observations and sparse and dense reward functions. Figure 2 places these choices in the standard Markov decision process (MDP) loop between agent and environment.
#Slide 06. Solution approaches, part 1

Simulator usage includes zero-shot and few-shot sim-to-real transfer, alongside alternatives such as offline learning and training on physical robots. Methods can also be model-free or model-based. Figure 3 illustrates sim-to-real transfer. Figure 4 follows experience tuples from the training environment through learning to the policy network used by the agent.
#Slide 07. Solution approaches, part 2

Three further categories complete the taxonomy. Expert input might come from human demonstrations or an oracle. Policy optimization includes planning and offline, on-policy, or off-policy RL. Policies and models can be represented by MLPs, CNNs, RNNs, or Transformers. Later sections use these categories to compare combinations associated with real-world success.
#Slide 08. Levels of real-world success

Inspired by technology readiness levels, the authors define levels 0 through 5, from simulation-only tests to commercial deployment. The same algorithm can support very different conclusions depending on where it was evaluated, making these levels central to comparisons across studies.
#Slide 09. Reading the competency reviews and their color key

The next sections cover locomotion, navigation, manipulation, MoMa, HRI, and multi-robot interaction. In each table, a reference number carries a color for the maturity of its evaluation. Read light blue as limited laboratory conditions, blue as diverse laboratory conditions, teal as limited real-world conditions, and dark teal as diverse real-world conditions.
#Slide 10. Locomotion overview

The table groups work into legged locomotion (quadrupeds and bipeds) and quadrotor flight control. Quadruped results cover a broad range extending to diverse real-world conditions, giving them the strongest maturity evidence here. Biped and flight studies cluster more heavily in limited settings.
#Slide 11. Locomotion findings

DRL for quadruped locomotion is relatively mature. Biped locomotion has more demanding dynamics and many degrees of freedom and is less mature. Successful approaches include zero-shot transfer of on-policy, model-free policies trained in simulation and distillation from a teacher with privileged information to a student.
Open questions include how to learn safely and efficiently on real robots, and how to integrate movement with other high-dimensional, complex, or long-horizon objectives.
#Slide 12. Navigation overview

Wheeled navigation has many studies in diverse laboratory or real-world settings. The aerial group contains fewer demonstrations in diverse real environments, illustrating the difference in maturity across platforms.
#Slide 13. Navigation findings

End-to-end RL has strong simulation results for indoor navigation, while classical modular stacks have the strongest real-world track record in this review. Generalization, interpretability, and safety remain challenges. Local planning and semantic exploration are promising uses for learning.
Open questions include how much of the navigation stack to learn, how to train navigation and locomotion jointly, and what role RL should play in safety-critical applications such as autonomous driving.
#Slide 14. Manipulation overview

The table organizes studies by pick-and-place (grasping, end-to-end systems, and pick-and-place tasks), contact-rich work (assembly, articulated objects, and deformable objects), in-hand manipulation, and non-prehensile manipulation. Reference numbers and colors indicate evidence maturity. The photos show the physical interactions that distinguish the four groups.
#Slide 15. Manipulation findings

RL has been most successful when a task is constrained, with known objects and environments, and enumerable in advance, with a definable set of goals and initial conditions. Grasping and in-hand manipulation provide examples.
Extending these results to an open world calls for multi-task, meta-, and lifelong learning, autonomous real-world learning with automatic rewards and resets, learning from human video, and better use of demonstrations.
#Slide 16. Open questions in manipulation

How should useful priors, such as symmetry and collision avoidance, enter the learning system? Most studies address a single isolated subtask with a specific action space. Combining multiple skills into an integrated system remains a central challenge and a major reason manipulation is still difficult.
#Slide 17. Mobile manipulation overview

The table separates whole-body control (WBC), short-horizon interactive tasks, and long-horizon interactive tasks. Most long-horizon work remains at the limited-laboratory level. Coordinating motion and manipulation over extended tasks is particularly demanding.
#Slide 18. Mobile manipulation findings

Short-horizon tasks show early sim-to-real successes. Action-space choices have a substantial effect on performance, and transfer across different robot morphologies matters. Multitasking, long-term memory, and safe exploration remain three bottlenecks for broader real-world use.
#Slide 19. Human–robot interaction overview

The table divides physical human–robot interaction (pHRI) into non-collaborative interaction, such as avoiding people in a crowd, collaborative work, and shared autonomy. Relatively few studies reach diverse laboratory conditions or beyond, illustrating the additional difficulty of evaluating RL when people participate.
#Slide 20. Human–robot interaction findings

The review finds fewer successes than for single-robot skills. Gathering human interaction data is costly, and behavior can be non-Markovian and exhibit bounded rationality. Enabling safe real-world learning alongside people and building more realistic simulations of human behavior are two priorities for expanding DRL in HRI.
#Slide 21. Multi-robot interaction overview

The slide groups studies into collision avoidance, loco-manipulation, and robot soccer. The robot soccer study numbered 191 reaches diverse real-world conditions, as do some collision-avoidance studies. Overall, the field’s evaluation maturity remains relatively low.
#Slide 22. Multi-robot interaction findings

Homogeneous teams have achieved results in cooperative settings, but complexity and scalability remain substantial challenges. Priorities include inter-agent communication, learning convergence and stability, and extension to general situations where agents may not cooperate.
#Slide 23. General trends

Locomotion and parts of navigation and manipulation have stronger real-world evidence than mobile manipulation, HRI, and multi-robot interaction. Among the approaches commonly seen in mature systems are zero-shot sim-to-real transfer, carefully engineered dense rewards, and on-policy learning where feasible.
#Slide 24. Key future directions

The survey calls for principled reward and action-space design, integration with classical model-based methods, and standardized benchmarking. Foundation models offer possible routes to generalization, language conditioning, and the generation of rewards and simulation assets.
#Slide 25. Additional table on problem formulation (Table 1)

Table 1 arranges studies by the action, observation, and reward choices that define their RL problems. It makes it possible to compare which formulations appear most often for a given task.
#Slide 26. Additional table on problem formulation (Table 1, continued)

The table extends the same criteria to more studies. When presenting a proposed task, I can use it to find work with comparable rewards, observations, and actions and cite that evidence for the formulation.
#Slide 27. Additional table on problem formulation (Table 2)

Table 2 adds another view of formulation choices and their similarities and differences across domains. In particular, it shows where studies rely on sparse or dense rewards and how the dimensionality of observations varies.
#Slide 28. Additional table on problem formulation (Table 2, continued)

The continuation gives the presentation’s conclusions a broader base of evidence. In summarizing the review, I can connect these entries to a concrete decision about why my chosen task needs a particular formulation.
#Slide 29. Additional table on solution approaches (Table 3)

Table 3 groups studies by their learning strategies. It is the main appendix reference for identifying which combinations of training methods and pipeline choices have produced successful results.
#Slide 30. Additional table on solution approaches (Table 3, continued)

The last slide completes the overview of methods. In my conclusion, I can use these comparisons to justify a concrete training strategy for a future project, such as zero-shot sim-to-real transfer or fine-tuning on a physical robot.
#5) My conclusions (draft)
- DRL has demonstrated success on physical robots, but the maturity of the evidence varies widely by task.
- Many successful systems combine tasks amenable to sim-to-real transfer with careful engineering of rewards, domain randomization, and action spaces.
- Major research challenges are stabilizing learning in the real world, integrating skills over long horizons, and generalizing interactions with humans or multiple agents.