Skip to main content
#robotics#reinforcement-learning#sim-to-real#real-world-robotics

Deep Reinforcement Learning for Robotics: A Paper Review

A review of “Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes.”

#Paper review

I came across this survey while choosing a paper to review for a class assignment. I wanted to understand where deep reinforcement learning (DRL) has worked on physical robots. Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes

#1) The paper at a glance

  • Paper: Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes
  • Version: arXiv v3, September 16, 2024
  • Two questions guide the survey:
    • How far has DRL progressed on real-world robotics problems?
    • Which areas have matured, and which remain difficult?

The survey classifies DRL research by what robots achieved in physical environments and uses that evidence to assess maturity. Its scope extends beyond comparing algorithms.

#2) The paper’s framework

The authors analyze robotics research along four dimensions.

DimensionWhat it describes
Robotic competencyThe skill learned, including mobility, manipulation, and interaction with humans or other robots
Problem formulationHow states, observations, rewards, and the action space define an RL problem
Solution methodThe training strategy, including simulation, sim-to-real transfer, and real-world learning
Level of real-world successHow closely the evaluation conditions resemble real deployment

#Levels of real-world success

LevelEvidence
L0Evaluation in simulation only
L1Evaluation in a limited set of laboratory conditions
L2Evaluation in diverse laboratory conditions
L3Evaluation in a limited set of real-world conditions
L4Evaluation in diverse real-world conditions
L5Deployment in a commercial product or service

#3) How the presentation is organized

Each of the 30 slide images is accompanied by its main point and my interpretation. In sequence, they introduce the taxonomy, define levels of real-world success, and examine evidence and open questions for each robotic competency.


#4) Slide-by-slide notes

#Slide 01. Title and authors

Title slide for the survey of real-world DRL in robotics, introducing authors from UT Austin, the University of Virginia, and Sony AI.

The presentation introduces a survey of how successfully DRL has been applied to physical robots. The authors represent UT Austin, the University of Virginia, and Sony AI. The taxonomy in the next few slides helps organize the breadth of the review.

#Slide 02. Contents and version history

Presentation contents, from the taxonomy and six competency reviews to research directions, supplementary tables, and the appendix.

The presentation has 13 sections: contents and history, taxonomy, competency review, locomotion, navigation, manipulation, mobile manipulation (MoMa), human–robot interaction (HRI), multi-robot interaction, general trends, key future directions, additional tables, and an appendix. The version history records the initial draft (v0.1, November 10, 2025) and its revision (v0.2, December 1, 2025).

#Slide 03. Why this survey?

Rationale for organizing DRL literature by evidence from real robots, a four-part taxonomy, and progress in transferring learning from simulation.

Earlier surveys often emphasized simulation or focused on a particular technique or task. This survey starts from real-world outcomes, introduces a taxonomy covering competency, formulation, solution, and success level, and places recent DRL advances in the context of transfer from simulation to reality. Together these choices make it possible to ask systematically where the technology has succeeded.

#Slide 04. Taxonomy of robot competencies

Figure 1 organizes locomotion, navigation, manipulation, mobile manipulation, and interaction with humans and other robots.

The hierarchy groups skills into mobility (locomotion and navigation), manipulation, and interaction with other agents (humans or multiple robots). Figure 1 shows how single-robot competencies combine in mobile manipulation and how the taxonomy extends to human and multi-robot interactions.

#Slide 05. Problem formulation

Figure 2 shows the action, reward, and observation loop between an agent and its environment alongside the choices used to formulate RL tasks.

The survey distinguishes low-level joint commands, mid-level task-space actions, and high-level temporally extended actions. It also distinguishes high- and low-dimensional sensor observations and sparse and dense reward functions. Figure 2 places these choices in the standard Markov decision process (MDP) loop between agent and environment.

#Slide 06. Solution approaches, part 1

Figures 3 and 4 show sim-to-real transfer and the pipeline from a training environment and experience tuples to a learned policy and RL agent.

Simulator usage includes zero-shot and few-shot sim-to-real transfer, alongside alternatives such as offline learning and training on physical robots. Methods can also be model-free or model-based. Figure 3 illustrates sim-to-real transfer. Figure 4 follows experience tuples from the training environment through learning to the policy network used by the agent.

#Slide 07. Solution approaches, part 2

The remaining solution dimensions cover expert input, policy optimization, and architectures for representing policies and models.

Three further categories complete the taxonomy. Expert input might come from human demonstrations or an oracle. Policy optimization includes planning and offline, on-policy, or off-policy RL. Policies and models can be represented by MLPs, CNNs, RNNs, or Transformers. Later sections use these categories to compare combinations associated with real-world success.

#Slide 08. Levels of real-world success

A six-level readiness scale, from L0 evaluation only in simulation to L5 deployment in a commercial product or service.

Inspired by technology readiness levels, the authors define levels 0 through 5, from simulation-only tests to commercial deployment. The same algorithm can support very different conclusions depending on where it was evaluated, making these levels central to comparisons across studies.

#Slide 09. Reading the competency reviews and their color key

Color legend for study maturity: light blue for limited lab, blue for diverse lab, teal for limited real, and dark teal for diverse real conditions.

The next sections cover locomotion, navigation, manipulation, MoMa, HRI, and multi-robot interaction. In each table, a reference number carries a color for the maturity of its evaluation. Read light blue as limited laboratory conditions, blue as diverse laboratory conditions, teal as limited real-world conditions, and dark teal as diverse real-world conditions.

#Slide 10. Locomotion overview

References for quadruped locomotion, biped locomotion, and quadrotor flight control, color-coded by evaluation maturity.

The table groups work into legged locomotion (quadrupeds and bipeds) and quadrotor flight control. Quadruped results cover a broad range extending to diverse real-world conditions, giving them the strongest maturity evidence here. Biped and flight studies cluster more heavily in limited settings.

#Slide 11. Locomotion findings

Findings for legged locomotion highlight quadruped maturity, zero-shot sim-to-real transfer, and teacher–student distillation.

DRL for quadruped locomotion is relatively mature. Biped locomotion has more demanding dynamics and many degrees of freedom and is less mature. Successful approaches include zero-shot transfer of on-policy, model-free policies trained in simulation and distillation from a teacher with privileged information to a student.

Open questions include how to learn safely and efficiently on real robots, and how to integrate movement with other high-dimensional, complex, or long-horizon objectives.

#Slide 12. Navigation overview

Navigation references organized by wheeled, legged, and aerial platforms and colored by maturity.

Wheeled navigation has many studies in diverse laboratory or real-world settings. The aerial group contains fewer demonstrations in diverse real environments, illustrating the difference in maturity across platforms.

#Slide 13. Navigation findings

Navigation findings compare end-to-end RL with modular navigation stacks and identify local planning and semantic exploration as promising components.

End-to-end RL has strong simulation results for indoor navigation, while classical modular stacks have the strongest real-world track record in this review. Generalization, interpretability, and safety remain challenges. Local planning and semantic exploration are promising uses for learning.

Open questions include how much of the navigation stack to learn, how to train navigation and locomotion jointly, and what role RL should play in safety-critical applications such as autonomous driving.

#Slide 14. Manipulation overview

Examples and reference groups for pick-and-place, contact-rich manipulation, in-hand manipulation, and non-prehensile manipulation.

The table organizes studies by pick-and-place (grasping, end-to-end systems, and pick-and-place tasks), contact-rich work (assembly, articulated objects, and deformable objects), in-hand manipulation, and non-prehensile manipulation. Reference numbers and colors indicate evidence maturity. The photos show the physical interactions that distinguish the four groups.

#Slide 15. Manipulation findings

Conditions associated with successful RL manipulation and approaches for extending it to open-world tasks.

RL has been most successful when a task is constrained, with known objects and environments, and enumerable in advance, with a definable set of goals and initial conditions. Grasping and in-hand manipulation provide examples.

Extending these results to an open world calls for multi-task, meta-, and lifelong learning, autonomous real-world learning with automatic rewards and resets, learning from human video, and better use of demonstrations.

#Slide 16. Open questions in manipulation

Open problems in incorporating symmetry and collision-avoidance priors and combining isolated manipulation skills into a system.

How should useful priors, such as symmetry and collision avoidance, enter the learning system? Most studies address a single isolated subtask with a specific action space. Combining multiple skills into an integrated system remains a central challenge and a major reason manipulation is still difficult.

#Slide 17. Mobile manipulation overview

Mobile manipulation studies grouped into whole-body control and short- and long-horizon interactive tasks, with maturity indicated by color.

The table separates whole-body control (WBC), short-horizon interactive tasks, and long-horizon interactive tasks. Most long-horizon work remains at the limited-laboratory level. Coordinating motion and manipulation over extended tasks is particularly demanding.

#Slide 18. Mobile manipulation findings

Short-horizon sim-to-real results and open challenges for mobile manipulation, including multitasking, long-term memory, and safe exploration.

Short-horizon tasks show early sim-to-real successes. Action-space choices have a substantial effect on performance, and transfer across different robot morphologies matters. Multitasking, long-term memory, and safe exploration remain three bottlenecks for broader real-world use.

#Slide 19. Human–robot interaction overview

Physical HRI references grouped into non-collaborative interaction, collaboration, and shared autonomy, with evaluation maturity shown for each.

The table divides physical human–robot interaction (pHRI) into non-collaborative interaction, such as avoiding people in a crowd, collaborative work, and shared autonomy. Relatively few studies reach diverse laboratory conditions or beyond, illustrating the additional difficulty of evaluating RL when people participate.

#Slide 20. Human–robot interaction findings

Challenges of gathering human interaction data and research directions for safer physical learning and more realistic human behavior models.

The review finds fewer successes than for single-robot skills. Gathering human interaction data is costly, and behavior can be non-Markovian and exhibit bounded rationality. Enabling safe real-world learning alongside people and building more realistic simulations of human behavior are two priorities for expanding DRL in HRI.

#Slide 21. Multi-robot interaction overview

Photos and color-coded references for multi-robot collision avoidance, loco-manipulation, and robot soccer.

The slide groups studies into collision avoidance, loco-manipulation, and robot soccer. The robot soccer study numbered 191 reaches diverse real-world conditions, as do some collision-avoidance studies. Overall, the field’s evaluation maturity remains relatively low.

#Slide 22. Multi-robot interaction findings

Research challenges in multi-robot communication, learning stability, scaling, and extending beyond cooperative settings.

Homogeneous teams have achieved results in cooperative settings, but complexity and scalability remain substantial challenges. Priorities include inter-agent communication, learning convergence and stability, and extension to general situations where agents may not cooperate.

Comparison of relatively mature competencies and less mature interaction tasks, with learning approaches common in successful systems.

Locomotion and parts of navigation and manipulation have stronger real-world evidence than mobile manipulation, HRI, and multi-robot interaction. Among the approaches commonly seen in mature systems are zero-shot sim-to-real transfer, carefully engineered dense rewards, and on-policy learning where feasible.

#Slide 24. Key future directions

Research directions in reward and action-space design, classical model-based methods, benchmarking, and foundation models.

The survey calls for principled reward and action-space design, integration with classical model-based methods, and standardized benchmarking. Foundation models offer possible routes to generalization, language conditioning, and the generation of rewards and simulation assets.

#Slide 25. Additional table on problem formulation (Table 1)

Table 1 compares action spaces, observations, and rewards used to formulate robotics tasks in the reviewed literature.

Table 1 arranges studies by the action, observation, and reward choices that define their RL problems. It makes it possible to compare which formulations appear most often for a given task.

#Slide 26. Additional table on problem formulation (Table 1, continued)

Continuation of Table 1, applying the same action, observation, and reward categories to additional studies.

The table extends the same criteria to more studies. When presenting a proposed task, I can use it to find work with comparable rewards, observations, and actions and cite that evidence for the formulation.

#Slide 27. Additional table on problem formulation (Table 2)

Table 2 compares problem formulations across domains, including reward density and observation dimensionality.

Table 2 adds another view of formulation choices and their similarities and differences across domains. In particular, it shows where studies rely on sparse or dense rewards and how the dimensionality of observations varies.

#Slide 28. Additional table on problem formulation (Table 2, continued)

Continuation of Table 2 with further references supporting cross-domain comparisons of problem formulations.

The continuation gives the presentation’s conclusions a broader base of evidence. In summarizing the review, I can connect these entries to a concrete decision about why my chosen task needs a particular formulation.

#Slide 29. Additional table on solution approaches (Table 3)

Table 3 organizes studies by solution approach, including sim-to-real transfer, model-free or model-based learning, and policy optimization.

Table 3 groups studies by their learning strategies. It is the main appendix reference for identifying which combinations of training methods and pipeline choices have produced successful results.

#Slide 30. Additional table on solution approaches (Table 3, continued)

Final portion of Table 3, completing the comparison of training pipelines used across the reviewed studies.

The last slide completes the overview of methods. In my conclusion, I can use these comparisons to justify a concrete training strategy for a future project, such as zero-shot sim-to-real transfer or fine-tuning on a physical robot.


#5) My conclusions (draft)

  1. DRL has demonstrated success on physical robots, but the maturity of the evidence varies widely by task.
  2. Many successful systems combine tasks amenable to sim-to-real transfer with careful engineering of rewards, domain randomization, and action spaces.
  3. Major research challenges are stabilizing learning in the real world, integrating skills over long horizons, and generalizing interactions with humans or multiple agents.

#6) Paper, presentation, and research groups

Connect