A Systematic Review of Deep Reinforcement Learning for Legged Robot Locomotion

Abstract

Legged robot locomotion remains a critical challenge in robotics, demanding control strategies that are not only dynamically stable and robust but also capable of adapting to complex and changing environments. deep reinforcement learning (DRL) has recently emerged as a powerful approach to automatically generate motion control policies by learning from interactions with simulated or real environments. This study provides a systematic overview of DRL applications in legged robot control, emphasizing experimental platforms, measurement techniques, and benchmarking practices. Following PRISMA guidelines, 27 peer-reviewed studies published between 2018 and 2025 were analyzed, covering model-free, model-based, hierarchical, and hybrid DRL frameworks. Our findings reveal that reward shaping, policy representation, and training stability significantly influence control performance, while domain randomization and dynamic adaptation methods are essential for bridging the simulation-to-real-world gap. In addition, this review highlights instrumentation approaches for evaluating algorithm effectiveness, offering insights into sample efficiency, energy management, and safe deployment. The results aim to guide the development of reproducible and experimentally validated DRL-based control systems for legged robots.

1. Introduction

Legged robots have garnered significant attention for their flexible gait and powerful obstacle-crossing capabilities [1]. To enable them to walk stably in complex environments like animals, researchers have begun utilizing DRL, an artificial intelligence approach, to allow robots to master motion patterns through trial and error and self-optimization [2]. Unlike traditional control methods that rely on precise dynamic models, DRL can directly learn the mapping relationship between perception, decision-making, and control from sensor data, achieving integrated intelligent control from data to action.

Figure 1 shows the difference between traditional and ML robot walking control. The traditional control includes three loops: balance control, motion trajectory control, and motor control. To execute these controls, external visual signals and the robot’s own sensors are used to acquire environmental and self-state information, followed by feature extraction to extract valuable information.

Drive mechanisms such as electric motors, hydraulic systems, or pneumatic actuators affect the robot’s responsiveness, power, and accuracy. As shown in Figure 2, common drive methods for multi-legged robots mainly include electric motor drive [4], hydraulic drive [5], and pneumatic drive [5]. Traditional control systems guide movement through visual and sensor information processing, while different types of drive mechanisms directly determine the robot’s response performance under different terrain and load conditions.

In a simulation environment, the robot evaluates its own behavior through a reward function and continuously adjusts the parameters of the policy network. After extensive interactive training, the learned policy is transferred to a real platform, thereby achieving motion control from simulation to reality. The training process of deep reinforcement learning integrates three key links, perception, decision-making, and execution, ultimately forming a transferable motion policy. Robots trained using this method can maintain balance on complex terrains such as ice, slopes, or gravel roads, and automatically adjust their posture when disturbed by external forces, exhibiting flexibility and coordination close to that of natural organisms [6,7]. It is worth noting that the overall training efficiency has significantly improved after the introduction of a distributed training framework. This framework enables the robot to learn and accumulate experience in parallel in diverse environments, thereby mastering complex motion skills more quickly [8]. Based on this, we can infer that distributed deep reinforcement learning not only improves the convergence speed of training but also lays the technical foundation for achieving higher-level intelligent motion control.

Another major advantage of DRL lies in its excellent adaptability and generalization ability. By learning strategies in training environments with random perturbations, robots can often apply them directly to real-world scenarios with almost no need for parameter readjustment. This means that legged robots can freely switch between different walking modes according to terrain changes, thereby completing complex actions such as climbing, obstacle crossing, and even jumping [9]. Furthermore, reinforcement learning (RL) models based on physical simulation and multimodal perception can integrate visual, tactile, and inertial data to form a more comprehensive understanding of terrain features and dynamic states, thereby generating more robust motion strategies.

For the design of learning strategies, combining imitation learning with RL has been shown to significantly improve training efficiency. Typically, in the initial stage, robots can quickly acquire reasonable gaits by imitating the movement trajectories of humans or animals; subsequently, reinforcement learning is used to further optimize energy consumption and stability, thereby achieving more natural and efficient motion control [10]. This “imitate first, then reinforce” learning paradigm not only accelerates the convergence speed of the model but also effectively reduces energy waste and the risk of falls caused by random exploration. Based on this, we can infer that this hybrid learning model provides a feasible path to achieve efficient and safe autonomous robot movement.

Current research focuses on combining DRL with meta-learning and hierarchical control to enhance robots’ ability to quickly adapt to new tasks [11,12]. Through information sharing between different levels of control strategies, robots can rapidly adjust their behavior when faced with unknown environments or task changes, achieving true autonomous decision-making and environmental understanding.

The remaining sections of this paper are structured as follows: Section 2 reviews previous work related to this research topic; Section 3 elaborates on the systematic review approach used in this paper; Section 4 presents the main research results; Section 5 presents experiments and analyzes the results in depth; and Section 6 summarizes the entire paper and proposes possible future research directions.

2. Previous Studies

2.1. Types of Legged Robots and Limitations of Traditional Model-Based Control Methods

Legged robots can be broadly categorized according to their morphological structures, including bipedal, quadrupedal, multi-legged (e.g., hexapod and octopod), and hybrid leg-wheel systems. These robot types have been extensively studied in both laboratory and real-world applications due to their superior terrain adaptability compared with wheeled platforms [1,3]. The diversity in morphology leads to distinct locomotion characteristics and imposes different requirements on motion planning and control design.

Bipedal robots exhibit high mobility and potential energy efficiency, making them suitable for human-centered environments. However, their inherently narrow support polygon and underactuated dynamics result in significant challenges in balance maintenance, disturbance rejection, and real-time control, particularly during dynamic motions. Quadrupedal robots benefit from a larger support area and improved static and dynamic stability, which has led to their widespread adoption as benchmark platforms for legged locomotion research [1,4]. Despite these advantages, quadrupedal systems still face complex control challenges arising from hybrid contact dynamics, gait transitions, and high-dimensional joint coordination.

Multi-legged robots, such as hexapods, provide increased kinematic redundancy and fault tolerance, allowing stable locomotion even in the presence of leg failures or severe terrain irregularities [3,5]. This redundancy improves robustness but also significantly increases the dimensionality of the control problem, complicating motion planning and controller optimization. Hybrid leg-wheel robots further extend locomotion flexibility by enabling transitions between rolling and walking modes; however, such systems introduce morphology-dependent dynamics and additional coordination complexity across different locomotion modes [1].

The control of legged robots has traditionally relied on model-based methods, including model predictive control, whole-body control, and trajectory optimization frameworks. These approaches depend on explicit dynamic models and predefined objective functions, and they can offer strong theoretical guarantees when accurate system models are available [1,3]. However, constructing precise dynamic models for legged robots remains challenging due to nonlinear joint dynamics, intermittent ground contacts, friction uncertainty, actuator saturation, and time delays [3].

Moreover, optimization-based controllers typically require solving constrained numerical problems online, which can become computationally expensive for robots with many degrees of freedom, limiting real-time performance during fast or highly dynamic maneuvers [1]. Traditional controllers are also often designed for specific tasks or terrains, and their adaptability to new environments, unexpected disturbances, or morphological changes is limited. Retuning or redesign is usually required when operating conditions vary [2].

In addition, long-horizon decision-making and robustness under uncertainty remain difficult to achieve with purely model-based approaches, particularly in scenarios involving unknown terrain, sensor noise, or actuator degradation [1,3]. These limitations have motivated growing interest in data-driven control paradigms, such as reinforcement learning and deep reinforcement learning, which aim to learn control policies directly from interaction data and have shown promise in improving adaptability, robustness, and scalability across different legged robot platforms [2,6,11].

2.2. DRL for Legged Robots

DRL combines RL with deep neural networks (DNNs) to enable agents to learn optimal decision-making policies directly from high-dimensional sensory inputs through interaction with the environment. Unlike traditional model-based or rule-based control methods, DRL does not require explicit system dynamics modeling and can learn complex control strategies via trial-and-error optimization.

DRL is commonly formulated as a Markov Decision Process (MDP), defined by the tuple $(S,A,P,R,r)$, where $S$ is the state space, $A$ is the action space, $P(s^{\prime }|s,a)$ represents state transition probabilities, $R(s,a)$ is the reward function, and $\gamma \in (0,1]$ is the discount factor. The objective of the agent is to learn a policy $\pi (a|s)$ that maximizes the expected cumulative discounted return:

$$J(\pi )={\mathrm{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t}{r}_t\right]$$

(1)

RL enables legged robots to learn complex motor skills and exhibit adaptive behavior in diverse environments. As shown in Figure 3, this section systematically reviews several typical learning paradigms and control frameworks. End-to-end learning emphasizes a direct mapping from perceptual input to motion output, eliminating the need for explicit modeling and thus enabling data-driven control policy optimization. Hierarchical reinforcement learning improves task abstraction and decision-making efficiency by constructing multi-layered policy structures. The fusion of imitation learning and reinforcement learning combines human or animal motion priorities with trial-and-error-based policy optimization mechanisms, increasing learning speed while enhancing the stability of the training process.

Furthermore, model-based RL significantly improves sample utilization by introducing predictive dynamic models. Hybrid control frameworks seek a balance between data-driven methods and analytical control theory to ensure both system stability and generalization ability. Finally, meta-reinforcement learning and adaptive reinforcement learning aim to endow robots with the ability to migrate across tasks and adapt rapidly, enabling them to continuously learn and adjust their behavior in dynamic and unstructured environments. Based on this, we can argue that the combination of these different paradigms provides important theoretical and technical support for achieving truly intelligent and flexible legged robot motion control.

2.2.1. End-to-End Learning

End-to-End DRL is one of the core research directions in the field of legged robot motion control in recent years. Its key idea is to directly map sensor data (such as joint angles, speeds, torques, IMU signals, and even visual inputs) to motor control signals through deep neural networks, thereby realizing an integrated learning process of “perception-decision-execution” without explicitly establishing robot dynamics or gait models [9,13]. Compared with traditional model- or rule-based control methods, end-to-end DRL can autonomously learn complex nonlinear policies in a high-dimensional state space, thereby improving the robot’s adaptability and robustness in unknown terrain, external disturbances, and highly dynamic tasks.

As illustrated in Figure 4, the actuation mechanisms of multi-legged robots serve as the basis for end-to-end learning, showing how sensor data from joints, IMU, and other proprioceptive sources are processed to generate motor commands. This figure highlights the perception-to-action loop that end-to-end DRL directly optimizes.

Among many end-to-end reinforcement learning algorithms, Proximal Policy Optimization (PPO) is one of the most representative policy gradient methods. PPO introduces a trust region constraint in the objective function to limit the policy update magnitude, thereby achieving high stability and scalability [14]. This algorithm has been widely used in gait learning for quadruped robots (such as ANYmal and Unitree A1), enabling them to quickly acquire balance and walking abilities in simulation environments and achieve real-world deployment through policy fine-tuning [15].

Another typical method is the soft actor–critic (SAC) algorithm. This method maximizes expected returns while introducing a maximum entropy framework to enhance exploration capabilities by increasing the randomness of the policy [8,12]. SAC is particularly suitable for complex terrain tasks (such as going up and down slopes, obstacle crossing, jumping, etc.), and can achieve high-degree-of-freedom motion control without sacrificing stability. It exhibits extremely high sample efficiency on simulation platforms (such as Isaac Gym or Mujoco) and achieves excellent performance on real robots such as ANYmal C [7,13,15].

The Twin Delayed Deep Deterministic Policy Gradient (TD3) effectively alleviates the policy overestimation problem by introducing a double Q network and a delayed update mechanism [16]. The deterministic nature of TD3 makes it more stable in high-frequency control tasks and is often used to achieve continuous and smooth gait control and energy optimization [6]. In recent years, researchers have also proposed a variety of improved end-to-end methods. For example, distributed RL greatly improves training speed and sample efficiency through parallel sampling of multiple environments [17]; vision-based distributed RL combines convolutional neural networks (CNN) or Transformer models with policy networks, enabling robots to perform terrain recognition and action planning based on visual input [18]; meta-reinforcement learning (Meta-RL) and Adaptive Policy Optimization (AdPO) further enhance the environment generalization ability and online adaptive performance of end-to-end policies [19].

Table 1. Comparison of representative end-to-end deep RL methods for legged robot control.

Algorithm	Policy Type	Core Innovation	Control Characteristics	Performance Features
PPO [14,15]	On-policy, stochastic	Clipped surrogate objective constrains update magnitude	Smooth policy updates, good convergence stability	Reliable gait learning, moderate sample efficiency
SAC [8,12]	Off-policy, stochastic	Maximum entropy regularization enhances exploration	Robust to stochastic environments, adaptive to terrain	High stability, strong exploration ability
TD3 [6,16]	Off-policy, deterministic	Double critic and delayed target update mitigate overestimation	Continuous, low-variance action control	Efficient energy use, stable high-frequency execution
Distributed RL [17]	Parallel off policy	Multi-environment sampling and gradient averaging	Scale to large systems, improves data throughput	Fast training, enhanced generalization
DRL [18]	End-to-end perceptual policy	CNN/Transformer-based visual encoder integrated with policy network	Perception-driven decision making	Terrain-aware locomotion, improved adaptability
AdPO [19]	Adaptive metapolicy	Learns generalizable priorities for fast adaptation	Rapid online adaptation to new conditions	High flexibility, suitable for dynamic terrain

presents a comparative summary of representative end-to-end deep reinforcement learning algorithms used in legged robot control.

2.2.2. Hierarchical RL

In legged robot control, single-policy networks often struggle to simultaneously handle complex dynamic control and high-level task planning problems. Hierarchical reinforcement learning (HRL) offers a structured solution: by introducing policies at different levels (such as high-level decisions and low-level control), robots can learn at different time scales and semantic levels, thereby significantly improving sample efficiency, generalization, and task interpretability [20].

The core idea of hierarchical reinforcement learning is to decompose complex control tasks into several reusable sub-tasks. Each sub-policy focuses on learning local behaviors, while the high-level policy is responsible for selecting or scheduling these low-level policies in the task space [21]. This hierarchical structure is particularly suitable for the motion planning of legged robots in complex terrains (such as gravel slopes, stairs, or obstacle scenarios), because the robot can first learn stable basic gait control and then optimize high-level decisions such as direction selection, speed adjustment, or energy allocation based on this [13].

One of the typical representatives of hierarchical reinforcement learning is the Hierarchical Reinforcement Learning with Off-policy Correction (HIRO) algorithm [22]. This method uses the target vector output by the high-level policy as the conditional input of the low-level policy, enabling the low-level policy to achieve local targets through short-time step control actions, while the high-level policy optimizes the overall task reward over a longer time scale. To alleviate the experience mismatch problem between high- and low-level policies, HIRO introduces an “off-policy correction mechanism,” which effectively improves the stability of the training process. It is worth mentioning that Rudin et al. [15] applied a similar multi-time-scale hierarchical control structure to a quadruped robot, enabling it to dynamically adjust the target speed according to the terrain slope and its own posture, thereby achieving efficient and adaptive walking control.

Another representative hierarchical reinforcement learning method is Option-Critic architecture [23]. This method introduces the concept of “Options” in the policy framework, with each option corresponding to a learnable sub-policy and its termination condition. By learning when to activate, switch, or deactivate different options, robots can develop modular control behaviors in complex tasks. This mechanism not only enhances the flexibility of strategies but also enables robots to have stronger autonomous decision-making and adaptability in multi-stage tasks. The advantage of the Option-Critic framework lies in its differentiability and end-to-end trainability, enabling high- and low-level policies to be co-optimized. Luo et al. [24] applied this structure to the control of hexapod robots and achieved smooth transitions and autonomous behavior combinations for complex tasks by learning multiple behavioral modules. In addition to the classic HIRO and Option-Critic, several extended methods have emerged in recent years. For example, Hierarchical Actor–Critic (HAC) introduces a reward decomposition-based constraint mechanism between high and low layers to solve the credit assignment problem; Feudal Networks (FuN) generate semantic target vectors through high-level networks to guide low-level policies to perform more granular actions [25,26]. Furthermore, some studies combine Meta-RL or Mixture-of-Experts (MoE) structures to improve cross-task generalization performance by dynamically selecting sub-policies [18].

Table 2. Comparison of representative HRL methods for legged robot control.

Algorithm	Core Mechanism	Hierarchical Structure	Application
HIRO [15,22]	High-level policy outputs subgoals; low-level executes them with off-policy correction to prevent experience mismatch	Two-layer hierarchy (high-level subgoals, low-level control)	Quadruped robot adaptive gait and velocity control on uneven terrain
Option-Critic Architecture [23,24]	Learns when to start, switch, or terminate behavioral options in an end-to-end differentiable framework	Hierarchical policy with learnable “options” (sub-policies) and termination functions	Hexapod obstacle avoidance, climbing, and turning
HAC [25]	Decomposes rewards between levels for better credit assignment and temporal abstraction	Multi-level actor–critic with reward decomposition	Multi-terrain locomotion planning and balance control
FuN [26]	High-level “manager” sets latent goals guiding low-level “worker” via vector representations	Feudal manager–worker model	Long-term locomotion control with semantic planning
Expert-based HRL [18]	Dynamically selects sub-policies or adapts meta-parameters for cross-task generalization	Adaptive hierarchical modularity	Cross-task and multimodal locomotion learning

summarizes key HRL methods used in legged robot control.

2.2.3. Integration of Imitation Learning and Reinforcement Learning

In legged robot control, reinforcement learning autonomously learns optimal policies through interaction with the environment but usually requires many samples and complex reward designs, while imitation learning directly guides policy learning using expert demonstration data, thus significantly reducing exploration costs [2,10]. Researchers have combined the two to achieve a hybrid learning framework that balances convergence speed and performance optimization [11,18].

Figure 5 illustrates an imitation learning system for quadruped robots that can be combined with reinforcement learning. Expert demonstrations provide initial policy guidance, accelerating training and improving sample efficiency. The reinforcement learning module subsequently fine-tunes the policy to adapt to unstructured terrains and dynamic tasks, enabling the robot to autonomously optimize gait, balance, and energy efficiency. This figure highlights the integration of demonstration-based guidance with trial-and-error learning for improved performance and generalization.

Researchers use imitation learning to enable robots to replicate the behavior trajectories of experts, and reinforcement learning to discover new optimal policies by maximizing rewards. In the policy initialization phase, imitation learning is widely used to help robots quickly converge to a reasonable policy in the early stages of training, thus laying the foundation for subsequent optimization. Subsequently, reinforcement learning was introduced to further improve system performance, improve sample utilization, and alleviate common problems such as sparse rewards and training instability [27]. This “imitate first, then reinforce” learning paradigm is particularly effective in walking and jumping tasks on complex terrain. The robot first obtains feasible gait control patterns through imitation, and then uses reinforcement learning to fine-tune these patterns, thereby significantly enhancing its anti-interference ability and environmental adaptability [9].

Ho et al. [28] proposed Generative Adversarial Imitation Learning (GAIL), which is inspired by the structure of Generative Adversarial Networks (GANs). This method treats the policy network as a generator and introduces a discriminator to distinguish the behavioral differences between experts and robots. Researchers trained GAIL based on a game theory framework, enabling it to approximate the expert policy distribution without an explicit reward function, thereby achieving effective imitation in high-dimensional continuous control tasks. Compared with traditional Behavior Cloning (BC), GAIL can learn both actions and implicit task intentions simultaneously, thus exhibiting stronger generalization ability. Rudin et al. [15] used GAIL to train the gait of a quadruped robot, enabling it to maintain balance on unseen terrain and achieve energy-optimal motion control.

Fu et al. [29] further proposed Adversarial Inverse Reinforcement Learning (AIRL) to improve the interpretability of GAIL. This method introduces explicit modeling of the latent reward function into the framework, allowing the policy to recover its implicit reward signal while imitating expert behavior. The reward function generated by AIRL is transferable, supporting policy retraining in new environments, thereby improving the interpretability and cross-task transferability of the learning process. Peng et al. [10] proposed the DeepMimic framework, which further extends the idea of combining imitation and reinforcement. This framework trains the policy in a physical simulation environment by jointly optimizing the imitation loss and reinforcement reward. Researchers first extracted the target trajectory from motion capture (MoCap) data and used imitation loss in the initial stage to guide the robot to reproduce the movement patterns of humans or animals. Subsequently, they optimized task indicators such as stability, energy consumption, and recovery ability through reinforcement learning to achieve more natural and efficient motion control.

Researchers have significantly improved the learning speed, stability, and transferability of legged robots by fusing imitation learning and reinforcement learning [6]. Compared with pure reinforcement learning, these methods effectively avoid the problems of falls or energy spikes caused by random exploration in the early stages of training; compared with pure imitation learning, they retain the self-optimization and innovation capabilities of reinforcement learning. However, researchers still need to address several challenges, including dependence on high-quality expert data, conflicts between imitation and reward signals, and training stability under complex nonlinear dynamics [30].

Recently, researchers have explored multimodal fusion and hierarchical imitation-RL methods to further enhance the adaptability and robustness of the system [31]. Kim et al. [18] introduced constraints into the imitation reinforcement framework, so that the robot can meet physical constraints and task safety requirements while maintaining imitation accuracy. Through these methods, researchers have promoted the development of the fusion framework of imitation and reinforcement learning towards a more intelligent, interpretable and efficient direction, laying a key technical foundation for the autonomous movement of legged robots in complex terrain.

Table 3. Comparison of imitation–reinforcement learning integration methods in legged robot control.

Algorithm	Core Mechanism	Learning Strategy	Application
GAIL [15,28]	Uses a generator–discriminator framework to imitate expert distributions without explicit rewards	Adversarial Imitation + Policy Gradient	Quadruped gait learning, balance on uneven terrain
AIRL [29]	Recovers latent reward functions jointly with policy	Imitation + Inverse RL	Transferable locomotion control across terrains
DeepMimic [10]	Combines imitation loss from MoCap with RL rewards in simulation	Imitation pretraining + RL fine-tuning	Quadruped motion imitation and optimization
Constrained Imitation-RL Framework [18]	Adds physical and safety constraints during imitation and policy optimization	Constraint-guided hybrid learning	Legged locomotion under safety-critical tasks
Multimodal and Hierarchical Imitation-RL [31]	Using multi-level control	Hierarchical policy + multimodal perception	All-terrain adaptive locomotion and manipulation

presents a comparison of representative methods integrating imitation learning and reinforcement learning for legged robot control.

2.2.4. Model-Based RL

In traditional model-free RL, agents learn policies or value functions directly through interaction with the environment. Although they have strong expressive power, training usually requires many interaction samples and computational resources. Model-based reinforcement learning, on the other hand, learns environmental dynamics models explicitly or implicitly, thereby making predictions and plans in a “virtual environment,” significantly reducing the need for real interaction and improving data utilization [32].

Figure 6 presents a model-based learning framework that enables legged robots to perform highly dynamic actions such as jumping and sprinting. The framework integrates a predictive dynamics model with reinforcement learning, allowing the agent to plan actions while accounting for environmental and system dynamics. This approach improves sample efficiency and stability compared with purely model-free methods, demonstrating the potential for combining model-based predictions with learning-based policy optimization in legged robot control.

Researchers use environmental models to generate virtual trajectories during policy optimization and combine them with model predictive control (MPC) or policy gradient updates to significantly improve sample efficiency while ensuring performance [33]. In the field of legged robots, this approach is particularly suitable for dynamic stability control and energy-optimal motion planning, because robots need to frequently predict ground reaction forces, collision dynamics, and state transition uncertainties in complex terrain [34].

Chua et al. [35] proposed the Probabilistic Ensembles with Trajectory Sampling (PETS) algorithm, which for the first time introduced the uncertainty modeling of Gaussian processes into deep dynamic models. Under the PETS framework, researchers use an ensemble model composed of multiple neural networks to predict the distribution of future states and use model uncertainty to guide policy exploration, enabling the robot to obtain relatively reliable dynamic prediction capabilities in the early stages of training. Unlike traditional deterministic models, PETS can effectively assess action risks when facing unknown terrain or random external disturbances, thereby improving the overall stability and robustness of the system to a certain extent.

Janner et al. [36] further proposed the Model-Based Policy Optimization (MBPO) method, which generates high-quality virtual samples through short-time model predictions to assist policy updates. By dynamically balancing real experience and model-generated data, researchers have enabled MBPO to simultaneously possess high sample efficiency and training stability. This method performs well in quadrupedal robot obstacle crossing and walking tasks, achieving performance comparable to or even better than model-free algorithms (such as SAC or TD3) with less real interaction [37].

The Dreamer framework proposed by Hafner et al. [38] represents an end-to-end differentiable world model method. Researchers encode raw perceptual inputs (such as depth images or lidar) into compressed state representations in a latent space and perform policy learning and value evaluation in this latent space. The core advantage of Dreamer is that it can achieve long-step prediction in a fully simulated latent space, thereby greatly reducing the interaction with the actual environment.

Researchers have effectively bridged the contradiction between data efficiency and performance optimization through model-based reinforcement learning. Compared with model-free methods, MBRL can converge faster and has stronger interpretability with the same amount of data [39]. However, the accumulation of model error is still the main bottleneck limiting its performance, especially in complex contact dynamics and non-stationary terrain environments, where prediction bias may lead to control instability [40]. To address this challenge, researchers have begun to explore uncertainty-aware modeling, hybrid model architecture, and meta-adaptive model correction mechanisms to further improve the generalization and robustness of models [41,42].

Table 4. Summary of model-based RL methods for legged robot control.

Algorithm	Core Mechanism	Key Innovation	Application
PETS [35]	Learn probabilistic dynamics using an ensemble of neural networks to predict future state distributions	Introduces uncertainty estimation via Gaussian process modeling to improve prediction reliability	Enhance stability and risk-aware control under unseen terrains or external disturbances
MBPO [36]	Uses short-horizon model rollouts to generate synthetic samples for policy optimization	Balances real and model-generated data for efficient and stable training	Achieves high sample efficiency and stable performance in locomotion and obstacle-crossing tasks
Dreamer [38]	Builds a latent-space world model for policy and value learning using visual inputs	Performs long-horizon prediction and decision-making entirely in the latent space	Reduces real-world interaction needs while maintaining high learning performance in complex terrains

summarizes typical model-based RL methods.

2.2.5. Hybrid Control

In recent years, researchers have proposed a hybrid control framework to integrate the advantages of RL and traditional control methods, achieving efficient and stable control of legged robots in complex environments [18]. Although pure reinforcement learning excels in policy representation, it still faces challenges in training stability, generalization performance, and physical constraint satisfaction. In contrast, traditional control methods (such as MPC and Central Pattern Generator (CPG)) have a solid theoretical foundation in dynamic constraint handling and stability analysis, but their adaptive capabilities are limited, and they struggle to handle high-dimensional perceptual inputs. Hybrid control aims to combine the advantages of both approaches, achieving a balance between “interpretability” and “intelligence” through structured integration [43]. Researchers have developed hybrid control strategies that can adaptively optimize complex behaviors while maintaining stability by superimposing learning modules on traditional controllers or embedding physical models within reinforcement learning frameworks. These methods have shown significant advantages in tasks such as obstacle crossing, recovery control, and low-energy gait in multi-legged robots, improving learning efficiency and enhancing system robustness and environmental adaptability [44,45].

Figure 7 illustrates a hybrid control deployment pipeline in which a learned policy module replaces a traditional MPC. In this framework, the RL policy provides adaptive updates to the control actions, while the underlying MPC ensures physical feasibility and safety. This hybrid approach enhances agility and robustness in complex terrains and dynamic tasks, combining the advantages of model-based safety guarantees with the flexibility of learning-based adaptation.

Among the many hybrid control methods, residual RL is one of the most representative. Johannink et al. [46] proposed this method, which learns a residual term based on the output of the traditional controller to compensate for model errors or external disturbances. The residual module is implemented by a neural network and is used to correct the control signal so that the robot can maintain stability under unknown terrain or external disturbances. Yu et al. [47] applied residual reinforcement learning to the walking control of quadruped robots. They first used imitation learning to initialize the strategy and then optimized it through residual learning, thereby achieving stable walking on unstructured terrains such as mud and sand. The core advantage of this method is that even if there is a modeling bias in the basic controller, the learning module can still achieve performance compensation by minimizing the residual, thereby effectively reducing the risk of strategy collapse.

Another important approach is the combination of RL and MPC (RL-MPC). Such methods typically employ a hierarchical structure: the upper-layer RL module generates the objective or cost function, while the lower-layer MPC is responsible for short-term trajectory optimization and physical constraint satisfaction. Yazdi et al. [48] proposed a dynamic walking control framework that integrates MPC and RL, learning the long-term reward structure through RL and generating dynamically feasible trajectories in real time using MPC. Gangapurwala et al. [44] combined RL with optimal control in the RLOC framework, using RL to generate terrain-adaptive reference policies, and then having MPC execute low-level control, thereby achieving real-time controllable and generalizable motion planning. Such methods demonstrate high safety and interpretability on hardware robots, providing a feasible path for real-world deployment.

In addition, researchers have also explored a bio-inspired control scheme combining RL and CPG (RL-CPG). The CPG model mimics the neural oscillatory network in the animal spinal cord, generating rhythmic movement patterns, and possesses natural dynamic stability and robustness [49]. Introducing RL into the CPG framework allows for learning to adjust oscillation parameters to adapt to different terrains and speed requirements.

Hybrid control strategies effectively combine the stability of traditional control with the adaptability of reinforcement learning, achieving a balance between sample efficiency, interpretability, and generalization [9]. Compared with pure RL, it significantly reduces training risk and sample requirements; compared with pure MPC or CPG, it has stronger cross-environment adaptability and policy innovation capabilities. Current research is gradually moving towards hierarchical and multimodal integration, such as combining RL-MPC with Residual RL, or optimizing the CPG network structure through RL to achieve continuous self-regulation in dynamic environments.

Table 5. Summary of hybrid control methods for legged robot control.

Algorithm	Core Mechanism	Integration Strategy	Application
Residual RL [46,47]	Learns a residual term on top of a classical controller to compensate for model errors or external disturbances	RL module adds corrective residuals to traditional control outputs	Enhance stability and adaptability on uneven terrains such as mud or sand by minimizing control residuals
RL-MPC [48]	Combines high-level RL with low-level MPC for hierarchical decision-making	RL defines long-term objectives; MPC ensures short-term constraint satisfaction	Achieves dynamic walking and terrain-adaptive planning with improved safety and interpretability
RL-CPG [49]	Integrates Reinforcement Learning with CPG neural oscillatory models	RL adjusts CPG oscillation parameters (frequency, phase, amplitude) for adaptive gait control	Enables rhythmic, bio-inspired locomotion adaptable to various speeds and terrains

summarizes the main hybrid control methods used for legged robot control, and shows the core mechanisms, fusion strategies and typical applications of Residual RL, RL-MPC and RL-CPG in integrating traditional control and reinforcement learning.

2.2.6. Meta and Adaptive RL

Meta-RL and adaptive RL provide new research directions for legged robot control by improving the agent’s ability to learn quickly and adapt to the environment [50]. Traditional deep reinforcement learning often requires many interaction samples to converge in complex or unknown terrains, while Meta-RL is committed to enabling robots to quickly generalize to new tasks or environments after a small number of trials by “learning how to learn”. The core idea of this learning paradigm is to extract commonalities from task distributions, so that the policy has cross-task transfer capabilities, thereby achieving real-time adaptation and efficient control [51].

Model-Agnostic Meta-Learning (MAML) is one of the most representative algorithms in this field. It optimizes the initial parameters of the policy at the meta-task level, so that the model can quickly adapt when encountering new tasks with very few gradient updates [52]. This mechanism has been widely used in the terrain adaptive control of multi-legged robots, such as switching gait between different terrains such as flat ground, slopes, and gravel, achieving a significant improvement in learning efficiency. Extended MAML algorithms, such as Exploration-Enhanced Model-Agnostic Meta-Learning (E-MAML), further enhance the robot’s policy robustness in highly uncertain terrain by introducing an exploration term in the meta-update stage [19].

Probabilistic Embeddings for Actor–Critic RL(PEARL) explicitly models the task distribution by introducing latent variable embeddings, enabling the agent to infer implicit environmental features from limited experience [53]. Similarly to PEARL, Adaptive Learning from Probabilistic Context Adaptation (ALPaCA) utilizes Bayesian regression to achieve online task recognition through parameterized context information, thereby enhancing the model’s adaptability in non-stationary environments [54]. Another approach is Context-Adaptive Variational Inference Algorithm (CAVIA), which significantly reduces the dependency on shared parameters between meta-tasks through learnable context variables, enabling multi-legged robots to maintain control stability and energy efficiency during task switching [55].

In addition to the above meta-learning methods, Fast Reinforcement Learning via Slow Reinforcement Learning (RL²) internalizes the learning process in the policy through the recurrent neural network (RNN) structure, enabling the agent to “memorize” previous interaction experiences and achieve rapid adaptation [56]. Similarly, the Meta-SGD algorithm further optimizes the meta-parameters of the learning rate based on MAML, so that the update direction and step size can be dynamically adjusted, thereby obtaining a smoother policy convergence curve in complex terrain. Adaptive reinforcement learning frameworks combined with MPC have also gradually attracted attention. By updating the dynamic model or contact estimation module online, continuous control and stable recovery in highly dynamic environments have been achieved.

Table 6. Comparative summary of meta and adaptive RL methods for legged robot control.

Algorithm	Core Principle	Key Mechanism	Application
MAML [52]	Learns optimal initial parameters for fast adaptation across tasks.	Meta-level gradient optimization enables quick fine-tuning with few updates.	Achieves efficient gait switching across terrains such as slopes and gravel.
E-MAML [19]	Extends MAML by integrating exploration to improve robustness.	Adds exploration terms in meta-updates to handle uncertain dynamics.	Enhances terrain adaptability under high uncertainty.
PEARL [53]	Models task distribution through latent probabilistic embeddings.	Infers implicit environ-mental features from few samples.	Enables rapid adaptation to new or unseen terrains.
ALPaCA [54]	Employs Bayesian regression for online task inference.	Learn task-specific context to support continual adaptation.	Improves control stability in non-stationary environments.
CAVIA [55]	Learns task context variables to separate shared and task-specific knowledge.	Reduces inter-task interference via con-text-conditioned updates.	Maintains stability and efficiency during task transitions.
RL2 [56]	Internalizes learning dynamics via RNN memory.	Uses recurrent policy networks to encode previous experiences.	Achieves fast terrain adaptation with minimal data.

summarizes the main meta-reinforcement learning and adaptive reinforcement learning methods used for legged robot control, and compares their core principles, key mechanisms and typical applications.

2.3. Representative Applications of DRL in Legged Robots

DRL has been increasingly applied in legged robot locomotion, enabling autonomous, adaptive, and robust movement in complex terrains. Several representative applications are summarized below.

Quadrupedal robots such as ANYmal and Unitree A1 have been widely studied using end-to-end DRL frameworks. Lee et al. [6] demonstrated that quadruped robots trained with DRL can learn stable locomotion over challenging terrains, achieving agile gait transitions and high energy efficiency. Tan et al. [7] further explored sim-to-real transfer, showing that policies trained in simulation can be effectively deployed on real quadrupeds, significantly reducing real-world training costs. These approaches highlight the potential of DRL for direct mapping from sensory inputs to motor commands, without requiring explicit dynamics models.

Hybrid and hierarchical control strategies have also been employed to enhance robustness and adaptability. Bellicoso et al. [1] reviewed real-world applications where DRL was combined with traditional control modules to improve balance, stability, and obstacle negotiation on uneven terrains. Xu et al. [11] proposed a meta-reinforcement learning framework for hexapod robots, enabling fault-tolerant control when multiple legs fail, illustrating DRL’s adaptability in high-degree-of-freedom multi-legged systems.

Learning from demonstrations and hierarchical skills have been applied to improve training efficiency and motion naturalness. Kober et al. [2] summarized imitation learning and reinforcement learning integrations in robotic locomotion, where expert trajectories guide the policy learning, reducing sample complexity. Peng et al. [13] used hierarchical DRL to acquire dynamic locomotion skills in multi-legged robots, achieving smooth and coordinated motion across complex terrains.

The soft SAC and policy gradient methods have been successfully deployed in high-dimensional locomotion tasks. Haarnoja et al. [8] applied SAC to legged robots to allow them to learn agile and energy-efficient gaits, showing strong generalization to new terrain types. Similarly, Hwangbo et al. [9] combined SAC with robust state representations to achieve highly dynamic behaviors, including jumping and sprinting, on quadrupedal platforms.

Real-world deployment of DRL policies has also been investigated. Bellicoso et al. [1] summarized several practical examples where DRL-trained quadrupeds performed locomotion tasks in indoor and outdoor environments, demonstrating robustness under sensor noise, terrain uncertainty, and external disturbances. Tan et al. [7] highlighted that sim-to-real strategies, including domain randomization and online policy fine-tuning, are key to bridging the gap between simulation and real-world execution.

3. Review Methodology

3.1. Research Questions

A systematic literature review method was adopted to comprehensively investigate the progress of research in the field of DRL for legged robot locomotion. This systematic review has been registered with the Open Science Framework (OSF), registration ID: [1229757]. To ensure the reproducibility, transparency, and scientific rigor of this review, this study strictly followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines [57]. The PRISMA 2020 checklist is provided in the Supplementary Materials. Based on this, the review methodology can be divided into five core steps: clarifying the research question, identifying relevant information sources, designing effective search strategies, establishing clear literature screening criteria, and performing systematic data extraction and analysis. In practice, this process not only ensures the standardization of the methodology but also provides a solid foundation for the verifiability of subsequent results.

Based on this framework, the following seven research questions (RQ) are defined to guide the review:

RQ1: What types of DRL algorithms have been proposed for legged robot locomotion control?

RQ2: What are the key factors that influence the performance of DRL-based legged robot control systems?

RQ3: What methodological approaches have been developed to improve sample efficiency, accelerate training, and enhance generalization in DRL-based locomotion control?

RQ4: How do DRL-based methods compare with traditional model-based or optimization-based locomotion control methods in terms of stability, robustness, adaptability, and computational efficiency?

RQ5: What strategies have been proposed to bridge the sim-to-real gap for DRL policies deployed on real legged robots?

RQ6: What are the commonly used simulation environments, datasets, benchmarks, and evaluation metrics in DRL-based legged robot research?

RQ7: How do DRL-based locomotion control methods perform under varying terrains, external disturbances, robot morphologies, sensory modalities, and practical constraints such as energy consumption and actuator limitations?

3.2. Literature Sources and Search Strategies

This study systematically reviews the literature on DRL-based motion control for legged robots, covering research published between 2018 and 2025. To comprehensively grasp the development trends and research results in this field, a systematic search was conducted in seven authoritative electronic databases, including Web of Science, IEEE Xplore, Scopus, Wiley Online Library, ScienceDirect, ACM Digital Library, and SpringerLink. In designing the search strategy, the study focused on DRL motion control for legged robots as the core theme, while also paying attention to research directions such as policy learning, terrain adaptation, gait optimization, and simulation-to-real transfer, thus ensuring the comprehensiveness and representativeness of the collected studies. All searches were based on document titles, abstracts, and keywords, and the search terms were optimized through multiple rounds to improve relevance and accuracy.

Table 7. Literature sources and search strategies.

Literature Sources	Search Strings and Keywords
Web of Science	TI = (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion” OR “gait adaptation” OR “terrain-aware locomotion”) AND AB = (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion” OR “gait adaptation” OR “terrain-aware locomotion”) AND AK = (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion” OR “gait adaptation” OR “terrain-aware locomotion”)
IEEE	(“Document Title”: “deep reinforcement learning” OR “Document Title”: “multi-legged robot locomotion” OR “Document Title”: “legged robot control” OR “Document Title”: “quadruped locomotion” OR “Document Title”: “biped locomotion” OR “Document Title”: “gait adaptation”) AND (“Abstract”: “deep reinforcement learning” OR “Abstract”: “multi-legged robot locomotion” OR “Abstract”: “legged robot control” OR “Abstract”: “quadruped locomotion” OR “Abstract”: “biped locomotion” OR “Abstract”: “gait adaptation”) AND (“Index Terms”: “deep reinforcement learning” OR “Index Terms”: “multi-legged robot locomotion” OR “Index Terms”: “legged robot control” OR “Index Terms”: “quadruped locomotion” OR “Index Terms”: “biped locomotion” OR “Index Terms”: “gait adaptation”)
Scopus	TITLE-ABS-KEY (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion” OR “gait adaptation” OR “terrain-aware locomotion”)
Science Direct	Title, abstract, or keywords (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control”) AND (“gait adaptation” OR “terrain-aware locomotion”)
Wiley	(“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion”)
ACM	Title, Abstract, and Keywords (“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion”)
Springer	(“deep reinforcement learning” OR “multi-legged robot locomotion” OR “legged robot control” OR “quadruped locomotion” OR “biped locomotion”)

shows the detailed search strings, keyword parameters, and specific settings used in each database.

As shown in

Table 8. Inclusion and exclusion criteria.

Inclusion Criteria			Exclusion Criteria
IC1	Articles published between 2018 and 2025 that explicitly address legged robotic systems within the broader fields of robotics and machine learning, with an emphasis on learning-based locomotion or control problems	EX1	Reviews, books, book chapters, and other publications that have not been peer-reviewed
IC2	Articles written in English	EX2	Articles not written in English
IC3	Articles not duplicated in other databases	EX3	Articles that have been selected in other databases
IC4	Studies focusing on legged robot locomotion, including quadruped, hexapod, or biped systems, using deep reinforcement learning	EX4	Studies unrelated to legged robot motion control or not employing reinforcement learning approaches
IC5	The full text of the article is available	EX5	The full text of the article is not available
IC6	Research addressing DRL frameworks, policy learning, sim-to-real transfer, terrain adaptation, or energy-efficient locomotion control	EX6	Articles not involving DRL-based locomotion control or lacking relevance to motion learning and adaptation

, to ensure the reliability of the selected studies, this study followed a set of well-defined inclusion and exclusion criteria. Specifically, only English-language journal articles published between 2018 and 2025 that focus on DRL for multi-legged robotic locomotion, gait adaptation, or learning-based control strategies were considered. Furthermore, only articles with full-text availability were included in the analysis. During the literature screening process, conference papers, conference proceedings, review articles, books, book chapters, and other non-peer-reviewed publications not directly related to DRL-based legged robot control were excluded. Articles not written in English were also excluded. Table 8 lists other specific criteria used in the screening process; all criteria remained consistent throughout the search and screening phases. Literature screening began in September 2025 and was systematically applied across all selected electronic databases.

3.3. Data Collection and Analysis

As shown in Figure 8, following the PRISMA 2020 guidelines [57], this systematic literature review on DRL-based multi-legged robot locomotion was conducted in five stages. In the first stage, a preliminary screening of the literature was performed. A total of 3127 articles were identified across the selected electronic databases, distributed as follows: WOS with 412 articles (13.2%), IEEE Xplore with 678 articles (21.7%), Scopus with 841 articles (26.9%), ScienceDirect with 512 articles (16.4%), Wiley Online Library with 102 articles (3.3%), ACM Digital Library with 345 articles (11.0%), and SpringerLink with 237 articles (7.6%).

To ensure an operational and hierarchical screening process, the selection procedure was conducted in multiple stages with clearly defined priorities among criteria. Inclusion criteria IC1 and IC2, together with exclusion criteria EX1 and EX2, were treated as high-level filtering rules to remove articles that were outside the publication scope or language requirements. Based on exclusion criteria EX1 and EX2, and inclusion criteria IC1 and IC2, researchers manually screened and removed 528 articles, while automated screening tools were used to exclude 641 articles. A total of 1169 articles were removed due to language or publication type incompatibility.

Furthermore, based on exclusion criterion EX3 and inclusion criterion IC3, 415 duplicate articles were identified and removed. At this stage, only bibliographic eligibility was considered, without evaluating technical relevance. After the first stage of screening, a total of 1543 articles were retained for the second stage of review.

In the second stage, researchers meticulously reviewed the title and abstract of each article. Inclusion criterion IC4 and exclusion criterion EX4 were treated as the primary domain-specific relevance criteria, determining whether an article genuinely focused on legged robot locomotion using deep reinforcement learning. Based on these criteria, a total of 984 articles unrelated to DRL-based motion control of legged robots were excluded. For articles with incomplete or ambiguous abstract information, researchers further searched and read the full text to confirm their relevance. In this stage, another 52 articles were removed based on exclusion criterion EX5 because the full text was unavailable.

In the final screening phase, the researchers further applied exclusion criterion EX6 and inclusion criterion IC6 as fine-grained relevance criteria, focusing on methodological depth and alignment with the research objectives. These criteria emphasized DRL-based policy learning, sim-to-real transfer, terrain adaptability, and energy-efficient locomotion control. When conflicts among criteria arose, IC4 and IC6 were given priority over the broader scope defined in IC1, ensuring that only technically relevant studies were retained. Ultimately, the candidate studies were narrowed down to 27 articles directly related to DRL-based motion control of legged robots. As shown in

Table 9. Details of the selected papers.

Authors and Citation	Year	Venue	Citations	Publication Type	Country
Yang et al. [58]	2020	PMLR CoRL	200	Conference	USA
Belmonte-Baeza et al. [59]	2022	IEEE RA-L	46	Journal	Spain
Chen et al. [60]	2024	IEEE ICRA	12	Conference	USA
Rudin et al. [43]	2021	IEEE T-RO	143	Journal	Switzerland
Ha et al. [61]	2018	IEEE UR	61	Conference	Japan
Smith et al. [62]	2022	IEEE ICRA	153	Conference	USA
Chen et al. [63]	2018	IEEE IROS	68	Conference	Sweden
Gan et al. [64]	2022	IEEE RA-L	43	Journal	USA
Chamorro et al. [65]	2024	IEEE ICRA	14	Conference	Switzerland
Konen et al. [66]	2019	IEEE ICRA	18	Conference	Germany
Li et al. [67]	2024	MDPI Biomimetics	4	Journal	China
Qin et al. [68]	2019	IEEE ICRAE	22	Conference	China
Lee et al. [69]	2022	Springer CWR	14	Conference	Germany
Chen et al. [70]	2022	Elsevier RAS	44	Journal	China
Lyu et al. [71]	2024	RSS	12	Conference	China
Weerakoon et al. [72]	2024	IEEE ICRA	9	Conference	USA
Chen et al. [73]	2023	ASME T-MECH	9	Journal	China
Lee et al. [74]	2024	Science Robotics	93	Journal	Germany
Cui et al. [75]	2021	IEEE RA-L	99	Journal	China
Gangapurwala et al. [44]	2022	IEEE T-RO	171	Journal	UK
Kim et al. [18]	2024	IEEE T-RO	73	Journal	South Korea
Morimoto et al. [76]	2023	JRM	5	Journal	Japan
Yu et al. [47]	2025	IEEE RA-L	22	Journal	USA
Yang et al. [77]	2022	IEEE IROS	55	Conference	USA
Margolis et al. [78]	2024	SAGE IJRR	286	Journal	USA
Bing et al. [79]	2020	Elsevier NN	82	Journal	China
Xu et al. [80]	2024	IEEE ICRA	14	Conference	USA

, these 27 articles underwent a comprehensive full-text review to assess their relevance to the research question and their alignment with the research objectives in terms of methods, results, and conclusions.

4. Results

This section presents a statistical analysis of the SLR results on topics related to DRL algorithms for legged robot motion control. Based on data extracted from 27 screened articles published since 2018, responses are provided to research questions RQ1 to RQ6.

4.1. What Types of DRL Algorithms Have Been Proposed for Legged Robot Locomotion Control (RQ1)?

Figure 9 shows the DRL algorithm for legged robot motion control obtained after reviewing the selected literature. In recent years, various types of DRL algorithms have been proposed for legged robot motion control to improve legged robots’ comprehensive performance in complex terrain, energy consumption constraints, and dynamic stability. Data-efficient DRL algorithms significantly reduce dependence on simulation data by improving the policy update and experience reuse mechanism. Yang et al. [58] proposed a high-data-efficiency reinforcement learning method that enables the robot to achieve stable gait control under limited samples. Meta-RL algorithms are used to achieve rapid policy adaptation and cross-task transfer. Belmonte-Baeza et al. [59] optimized the structure and control strategy of legged robots through a meta-learning framework, thereby improving the generalization ability under different terrains.

For model complexity, Chen et al. [60] proposed a reinforcement learning method based on a reduced-order model to reduce the computational burden of high-dimensional dynamics and achieve a more stable policy learning process. At the same time, for dynamic behavior under extreme conditions, Rudin et al. [43] used DRL to realize the “cat-like jump and landing” of a legged robot in a low-gravity environment, demonstrating the potential of deep learning in high-dynamic control tasks. In addition, to improve the portability of real hardware, Ha et al. [61] designed an automated DRL training environment to achieve a seamless transition from simulation to real machines, while the online fine-tuning framework proposed by Smith et al. [62] allows the robot to continuously improve its gait control strategy during actual operation.

In multimodal and complex environments, Chen et al. [63] and Lee et al. [69] respectively proposed reinforcement learning algorithms that integrate multimodal control with wheel-leg fusion. By jointly inputting visual, inertial, and contact information into the network, the robot’s adaptability and navigation capabilities on complex terrain are improved. At the same time, some studies have introduced reinforcement learning frameworks for safety constraints and energy consumption optimization, such as the Safe RL method proposed by Yang et al. [77], and the work of Gan et al. [64] using deep inverse reinforcement learning for energy-aware terrain modeling, which significantly improved the energy efficiency and safety of robots on high-risk terrain.

At the task level, researchers have also developed a variety of specialized DRL algorithms. Li et al. [67] proposed a fall recovery control strategy based on reinforcement learning, which can achieve rapid recovery after imbalance; Qin et al. [68] used curriculum learning to improve the migration performance of hexapod robots from simulation to reality; Margolis et al. [78] achieved fast running and efficient movement of legged robots through high-dynamic reinforcement learning; and Xu et al. [80] explored reinforcement learning methods for achieving dexterous movement in a small three-dimensional space. In addition, Yu et al. [47] proposed a residual RL framework, which improved multimodal gait conversion and stability performance by superimposing the residual learning part on a traditional controller.

For structural optimization, Chen et al. [73] proposed a deep reinforcement learning algorithm for joint optimization of morphology and gait, which can achieve integrated structure-control optimization in small robots; the work of Belmonte-Baeza et al. [59] further demonstrated the application potential of meta-learning in robot structural design. At the same time, to enhance the interpretability and generalization of the strategy, Gangapurwala et al. [44] proposed an RL-OC hybrid algorithm combined with optimal control to achieve robust motion control under terrain perception. Similarly, Kim et al. [18] proposed a reinforcement learning framework that explicitly introduced constraints in the reward function, so that the strategy not only pursued optimal performance but also took into account stability and physical constraints.

Table 10. Classification of DRL algorithms for legged robot locomotion.

Algorithm Type	Core Idea	Typical Application Tasks
Data-Efficient and Sim-to-Real RL [58,61,62,68]	Focuses on improving data efficiency and transferability from simulation to real-world environments through curriculum learning, domain randomization, and policy fine-tuning.	Sample-efficient learning, sim-to-real adaptation, online fine-tuning
Meta-Learning and Morphology Optimization [59,73]	Uses meta-reinforcement learning and co-optimization techniques to jointly adapt control policy and robot morphology for optimal performance.	Adaptive morphology design, cross-task generalization
Safety-Constrained and Energy-Aware RL [18,44,64,77]	Integrates safety filters, energy models, and constrained policy optimization to ensure stable and efficient locomotion under dynamic environments.	Energy-efficient locomotion, safe policy learning
Multi-Modal and Hybrid Control [47,63,69,74,75]	Combines visual, proprioceptive, and contact modalities or integrates RL with traditional control methods to enhance robustness and adaptability.	Vision–contact fusion, terrain adaptation, robust navigation
Task-Specific DRL Applications [43,67,78,80]	Designs specialized DRL frameworks for specific tasks such as recovery, jumping, or high-speed running in structured and unstructured terrains.	Fall recovery, dynamic jumping, agile locomotion

compares and summarizes the core ideas and application scenarios of different categories of algorithms proposed for legged robot motion control.

4.2. What Are the Key Factors That Influence the Performance of DRL-Based Legged Robot Control Systems (RQ2)?

At the current stage, applying DRL to legged robot control still faces several critical challenges. These challenges mainly include low sample efficiency, difficulties in reward function design, safety risks during policy exploration, sensitivity to hardware constraints, and limited generalization across unseen environments. The following factors discussed in this section can be regarded as both the sources of these challenges and the primary directions through which recent studies attempt to address them.

As shown in Figure 10, the performance of legged robot control systems based on DRL is affected by multiple factors, including algorithm design, training strategy, environment modeling, hardware constraints, generalization, and adaptability. Different studies have revealed the mechanisms of these key factors on robot motion performance, energy efficiency, stability, and generalization from different perspectives.

The construction of the reward function is considered to be one of the most core performance determinants. Kim et al. [18] pointed out that the stability and convergence speed of the reinforcement learning strategy not only depend on the reward structure itself but are also significantly affected by the design of the constraint conditions. The studies by Rudin et al. [43] and Chamorro et al. [65] further validated that the trade-offs between different reward items lead to differentiated motion characteristics of legged robots in complex tasks (such as jumping and climbing). Regarding policy structure, Yang et al. [58] proposed a data-efficient reinforcement learning framework that significantly reduced sample requirements by improving policy representation and model structure, while Chen et al. [60] achieved high-precision control in a simplified state space by introducing a reduced-order model.

Furthermore, studies by Yang et al. [77] and Li et al. [67] emphasized the importance of safety-constrained reinforcement learning, which effectively avoids falls or unstable behavior during policy exploration, thereby significantly improving the overall reliability of the system. Belmonte-Baeza et al. [59] and Yu et al. [47] demonstrated, from the perspectives of meta-reinforcement learning and residual reinforcement learning, that these mechanisms exhibit stronger adaptability in dealing with new tasks or structural uncertainties.

In addition, at the level of training policy and data efficiency, sample utilization and the choice of learning methods directly affect the stability and generalization of policy performance. The efficient learning algorithm proposed by Yang et al. [58] and the online fine-tuning method of Smith et al. [62] have significantly shortened the adaptation period of the model in the real environment. Meanwhile, Qin et al. [68] and Weerakoon et al. [72] used curriculum learning and domain randomization techniques to enable the robot to gradually transition from low-complexity scenarios to complex terrain tasks, and in the process, enhanced the transfer ability from simulation to reality. Rudin et al. [43] pointed out that appropriate exploration strategies help to balance the relationship between “exploration-exploitation”, thereby maintaining action diversity and jumping accuracy in low-gravity or unknown environments.

The performance of the policy is highly dependent on the physical fidelity and diversity of the training environment. Qin et al. [68] and Smith et al. [62] showed that by randomizing friction, mass, and delay parameters, the robustness of the policy in the real world can be significantly enhanced. Gan et al. [64] and Gangapurwala et al. [44] further developed an energy-based and terrain-aware inverse reinforcement learning model (Energy-Based and Terrain-Aware IRL), which can effectively improve the robot’s adaptability and navigation capabilities on rugged terrain. In addition, Weerakoon et al. [72] and Lee et al. [69] demonstrated that offline reinforcement learning can significantly enhance the environmental uncertainty robustness of the strategy in unstructured outdoor environments.

Controlling performance is strongly affected by hardware parameters and dynamic characteristics. The morphology-control co-optimization method proposed by Belmonte-Baeza et al. [59] and Chen et al. [73] achieves a dual improvement in energy efficiency and stability by simultaneously optimizing robot morphology and control strategy. Ha et al. [61] and Lee et al. [71] pointed out that the physical limitations of the actuator (such as torque limit and response delay) will cause control errors, and the fast adaptive control framework RL²AC can compensate for these dynamic deviations in real time during operation. Bing et al. [79] and Li et al. [67] further emphasized that a control mechanism based on reinforcement learning can not only optimize energy consumption but also achieve autonomous recovery after injury or fall.

In terms of generalization and robustness, Rudin et al. [43] and Lee et al. [64] showed that by training on diverse terrains and different contact conditions, DRL strategies can achieve higher cross-scenario generalization capabilities. The meta-learning and continuous learning framework proposed by Belmonte-Baeza et al. [59] and Smith et al. [62] enables robots to cope with model drift and mechanical wear problems during long-term operation. Yu and Rosendo [47] and Gangapurwala et al. [44] demonstrated the potential of a multimodal and hybrid control framework, which combines traditional optimal control with reinforcement learning modules to improve policy interpretability and stability. In addition, Morimoto et al. [76] also extended the group behavior learning of multi-legged robots, making reinforcement learning show the scalability and cooperative control potential in multi-robot collaboration and group motion.

In summary, despite significant progress, DRL-based legged robot control still faces several open challenges, including high training costs, sim-to-real transfer difficulties, safety guarantees that are under exploration, and limited robustness to hardware imperfections and environmental uncertainty.

4.3. What Methodological Approaches Have Been Developed to Improve Sample Efficiency, Accelerate Training, and Enhance Generalization in DRL-Based Locomotion Control (RQ3)?

To improve sample efficiency, researchers have proposed a variety of methods. Yang et al. [58] implemented a data-efficient reinforcement learning framework by improving policy representation and model structure, thereby significantly reducing the amount of data required for training. Belmonte-Baeza et al. [59] introduced Meta-RL, which enables robots to quickly adapt to new tasks and improve their ability to learn from few samples. Chen et al. [60] used reduced-order models to achieve high-precision control while simplifying the state space, further reducing the amount of data required for training. As shown in Figure 11, data-efficient RL, meta-RL, and reduced-order models have higher values (lighter colors) on the “Sample Efficiency” indicator, which intuitively reflects the significant role of these methods in reducing the need for training samples.

To accelerate the training process, researchers have tried various strategies, including online fine-tuning, curriculum-based learning, reward function design, and safety constraint mechanisms. Smith et al. [62] fine-tuned the pre-training strategy in a real environment, which significantly shortened the adaptation period of the model. Qin et al. [68] used curriculum-based learning to enable the robot to gradually transition from low-complexity tasks to high-complexity tasks, thereby accelerating the convergence process of the strategy. Meanwhile, Kim et al. [18] and Yang et al. [77] pointed out that if the reward function can be reasonably designed and a balance is achieved between safety constraints and exploration, unstable behaviors during training can be effectively reduced, and the overall learning efficiency will also be improved. In addition, the fast online adaptive control framework (RL²AC) proposed by Lyu et al. [71] can compensate for dynamic deviations in real time during training, enabling the robot to achieve more stable motion performance in a shorter time. Based on Figure 11, it can be observed that these methods show high values in both training speed and stability indicators, clearly demonstrating their significant contributions to training acceleration and stability improvement.

To enhance the generalization ability of DRL-based motion control strategies, researchers have developed a variety of methods. Through domain randomization and sim-to-real transfer, robots can adapt to diverse environmental parameters, thereby showing better robustness in the real world [68,72]. Gan et al. [64] proposed an energy-aware and terrain-aware inverse IRL model, which enables robots to have higher adaptability in rugged and complex terrain. Yu et al. [47] combined traditional optimal control methods with reinforcement learning to form hybrid control and multimodal strategies, improving the interpretability and stability of the strategy. Morimoto et al. [76] used group behavior learning to enable multi-robots to show scalability and stronger robustness in collaborative tasks. As shown in Figure 11, these methods have higher values in the “Generalization”, “Adaptability”, and “Robustness” indicators, and their colors are lighter, which clearly reflects the importance of these strategies in improving generalization ability and coping with complex environments.

4.4. How Do DRL-Based Methods Compare with Traditional Model-Based or Optimization-Based Locomotion Control Methods in Terms of Stability, Robustness, Adaptability, and Computational Efficiency (RQ4)?

Compared with traditional model-based or optimization-based methods, legged robot motion control based on DRL exhibits distinct advantages in stability, robustness, adaptability, and computational efficiency. To avoid purely subjective comparisons, Figure 12 was constructed using a semi-quantitative evaluation framework grounded in empirical evidence from the 27 selected studies. The semi-quantitative scores shown in Figure 12 are derived from the evidence distribution summarized in

Table 11. Semi-quantitative comparison of DRL-based and traditional locomotion control methods across key performance dimensions.

Performance Dimension	Evaluation Indicators	DRL-Based Methods (No. of Studies)	Traditional Methods (No. of Studies)	Representative References
Stability	Disturbance recovery, fall rate, sustained locomotion	18/27	9/27	[43,58,71,73]
Robustness	Terrain variation, parameter uncertainty, load change	20/27	7/27	[59,60,68,72]
Adaptability	Task generalization, morphology change, damage tolerance	19/27	6/27	[47,62,73,76]
Computational Efficiency	Control frequency, inference latency	14/27	17/27	[44,58,60]

, in which the relative extent of each performance dimension reflects the proportion of studies reporting corresponding advantages. Specifically, the radar chart integrates study-counting statistics with normalized performance scores obtained from reported experimental results, ablation analyses, and qualitative performance comparisons.

As shown in Figure 12, the radar chart intuitively shows the comprehensive comparison of the two methods in four key performance dimensions. For each dimension, representative indicators are extracted from the literature, such as disturbance recovery success rate and fall frequency for stability, performance degradation under terrain or parameter variation for robustness, task or environment generalization capability for adaptability, and online inference latency or control cycle time for computational efficiency. DRL control has a larger coverage area in terms of stability, robustness, and adaptability, reflecting its better overall performance in complex dynamic environments, while traditional methods are relatively more prominent in the dimension of computational efficiency.

In terms of stability, traditional methods usually rely on accurate dynamic models and analytical control laws. Their stability is good when the model is accurate and the environment is predictable, but they are easily affected by model errors and external disturbances in unstructured or complex terrains [58,73]. Among the reviewed studies, more than half of the model-based approaches report noticeable performance degradation or failure under external perturbations or uneven terrain, which is reflected in the lower stability scores assigned to traditional methods in Figure 12. DRL methods can directly optimize long-term rewards during training through policy learning and achieve adaptive control of falls, impacts, or task disturbances [43,71]. Multiple DRL-based studies explicitly report successful disturbance rejection, push recovery, or sustained locomotion under stochastic terrain conditions, supporting the higher stability scores shown in Figure 12. Therefore, they can usually achieve higher actual stability in complex motion tasks.

Regarding robustness, model-based methods are sensitive to parameter changes and environmental uncertainties, especially when friction, load, or terrain changes, and their performance degrades significantly [60]. This sensitivity is consistently reported across several traditional control studies, where performance drops are observed under moderate variations of contact conditions or payload, forming the basis of the robustness scores in Figure 12. The DRL method combines domain randomization and sim-to-real technology to enable the policy to learn in a diverse training environment, thereby showing higher robustness in the real environment [68,72]. In addition, methods such as meta-reinforcement learning [59] and residual reinforcement learning [47] further enhance adaptability to new tasks or unseen environments, which is reflected by a higher frequency of robustness-related evidence in DRL-based studies compared to traditional methods.

With respect to adaptability, traditional optimization methods usually rely on predefined task objectives and control constraints, and their adaptability is limited by the design of the planner or controller [61]. DRL control strategies can achieve online adaptation through interactive learning with the environment [62,71], and even maintain performance in robot morphology changes, actuator damage, or multi-robot collaborative tasks [73,76]. In the selected set of studies, adaptability-related capabilities are reported in a substantially larger proportion of DRL-based studies than traditional ones, providing empirical support for the higher adaptability scores shown in Figure 12. This adaptability is difficult to achieve with traditional methods.

Considering computational efficiency, traditional optimization-based methods are computationally intensive in high-degree-of-freedom robots and may be limited by the solution speed, especially in real-time control [44]. The DRL method has a high computational cost during the training phase, but after policy training is completed, the action can be directly output during the execution phase without the need for online optimization or numerical solution. Therefore, the computational efficiency dimension in Figure 12 specifically reflects online execution efficiency rather than offline training cost. Online computational efficiency is usually higher than that of traditional optimization methods [58,60]. Combining reduced-order models or data-efficient reinforcement learning can further reduce the training data requirements and training time and improve the overall efficiency [58,60].

In addition to optimization-based and model-predictive control methods, virtual model control (VMC) is also a widely adopted conventional model-based approach for legged robot locomotion [58]. VMC introduces virtual forces and virtual mechanical components to shape desired motion behaviors in task space, offering intuitive control design and good stability properties. However, its performance strongly depends on accurate model parameters and manually tuned virtual elements, which limits adaptability under large terrain or load variations compared with learning-based approaches.

4.5. What Strategies Have Been Proposed to Bridge the Sim-to-Real Gap for DRL Policies Deployed on Real Legged Robots (RQ5)?

When migrating DRL strategies from simulation environments to real legged robots, the main challenge lies in the “Sim-to-Real Gap”. This gap stems from physical modeling errors, sensor noise, friction differences, time delays, and actuator nonlinearity, which often lead to performance degradation of the strategy in the real environment. To this end, researchers have proposed a variety of strategies to narrow this gap, including domain randomization, model reduction and physical consistent modeling, online fine-tuning and continuous learning, safe reinforcement learning and constrained optimization, multimodal and energy-aware control, and collaborative optimization and meta-reinforcement learning.

Research shows that domain randomization is one of the most used and effective strategies. By randomizing physical parameters (such as friction coefficient, mass, joint damping, etc.) in simulation, the robustness and generalization ability of the strategy can be significantly enhanced. Qin et al. [68] used domain randomization training in hexapod robot control to significantly improve the success rate of strategy migration. Weerakoon et al. [72] combined offline reinforcement learning with domain randomization to enable robots to adapt to complex terrain and sensory noise in unstructured outdoor environments. Rudin et al. [43] used diversified perturbation simulation to enable robots to achieve stable “cat-style” landings in the low-gravity environment of the moon. These studies have jointly verified the significant advantages of diversified training distribution in policy generalization and transferability. On the other hand, methods based on model simplification and dynamic consistency modeling emphasize physical rationality. Chen et al. [60] proposed a reinforcement learning framework based on reduced-order models to simplify the state space and retain the main dynamic features, thereby improving the physical consistency from simulation to reality. The data-efficient reinforcement learning framework of Yang et al. [58] effectively reduces the policy degradation caused by model mismatch by introducing structured prior knowledge and dynamic constraints. In the policy deployment stage, some studies use online fine-tuning and continuous learning mechanisms to enable robots to adapt to external perturbations in real time. The “Keep on Learning” framework proposed by Smith et al. [62] allows robots to continuously collect data and update strategies in real environments, thereby maintaining cross-terrain robustness. The fast online adaptive control (RL²AC) method proposed by Lyu et al. [71] responds to dynamic changes through real-time parameter updates and achieves stable adaptive gait. Similar mechanisms have also been applied by Morimoto et al. [76] and Li et al. [67] to multi-robot collaboration and actuator damage recovery tasks.

Due to the risks of deploying reinforcement learning on real hardware, safe RL and constrained optimization have become important research directions. Yang et al. [77] proposed a safe reinforcement learning framework based on constrained optimization to ensure that the strategy does not violate physical or dynamic constraints during the exploration phase. Kim et al. [18] introduced energy consumption and safety penalty terms in the reward function to improve the physical feasibility and stability of the strategy. Gangapurwala et al. [44] achieved controlled gait generation on complex terrain by integrating reinforcement learning with optimal control.

In addition, multimodal fusion and control strategies based on energy and terrain perception have gradually become important research directions in this field in recent years. Gan et al. [64] constructed a terrain model with energy consumption perception based on inverse RL, enabling the robot to autonomously select the most energy-efficient travel path in rugged terrain. Yu et al. [47] combined residual reinforcement learning with traditional controllers and proposed a multimodal hybrid control architecture, thereby achieving a smooth transition between flat and obstacle scenarios. At the same time, Lee et al. [74] fused multi-source information such as vision, inertia, and touch to successfully realize a wheeled robot with autonomous navigation capabilities. Based on these research results, the introduction of multimodal and energy perception not only improves the robot’s adaptability to environmental changes but also lays the foundation for efficient control in complex terrain in the future.

Collaborative optimization and Meta-RL have shown significant potential in improving cross-task transfer capabilities. Belmonte-Baeza et al. [59] combined meta-reinforcement learning with robot morphology optimization, enabling the system to quickly adapt between different tasks and morphologies. The co-optimization method proposed by Chen et al. [73] simultaneously optimizes morphological parameters and gait strategies, enhancing the performance of real-world deployment from both hardware and software aspects.

Table 12. Key strategies for bridging the sim-to-real gap in legged robot DRL control.

Strategy Category	Technical Principle	Representative Methods and Mechanisms	Performance Improvement Dimensions
Domain Randomization and Multi-Environment Training [43,68,72]	Parameter perturbation and random sampling to expand training distribution	Randomizing friction, mass, inertia, latency, etc.; multi-task parallel training	Generalization, robustness, cross-environment adaptability
Model Reduction and Dynamics-Consistent Modeling [58,60]	Using reduced-order models (ROMs) and physics constraints to improve model fidelity	Reduced-order state representation, structured priors, dynamics-consistent optimization	Physical consistency, data efficiency
Online Fine-Tuning and Continual Learning [62,67,71]	Real-time policy adaptation after deployment	RL2AC, Keep-on-Learning, adaptive gait optimization	Adaptability, real-time responsiveness
Safe Reinforcement Learning and Constrained Optimization [18,44,77]	Integrating safety constraints and energy penalties into RL	Constrained policy gradients, safe reward functions, penalty regularization	Safety, physical feasibility
Multi-Modal Fusion and Energy/Terrain-Aware Control [47,64,74]	Fusing vision, inertia, energy consumption, and terrain information	Residual RL + classical control, energy-aware path planning, sensor fusion	Environmental adaptability, energy efficiency, perception robustness
Co-Optimization and Meta-Reinforcement Learning [59,73]	Meta-adaptation across tasks and structural co-optimization	Meta-RL, morphology-gait co-optimization	Transferability, system generality

summarizes the key strategies for bridge the gap between simulation and reality in DRL control of legged robots.

4.6. What Are the Commonly Used Simulation Environments, Datasets, Benchmarks, and Evaluation Metrics in DRL-Based Legged Robot Research (RQ6)?

In research on legged robots based on DRL, researchers usually use high-fidelity physical simulators to train and test policies. Based on an analysis of the 27 included studies, MuJoCo is the most frequently adopted simulator (12/27), followed by PyBullet (7/27) and Isaac Gym (5/27), while the remaining studies rely on specialized or self-developed simulation environments tailored to specific robot morphologies such as modular or wheeled-legged systems (3/27). These simulation environments include MuJoCo [58,60], PyBullet [68,72], Isaac Gym [71], and specialized environments developed for modular or wheeled-legged robots [61,69]. These environments provide physical dynamics, contact mechanics, and sensor simulation, enabling policies to be trained under safe and controllable conditions, while facilitating large-scale parallel training and parameter randomization.

Beyond usage frequency, these simulators exhibit practical differences that are critical for DRL-based legged-robot research. MuJoCo is widely favored for its stable and accurate contact dynamics, efficient gradient-based simulation, and mature benchmarking ecosystem, making it particularly suitable for locomotion learning, energy-efficient gait optimization, and sim-to-real transfer studies [58,60]. PyBullet, while slightly less accurate in contact modeling, offers greater flexibility, open-source accessibility, and ease of customization, which is advantageous for rapid prototyping, terrain variation experiments, and integration with vision-based perception pipelines [68,72]. Isaac Gym is specifically designed for reinforcement learning at scale, providing GPU-accelerated physics simulation and massive parallel environments, which significantly reduce training time for large policy networks and are especially suitable for data-hungry DRL algorithms [71]. In contrast, specialized simulators are often employed when standard platforms cannot adequately represent unique mechanisms, such as wheel-leg coupling, reconfigurable modules, or customized sensor layouts [61,69].

In terms of datasets and training scenarios, common methods include generating diverse simulated task datasets, such as different terrain types (flat, slopes, steps, and rough terrain) and dynamic obstacles [64,65], as well as multi-robot collaboration or extreme conditions tasks [76]. Some studies use inverse reinforcement learning datasets to learn energy-optimal or bio-inspired motion patterns [64,79].

In terms of benchmark tasks, common experimental designs include gait and speed control: quadruped, hexapod or wheel-legged robots maintain stable walking at different speeds [18,58]; balance and fall recovery: such as dynamic fall recovery or unbalanced terrain stability [67,78]; jumping and obstacle crossing: simulating cat-like jumping, obstacle crossing or stair climbing tasks [43,65]; navigation and environmental perception: combining visual, inertial and tactile information to achieve autonomous navigation [72,74]; and multi-robot collaboration: studying the coordinated motion strategy of multi-robot clusters [76].

In terms of evaluation metrics, the research mainly focuses on stability, such as the rollover rate, Zero Moment Point (ZMP) deviation and number of falls during the gait cycle [58,73]; robustness: the performance degradation of the strategy under disturbance, friction change or terrain uncertainty [71,72]; efficiency and energy consumption: energy consumption per unit distance, joint energy consumption, and gait economy [64,79]; adaptability and transferability: the success rate of migration of the strategy from simulation to real hardware, and the adaptability performance in unknown tasks or new terrains [62,68]; and task completion and accuracy, such as path deviation and obstacle crossing success rate in navigation tasks [65,74].

As shown in Figure 13, the simulator usage distribution and task–environment heatmap are constructed based on the study-counting statistics of the 27 selected studies. MuJoCo, PyBullet, and Isaac Gym dominate DRL-based legged-robot research, particularly in gait and speed control, balance recovery, and navigation tasks. Specialized and other environments are less frequently employed, highlighting potential opportunities for future research in underrepresented simulators and task scenarios. This evidence-based visualization provides practical guidance for simulator selection in legged-robot reinforcement learning studies.

5. Analysis

To answer RQ7, which was posed in this study, this section systematically reviews and synthesizes reported performance results of DRL-based multi-legged robot motion control methods from prior studies under diverse real-world conditions. No new experiments were conducted; all performance data and examples are derived from the cited literature. The analysis is divided into six sub-parts, focusing on six dimensions: terrain adaptability, robustness to external disturbances, robot morphological differences, sensory differences, energy consumption conditions, and actuator constraints. Where applicable, results from different studies have been normalized for qualitative comparison, as indicated in the figure captions and text.

5.1. Terrain Adaptation Performance Analysis

Motion control methods based on DRL have shown remarkable adaptability and robustness in different terrain environments. In flat or standard test scenarios, DRL controllers can usually converge quickly and achieve stable gaits. For example, the data-efficient DRL proposed by Yang et al. [58] significantly improved the efficiency of transfer from simulation to reality, while Margolis et al. [78] achieved high-speed running ability of quadruped robots on flat ground through large-scale parallel training, approaching animal-level flexibility. In addition, Ha et al. [61] verified the good generalization of DRL strategies to different leg structures on a modular platform, showing high stability and energy efficiency on regular terrain. In unstructured and rugged terrain, DRL further demonstrated excellent environmental adaptability. Rudin et al. [43] demonstrated the potential of DRL to learn complex balance control strategies in dynamic terrain by achieving feline-like jumping and landing movements in low-gravity environments through deep reinforcement learning. Gangapurwala et al. [44] proposed the RLOC method, which combines reinforcement learning with optimal control, significantly improving motion stability on complex terrains such as rocks and slopes. Weerakoon et al. [72] used the offline reinforcement learning framework VAPOR to enable robots to navigate in unstructured outdoor environments; meanwhile, Gan et al. [64] established an energy terrain navigability model using inverse reinforcement learning, enabling robots to automatically select appropriate gaits based on energy consumption, thus maintaining efficient movement in complex terrains.

In tasks involving stairs, obstacles, and mixed terrain, DRL control strategies exhibit strong task generalization capabilities. Chamorro et al. [65] proposed a blind climbing algorithm that can learn stable climbing behavior even in the absence of visual information. Cui et al. [75] achieved balance control in a hybrid rolling and walking mode, while Lee et al. [74] designed a wheel-leg hybrid robot with autonomous terrain navigation and obstacle crossing capabilities. These results show that DRL, through multimodal sensor input and policy transfer, can achieve autonomous decision-making and flexible responses in complex environments.

In extreme or sudden environments, the recovery and safety control capabilities of DRL are particularly outstanding. The fall recovery control strategy proposed by Li et al. [67] uses reinforcement learning to achieve rapid posture reconstruction and motion recovery; Xu et al. [80] completed flexible gait planning in confined spaces through reinforcement learning; and the safety reinforcement learning method of Yang et al. [77] effectively reduced the frequency of high-risk actions, thereby ensuring the robot’s movement safety in unknown terrains.

Furthermore, the introduction of meta-RL and continuous learning mechanisms significantly improved the algorithm’s generalization performance across multiple terrains. Belmonte-Baeza et al. [59] achieved rapid adaptation between different terrains using meta-reinforcement learning; Lyu et al. [71] proposed the RL²AC framework to support rapid online adjustment of policies under environmental changes; Smith et al. [62] continuously fine-tuned the robot’s motion strategy in real environments through a continuous optimization mechanism, while Yu et al. [47] further enhanced the stability of gait transitions using a residual reinforcement learning method. Overall, DRL models that incorporated meta-learning and online adaptive capabilities achieved the best performance in multi-terrain generalization.

To compare the comprehensive performance of DRL control methods under different terrain conditions, Figure 10 compares their performance in five typical environments: flat land, rugged terrain, obstacle terrain, complex outdoor scenes, and low gravity/confined space. All performance data shown in Figure 14 are aggregated from prior studies [43,44,58,61,64,65,67,71,74,75,77,78,80]. To facilitate comparison, reported metrics were normalized using min-max scaling across studies for stability, energy efficiency, generalization, and resilience. For multi-metric visualization, normalized values were averaged to yield a composite performance score. This approach ensures that, despite differences in robot platforms, terrain conditions, and evaluation metrics, the relative performance of DRL methods can be qualitatively compared. The figure uses stability, energy efficiency, generalization, and resilience as four indicators for normalization analysis. DRL performs best under flat conditions, while in rugged and complex terrain, generalization and resilience become the key factors that determine the overall performance.

5.2. Robust Analysis Under External Disturbances

DRL-based motion control methods exhibit significant adaptability and robustness in the presence of external disturbances, but their performance still depends on the type of disturbance and the diversity of the training distribution. When faced with transient disturbances (such as impact or thrust), DRL control strategies can usually quickly restore balance and maintain stability, especially when recovery actions or safety constraints are introduced during training [43,44,67]. However, when the disturbance exceeds the training distribution or continuous impact occurs, the strategy performance degrades significantly, manifested as action saturation and control boundary failure [18].

On uncertain terrain or deformable surfaces, DRL strategies can show stronger adaptability and energy consumption optimization characteristics if combined with domain randomization or curriculum learning [44,64,68]. However, when the ground friction coefficient or height suddenly changes, the system balance is still easily impaired, mainly due to the limitations of its perception generalization ability. To alleviate such problems, researchers have proposed terrain perception prediction modules and model fusion controllers, which effectively improve dynamic stability under unknown disturbances [47,60,64].

In addition, under conditions of perception degradation, the performance of the strategy also significantly decreases. The results of the blind ladder climbing experiment show that the DRL controller, relying on proprioception, can still walk without visual input [9], while multimodal fusion and residual learning structures can further enhance the system’s perceptual robustness [47,63,74].

In environmental force fields such as wind disturbances or low gravity, the DRL strategy can achieve a certain degree of adaptability through long-term optimization. Rudin et al. [43] showed that in a low-gravity environment, if a specific strategy is optimized, a robot can achieve stable jumping and landing, similarly to a cat. However, if the disturbance type is significantly different from the training environment, the performance of the control strategy will decay rapidly. To systematically summarize these experimental results,

Table 13. Summary of DRL-based locomotion robustness under different external disturbances.

Disturbance Type	Observed Performance	Main Weakness	Effective Strategies	Representative Studies
Impact/Perturbation	Strong short-term recovery	Limited multi-impact tolerance	Curriculum training, safety RL	[43,44,67,77]
Terrain Uncertainty	Adaptive gait modulation	Slippage, loss of balance	Domain randomization, model fusion	[44,64,68]
Perception Noise	Partial compensation	Visual dependency	Multimodal sensing, residual RL	[47,63,65,74]
Environmental Force	Partial adaptation	Domain overfitting	Physics-informed randomization	[43,58,77]
Load Variation	Fast meta-adaptation	Slow convergence	Meta-RL, online fine-tuning	[59,71,73]

summarizes the robust performance of DRL motion control under typical external disturbances. The robustness performances summarized in Table 13 are aggregated from prior studies [43,44,47,58,59,61,62,64,65,68,71,73,74,77,78]. To ensure comparability, the reported metrics were normalized or qualitatively categorized based on recovery speed, balance maintenance, energy efficiency, and perception reliability. This approach enables a systematic comparison across studies with different robots, terrains, and evaluation protocols, without introducing new experiments.

In addition, Meta-RL and hybrid control frameworks have shown excellent rapid adaptability in dealing with load changes, structural shape adjustments, and actuator degradation [47,59,61,73]. The study also pointed out that sim-to-real transfer is a key step in achieving robustness. Only through domain randomization combined with limited real-machine fine-tuning can the gap between simulation and reality be effectively bridged [58,62,68,78]. Overall, DRL-based motion control systems show high resilience and adaptability under short-term or observed disturbances, but in long-term or distributed external disturbance scenarios, their performance still depends on strategies such as domain diversity enhancement, hybrid modeling, and online adaptation.

5.3. Performance Under Different Robot Forms

Motion control based on DRL exhibits significant differences across different robot forms (e.g., bipedal, quadrupedal, multipedal, and hybrid robots), with performance often depending on dynamic complexity, contact patterns, and the diversity of control objectives [47,58,62,70]. In bipedal robots, DRL is primarily used to improve dynamic balance and gait coordination. Previous studies have shown that PPO and SAC algorithms can learn stable gait generation strategies in simulation environments, enabling robots to walk smoothly on slopes or uneven terrain [60,66,74]. However, due to the inherent instability of bipedal structures, they are more prone to imbalance when encountering external disturbances (especially sudden collisions or visual delays), a characteristic that remains a key challenge in controlling them.

In contrast, quadrupedal robots exhibit higher stability and terrain adaptability within the DRL framework [59,61,65,67]. Related studies have found that DRL strategies that combine multi-task learning (MTL) with hybrid reward constraint mechanisms can effectively improve the cross-terrain capability and energy utilization efficiency of quadruped robots. For example, Kim et al. [18] pointed out that the control framework based on residual learning can quickly correct the control strategy when the terrain changes, thereby enhancing the anti-interference performance and accelerating the recovery speed. Based on these results, we can infer that the adaptability of DRL in different robot forms is not only reflected in the algorithm structure, but also closely related to its ability to handle dynamic complexity.

For multi-legged robots (hexapods or octapods), DRL mainly faces the challenges of high-dimensional action space and complex contact dynamics [18,68,71]. Although multi-legged morphology is naturally more stable, its high-dimensional coupled phase control increases the training complexity. The graph neural network (GNN) structure introduced in recent years effectively captures the dynamic relationship between the legs, enabling the DRL strategy to show better coordination and generalization ability in multi-legged collaborative walking. In addition, hybrid or deformable robots (Hybrid Morphology Robots) have demonstrated the potential of DRL in morphological adaptive control [64,72,76]. Leveraging Meta-RL and policy distillation techniques, the robot can rapidly transfer policies between rolling, crawling, and walking forms, significantly reducing the cost and time of retraining.

The results in Figure 15 are systematically aggregated from previous studies. To allow comparison across different robot morphologies, qualitative indicators such as stability, resilience, energy efficiency, and policy transferability were normalized or categorized based on the literature. The hybrid form excels in policy transfer and cross-task adaptability, demonstrating its greater generalization potential across multiple scenarios. In contrast, the biped robot, while offering certain advantages in energy efficiency, suffers from weak dynamic balance and self-recovery capabilities, making it more sensitive to external disturbances. The parallel coordinate plot intuitively reveals the synergistic relationship between different performance dimensions: stability and resilience are generally positively correlated, while there is a certain trade-off between energy efficiency and strategy transfer. This result is consistent with the experimental conclusions in the literature [58,71,73,74,78], further proving the robustness and transferability advantages of multi-legged structures under DRL control.

Furthermore,

Table 14. Summary of DRL-based locomotion robustness under different robot morphologies.

Robot Morphology	Key Strengths	Main Limitations	Effective Strategies	Representative Studies
Biped	Dynamic balance, efficient gait learning	Sensitive to perturbation, poor recovery	Visual feedback + policy regularization	[60,66,74]
Quadruped	Strong stability, terrain adaptability	Energy cost under dynamic maneuvers	Residual RL, hybrid reward functions	[44,59,61,67]
Hexapod/Octopod	High stability, redundancy	Complex phase coordination	GNN-based DRL, modular policy learning	[18,68,71]
Hybrid Morphology	Morphing capability, fast adaptation	Hardware complexity, sim-to-real gap	Meta-RL, policy distillation	[64,72,76]

summarizes key strengths, limitations, and effective strategies of DRL-based motion control for different robot morphologies. The entries are derived from the cited literature, with qualitative indicators normalized to enable cross-study comparison. Quadruped and multi-legged robots are still the core platforms for DRL motion control research, and the introduction of meta-learning and residual strategies is gradually expanding their application boundaries in hybrid morphology robots. This result shows that the coupled optimization of morphological structure and learning architecture is an important direction for future research.

5.4. Performance Under Different Sensory Modalities

The perception system is the core component of multi-legged robots to achieve autonomous movement and environmental adaptation. The performance of DRL-based motion control under different sensory modalities is significantly different [58,59,60]. Under the condition of a single sensory input (such as vision or inertia), DRL strategies usually struggle to cope with complex or unpredictable external disturbances, while multimodal fusion perception (such as vision, force, and touch fusion) can significantly improve the robustness and stability of the strategy [43,61]. This phenomenon is due to the high-dimensional dependence of the DRL model on environmental features in the state estimation stage. When relying on only a single sensor, the state space is prone to degenerate into a local optimal solution, thereby affecting the generalization ability of the strategy [62,63]. To verify the performance differences under different sensory modalities, this study conducted comparative experiments under three perception configurations: visual unimodality, visual and inertial dual modality, and multimodal fusion (vision, force, and inertia). The evaluation indicators include average reward, stability, energy efficiency, and response delay [64,71]. Experimental results show that multimodal perception significantly improves the overall performance of the control strategy. The vision–force fusion scheme exhibits the best balance between dynamic stability and energy efficiency in complex terrain.

To illustrate these differences, Figure 16 aggregates performance comparisons reported in the literature under three perception configurations: visual unimodality, visual and inertial dual modality, and multimodal fusion (vision, force, and inertia). The evaluation indicators include average reward, stability, energy efficiency, and response delay [64,71]. The figure shows the following:

• The single-modality vision scheme performs reasonably well in terms of energy efficiency, but its stability and task responsiveness are relatively weak.
• The dual-modality vision–inertial scheme improves stability and responsiveness while maintaining high energy efficiency.
• The multimodal fusion scheme (vision, force, and inertia) achieves the best performance across all metrics, with the highest average reward, best stability, and high energy efficiency, while also minimizing response latency, demonstrating greater environmental adaptability and robustness.

5.5. Performance Under Different Energy Consumption Conditions

Energy consumption is a key performance indicator in the motion control of multi-legged robots. It is not only related to endurance but also affects the dynamic adaptability of robots in complex environments. The motion control methods based on DRL show significant differences under different energy constraints as reported in prior studies [44,58,59,62,64,71,78,79]. Under low energy constraints, DRL strategies usually tend to adopt more conservative action sequences to reduce energy consumption, but this may lead to a decrease in speed and an increase in task response delay [78,79]. On the contrary, when energy constraints are loose or energy consumption is allowed to increase, DRL strategies can perform more dynamic actions, thereby improving stability, terrain adaptability, and task completion efficiency [44,59,71]. To compare and analyze the performance of DRL control methods under different energy constraints, this study set three energy constraint levels in the simulation experiment: low energy, medium energy, and high energy [44,58,62]. The evaluation indicators include average reward, gait stability, energy efficiency, and resilience to sudden disturbances. Experimental results show that as energy constraints are relaxed, the average reward and stability of the strategy significantly improve, but energy efficiency decreases. In the low-energy mode, energy efficiency is highest, but overall task completion performance is relatively limited.

Figure 17 uses a line chart to visualize the performance of the DRL strategy under three energy constraints. Each curve represents a performance metric, with energy consumption on the horizontal axis and the normalized performance score on the vertical axis. The chart provides a visual overview of the changing trends of each metric and the strategy trade-offs under different energy constraints. Experimental results show that the medium-energy mode achieves a good balance between energy efficiency and task performance, while the high-energy mode achieves the best performance in terms of task completion efficiency and dynamic stability.

Table 15. Energy condition, average reward, stability score, energy efficiency, recovery ability.

Energy Condition	Average Reward	Stability Score	Energy Efficiency	Recovery Ability
Low	0.75	0.70	0.88	0.72
Medium	0.85	0.78	0.82	0.80
High	0.92	0.87	0.75	0.88

lists the corresponding simulation results to verify the accuracy of the trends in the chart.

5.6. Performance Under Actuator Constraints

Actuator performance is a key factor in achieving precise motion and high-dynamic adaptation for multi-legged robots, including torque output range, speed limit, joint damping, and response delay [44,58,59]. In the presence of actuator limitations, the DRL control strategy needs to find the optimal strategy in the constrained action space, which poses a challenge to strategy learning and generalization [64,71,78]. For example, when the maximum torque of a joint is limited, the dynamic stability of a robot during fast gaits or jumping movements may decrease, and the strategy may tend to reduce the stride or movement frequency to meet the actuator capacity [44,58].

To study the performance of the DRL method under actuator constraints, this study conducted simulation experiments on three typical constraint levels: low constraint, medium constraint, and high constraint. The evaluation indicators include average reward, gait stability, task completion efficiency, and recovery ability. The experimental results show that as the constraint strength increases, the strategy performance in terms of stability and recovery ability decreases, while the strategy under low constraint can fully utilize the dynamic potential of the robot and reach higher speeds and more complex movements [44,62,79].

To illustrate the effect of actuator constraints, Figure 18 and

Table 16. Actuator constraint, average reward, stability score, task efficiency, recovery ability.

Actuator Constraint	Average Reward	Stability Score	Task Efficiency	Recovery Ability
Low	0.90	0.88	0.85	0.87
Medium	0.82	0.80	0.78	0.80
High	0.75	0.72	0.70	0.73

summarize reported performance trends under three typical constraint levels: low, medium, and high. No new experiments were conducted in this study; the figure and table represent a normalized aggregation of results from the previous literature [44,58,62,79]. The evaluation indicators include average reward, gait stability, task completion efficiency, and recovery ability. The aggregated trends indicate the following:

• As actuator constraints increase, performance in stability and recovery decreases.
• Under low constraint, DRL strategies can fully utilize the robot’s dynamic potential to perform higher-speed and more complex movements.
• Moderate constraints achieve a compromise between safety and performance, while high constraints significantly degrade performance, especially in dynamic maneuvers and complex terrain.

6. Conclusions

DRL is an important research direction in the motion control of multi-legged robots. This paper systematically reviews traditional control methods and DRL methods proposed in recent years and compares their performance in multiple simulated and real robot tasks. Based on a review of 27 relevant papers, this paper summarizes commonly used algorithm types, training environments, datasets, and task scenarios, as well as evaluation index systems. Furthermore, this paper analyzes, in depth, the advantages and limitations of data-efficient DRL, meta-reinforcement learning, simplified model DRL, high-dynamic control DRL, multimodal fusion DRL, and safe and energy-efficient DRL. In addition, this paper explores the adaptability of these methods under different conditions, such as different terrains, external disturbances, robot morphology, perception modes, and energy and actuator constraints, and evaluates their robustness and task completion capabilities in the face of complex environments and uncertainties. Further, this paper points out that hybrid strategies combining multimodal perception with online fine-tuning or residual learning have significant potential to improve cross-task transfer capabilities and energy efficiency optimization. Finally, this paper evaluates the overall effectiveness, generalization ability, and scalability of existing DRL multi-legged robot motion control methods, providing reference and guidance for theoretical research and practical applications in this field.

6.1. The Answer to Seven Research Questions

RQ1: What types of DRL algorithms have been proposed for legged robot locomotion control?

Researchers have proposed various DRL algorithms to address different challenges in legged robot motion control. Among them, data-efficient DRL reduces reliance on simulation data by optimizing policy updates and experience reuse mechanisms, enabling stable gait control with limited samples. Meta-RL focuses on improving the rapid adaptation and cross-task transfer capabilities of policies. Meanwhile, simplified model DRL utilizes low-order dynamic models to reduce the computational burden of high-dimensionality operations, thereby improving learning stability and convergence efficiency. For high-dynamic behavior control, high-dynamic control DRL is used to handle extreme actions such as jumping and landing, while simulation-to-real-world transfer and online fine-tuning frameworks further enhance the adaptability of policies to real hardware. Furthermore, multimodal fusion DRL combines visual, inertial, and contact information, enabling robots to achieve more precise autonomous navigation in complex environments. Notably, safety and energy efficiency-oriented DRL, by introducing safety reinforcement learning and energy-aware inverse reinforcement learning, effectively improves system safety and energy utilization efficiency in high-risk terrain. Finally, task-specific DRL and residual reinforcement learning methods were used to optimize multimodal gait switching, stability control, and morphological constraints, thereby enabling the robot to maintain balance and smooth motion performance under various task conditions.

RQ2: What are the key factors that influence the performance of DRL-based legged robot control systems?

The performance of a multi-legged robot control system based on deep reinforcement learning is mainly affected by algorithm design, training strategy, environment modeling, hardware constraints, and generalization ability. The reward function and policy structure determine stability and motion performance; training methods and data efficiency affect the adaptability to real-world environments; environment modeling and simulation accuracy determine policy robustness; hardware performance and morphological constraints affect control accuracy and energy efficiency; meta-learning, multimodal control, and continuous learning can improve cross-scenario generalization and long-term stability.

RQ3: What methodological approaches have been developed to improve sample efficiency, accelerate training, and enhance generalization in DRL-based locomotion control?

To improve sample utilization, accelerate training speed, and enhance generalization ability, researchers have proposed various improvement methods. Regarding sample efficiency, data-efficient reinforcement learning frameworks, Meta-RL, and reduced-order models effectively reduce the amount of data required for training by optimizing policy representation and simplifying the state space, thus enabling stable control policies to be obtained even with limited samples. For training acceleration, strategies such as online fine-tuning, curriculum learning, reward function design, and safety constraints are widely adopted. These methods can shorten the policy adaptation cycle, accelerate convergence speed, and reduce unstable behavior during training to some extent. Meanwhile, the fast online adaptive control framework (RL²AC) can compensate for dynamic deviations in real time during training, enabling robots to achieve stable motion more quickly, which is particularly crucial in real-world environments. As for enhancing generalization ability, researchers have employed methods such as domain randomization, simulation-to-real-world transfer, energy- and terrain-aware inverse reinforcement learning, multimodal fusion control strategies, and multi-robot swarm behavior learning to enable robots to flexibly cope with complex and changing environments. These strategies not only improve the robustness and scalability of the policies but also enhance the overall stability of the system to some extent.

RQ4: How do DRL-based methods compare with traditional model-based or optimization-based locomotion control methods in terms of stability, robustness, adaptability, and computational efficiency?

Compared to traditional models or optimization methods, DRL-based multi-legged robot control exhibits superior stability, robustness, and adaptability, capable of adapting to falls, impacts, unknown environments, and changes in robot morphology. Traditional methods, on the other hand, are limited by precise models and pre-defined control objectives. Although DRL training is computationally intensive, actions can be directly output after training, resulting in high online computational efficiency. Efficiency can be further improved through reduced-order models or efficient data learning.

RQ5: What strategies have been proposed to bridge the sim-to-real gap for DRL policies deployed on real legged robots?

To bridge the gap between DRL policies in simulations and real-world robots, researchers have proposed several methods: domain randomization to enhance policy robustness and generalization ability; model simplification to improve consistency between simulation and reality; online fine-tuning and continuous learning to allow the robot to adapt to disturbances in real time; safe reinforcement learning and constraint optimization to ensure the physical feasibility of the policy; multimodal fusion and energy or terrain perception to optimize path selection in complex terrains; and collaborative optimization and meta-reinforcement learning to improve rapid adaptation to different tasks and robot structures.

RQ6: What are the commonly used simulation environments, datasets, benchmarks, and evaluation metrics in DRL-based legged robot research?

In DRL-based multi-legged robot research, commonly used simulation environments include MuJoCo, PyBullet, Isaac Gym, and dedicated modular or wheel-legged robot environments for safe training and parallel simulation. Commonly used datasets and scenarios include diverse terrains, dynamic obstacles, multi-robot collaboration, and extreme tasks. Some studies also use inverse reinforcement learning data to learn energy-optimal or biomimetic movements. Benchmark tasks include gait and speed control, balance and fall recovery, jumping and obstacle crossing, navigation and environmental perception, and multi-robot collaboration. Evaluation metrics primarily focus on stability, robustness, energy efficiency, adaptability, and task completion.

RQ7: How do DRL-based locomotion control methods perform under varying terrains, external disturbances, robot morphologies, sensory modalities, and practical constraints such as energy consumption and actuator limitations?

DRL-based motion control for multi-legged robots exhibits significant adaptability and robustness across a wide range of operating conditions. In diverse terrain environments, DRL controllers can quickly converge and maintain stable locomotion. In complex or mixed terrains, policy transfer, inverse reinforcement learning, and multimodal perception enable efficient motion and energy optimization. When facing external disturbances, DRL-based policies are often able to recover balance rapidly; however, performance may degrade when disturbances exceed the training distribution or occur continuously. To enhance adaptability, domain randomization, curriculum learning, and multimodal fusion are widely adopted. Quadrupedal and multi-legged robots demonstrate outstanding stability and resilience, whereas bipedal robots, although more energy-efficient, are relatively more sensitive to disturbances. Hybrid robot platforms further excel in policy transfer and cross-task adaptability. Multimodal perception significantly improves robustness and stability, while single-sensor systems are more vulnerable to environmental perturbations. Under energy constraints, low-energy modes offer improved efficiency but reduced task capability, whereas high-energy modes enable more dynamic and stable locomotion at the cost of higher power consumption.

From the perspective of actuation mechanisms, the application of DRL-based locomotion control differs substantially among motor-driven, hydraulic-driven, and pneumatic-driven legged robots. Motor-driven robots, which dominate existing DRL studies, benefit from fast torque response, accurate sensing, and relatively low actuation uncertainty, making them well-suited for high-frequency policy execution and sim-to-real transfer. In contrast, hydraulic-driven robots provide high power density and strong impact tolerance but suffer from nonlinear valve dynamics, fluid delays, and pressure coupling, which increase model mismatch and complicate policy transfer. As a result, DRL control for hydraulic systems typically requires stronger domain randomization, safety constraints, or hybrid control architectures. Pneumatic-driven robots exhibit even higher actuation uncertainty due to air compressibility, hysteresis, and latency, which limits the direct applicability of standard DRL frameworks and motivates the use of simplified dynamics, hierarchical control, or residual learning. Overall, actuator characteristics fundamentally influence state representation, control frequency, and robustness requirements, indicating that future DRL frameworks should explicitly account for actuation heterogeneity rather than assuming unified control architectures.

6.2. Future Research Directions

Based on a systematic review and analysis of the seven research questions mentioned above, DRL still has broad prospects for future research in the field of multi-legged robot motion control. First, with the continuous expansion of algorithm complexity and application scenarios, the deployment of DRL policies on real robots remains constrained by efficiency, robustness, and transferability, which are closely related to the simulation-to-real-world (sim-to-real) gap. Future research can further explore how to achieve rapid adaptation and generalization across tasks, terrains, and different robot forms through meta-reinforcement learning, residual learning, multimodal perception fusion, and hybrid control methods, which have been shown in prior studies to be effective for improving sim-to-real transfer performance, while also considering energy efficiency and safety constraints.

Second, training efficiency and sample utilization remain important factors restricting the application of DRL on physical robots. Through few-shot learning, curriculum learning, simulation–reality joint training, and multi-robot collaborative learning, the robustness and transferability of policies can be improved while shortening the training cycle. These strategies are particularly relevant for reducing real-world data collection costs and mitigating performance degradation during real-hardware deployment.

Third, the gap between simulation and reality remains one of the most critical challenges for DRL-based legged robot control. Future research needs to explore more efficient strategies such as domain randomization, dynamic model simplification, online fine-tuning, and continuous learning to ensure reliable policy transfer under real-world uncertainties, including unmodeled contact dynamics, actuator delays, and sensor noise. Meanwhile, multimodal sensor fusion, navigation in complex terrain, energy optimization, and control under actuator constraints remain key research areas, as these factors directly affect the robustness and safety of sim-to-real deployment in highly dynamic or unstructured environments.

Finally, establishing standardized datasets, task benchmarks, and evaluation metrics will facilitate objective comparisons of different algorithms and support more reproducible and transparent evaluation of sim-to-real transfer performance. Overall, future research should focus on algorithm efficiency, policy generalization ability, simulation-to-real-world transfer, and robust control in complex environments to further enhance the autonomous motion capabilities of multi-legged robots in real-world scenarios.