Intelligent Gait Synthesis for Autonomous Ground Robots: A Reinforcement Learning Approach

Highlights

What are the main findings?

• A reinforcement learning contrastive optimization framework is proposed that integrates contrastive learning with an asymmetric actor–critic architecture.
• The proposed method enhances policy temporal consistency by aligning historical and future actions, significantly improving control stability.

What are the implications of the main finding?

• The reinforcement learning contrastive optimization framework achieves zero-shot transfer from simulation to real-world environments.
• The robot achieves stable locomotion over complex unstructured real-world terrains such as stairs and slopes while relying solely on proprioceptive sensing.

Abstract

We propose Reinforcement Learning Contrastive Optimization (RLCO), a novel quadruped robot locomotion control framework that synergistically integrates contrastive learning with reinforcement learning. This framework addresses two critical limitations of existing reinforcement learning methods in quadruped motion control: low sample efficiency and insufficient stability in action sequences. To meet the temporal coherence requirements of motion policies in complex environments, we develop a history–prediction action alignment mechanism through contrastive learning. This approach ensures that an action sequence is consistent over time. It does this by reducing the difference between past actions and predicted actions. This approach greatly enhances the stability and reliability of motion control. The proposed co-optimization mechanism preserves reinforcement learning’s exploration capability for complex tasks while improving the physical plausibility and predictability of action sequences. Experimental results demonstrate that our method achieves notable improvements in motion control precision and environmental adaptability in unstructured terrains. Through comparative analysis of different training strategies, we systematically validate the effectiveness of the RLCO framework. Field tests in outdoor environments with stairs, slopes, and grassy terrain confirm the robot’s capabilities. The quadruped robot rapidly adapts to diverse ground conditions.

1. Introduction

Quadruped robots exhibit significant advantages in mobility across unstructured terrains due to their unique bio-inspired characteristics. Quadruped robots have significant advantages over wheeled or tracked robots. They possess independent footholds, enabling them to perform tasks in complex environments such as staircases and uneven or rugged terrains [1,2]. However, traditional control methods for quadruped robots have significant limitations. Their sophisticated algorithms require extensive computational resources and do not adapt well to complex terrain. This results in limited task diversity and poor environmental adaptability [3].

In recent years, reinforcement learning (RL) has achieved groundbreaking advancements in robotic motion control [4,5]. In particular, deep reinforcement learning techniques have endowed quadruped robots with the capability for continuous learning through environmental interactions, enabling them to proactively adapt to novel environmental conditions [6,7,8]. Compared to traditional control methods, deep reinforcement learning-based control strategies significantly enhance the stability and adaptability of agents in complex terrains [9]. Within deep reinforcement learning frameworks, the policy network optimizes its parameters through sustained environmental interactions, thereby learning strategies to maximize cumulative rewards during training. However, existing deep reinforcement learning approaches for quadruped robot locomotion still face significant challenges. These include slow convergence rates, low learning efficiency, and difficulties in real-world deployment. A particular challenge is finding an effective balance between exploring complex environments and maintaining the robot’s body stability [10].

Inspired by the locomotion mechanisms of legged animals, researchers have integrated visual perception with deep reinforcement learning to enhance the mobility of quadruped robots in unstructured terrains. These approaches typically rely on external sensors for environmental perception prior to implementing motion control [11,12]. However, the reliability of external sensors shows significant limitations in complex environments; for instance, cameras and radars struggle to accurately assess terrain physical characteristics under harsh conditions [13,14]. In contrast, proprioceptive sensing methods enable quadruped robots to effectively estimate both surrounding terrain features and self-posture information [15,16]. Nevertheless, reinforcement learning-based motion control relying solely on proprioceptive perception still faces substantial challenges, particularly in achieving continuous and stable control through precise terrain inference requirements.

Although traditional deep reinforcement learning (DRL) methods enhance the robustness of policies to variations in physical parameters, they do not impose explicit constraints on the inherent temporal continuity of action sequences. This can lead to incoherent or high-frequency actions when the robot encounters sudden changes or complex dynamic environments. We propose the RLCO framework to address key limitations in traditional DRL for robotic motion control, such as low training efficiency and deployment difficulties. As illustrated in Figure 1, our framework enhances the PPO-based actor–critic architecture by integrating contrastive learning. To improve feature representation, this approach uses contrastive learning, a powerful unsupervised technique from computer vision and natural language processing [17,18,19]. Its core principle involves maximizing consistency or similarity between data samples to learn discriminative latent representations while minimizing feature redundancy by enforcing each feature to focus on distinct data attributes. Specifically, we compute the covariance matrix of latent representations and apply regularization terms to drive diagonal elements towards 1 and non-diagonal elements towards 0, thereby reducing feature interdependence. Within our RLCO framework, we introduce contrastive loss computation to narrow the discrepancy between historical actions and predicted actions in the actor–critic architecture. This modification effectively mitigates instability in policy learning and accelerates policy updates. The enhancement enables robots to better adapt to diverse locomotion tasks while significantly improving policy generalization capabilities.

Figure 1. The proposed RLCO reinforcement learning-based quadruped robot motion control method integrating contrastive learning. This method effectively guides the policy network to learn control strategies with strong robustness by introducing a contrastive learning loss mechanism.

The main contributions of this study are reflected in three aspects:

• We propose a framework named RLCO that enhances policy consistency during policy optimization through contrastive learning. By minimizing the divergence between historical action policies and future action policies, this framework effectively addresses the control instability caused by policy drift in traditional reinforcement learning.
• We develop a dual advantage function approach to simultaneously maximize rewards and minimize costs. Through the introduction of independent reward advantage functions and cost advantage functions, we achieve dual-objective optimization for both reward maximization and cost reduction.
• We design an asymmetric actor–critic training framework to enhance learning efficiency and policy robustness. The approach successfully achieves zero-shot transfer of control policies to physical robots that were able to complete stability testing across various terrain scenarios in real-world environments.

2. Reinforcement Learning Preliminaries

We design a reinforcement learning framework named RLCO for quadruped robot motion control. RLCO is based on a Markov decision process (MDP) with continuous state and action spaces [20]. Within this framework, the agent learns an optimal control policy $\pi \left(a_t\right|s_t)$ through dynamic interactions with the environment, where $s_t\in S$ represents the state space at time t and $a_t\in A$ denotes the corresponding action space. Specifically, after the agent executes an action in state $s_t$, the environment transitions to a new state $s_{t+1}\in S$ and returns a reward signal $R_t$. This reward signal guides the policy network to iteratively refine its decision-making process.

To achieve precise joint torque control, we employ a neural network-based policy generator that takes the state as input and outputs target joint angle commands. These commands are subsequently converted into actual torque signals through a proportional–derivative (PD) feedback controller, with an action scaling coefficient of 0.25 implemented to balance control accuracy and actuator dynamic range. This integrated approach generates desired torque signals for controlling the joints of a quadruped robot. We employ an asymmetric training framework designed to guide policy learning with privileged information, aiming to achieve zero-shot transfer from a simulated environment to the real world.

In our asymmetric training framework, we leverage privileged information to guide policy learning in order to achieve zero-shot transfer from simulation to the real world. A core information asymmetry exists during the training phase. The critic network acts as a teacher; along with a privileged state encoder, it has access to all available privileged information directly from the simulation. This includes the robot’s linear velocity, foot contact states, and physics randomization parameters (e.g., payload mass, ground friction, coefficient of restitution, motor strength) as well as ground truth terrain and elevation data. This information enables the critic to accurately evaluate the value of the current state, providing high-quality and low-variance gradient signals to the actor network. Conversely, the actor, which represents the student policy to be deployed on the hardware, is strictly limited to non-privileged information. Its inputs rely entirely on onboard sensor data, specifically the angular velocity and gravity vector from the IMU, external control commands, proprioceptive joint data (joint angles and velocities), and the recent action history.

During deployment, we employ a teacher–student distillation strategy to enable the actor network to learn optimal decisions without access to privileged information. We introduce a privileged information loss function $L_{priv}$, which drives the history encoder to mimic the latent embedding generated by the privileged encoder. By minimizing the distance between these two representations, the actor policy learns to infer critical environmental features implicitly from proprioceptive history.

Ultimately, this training strategy allows the actor network to master a robust control policy that can adapt to varying environmental dynamics even with only partial observations. Upon completion of training, this actor policy, which relies solely on non-privileged information, can be directly deployed to the real robot without any additional fine-tuning in order to achieve seamless zero-shot transfer.

3. Methodology

3.1. Observation

The proposed observation framework in our design consists of two components: an observation space and a privileged latent state space. The observation space is primarily defined by data from two sources: the robot’s proprioceptive sensors, and its inertial measurement unit (IMU). This space encompasses a range of specific measurements, including the robot’s body angular velocity, projected gravity vector, command tracking signals, joint positions, and joint velocities, along with the robot’s previous actions. To mitigate the interference of varying physical units on the policy network training, the observational data undergo standardization via the physical normalization layer. Additionally, an action history buffer employs a sliding window mechanism is used to retain the most recent six control commands, which are concatenated along the dim = −1 dimension to construct a joint action–state representation. This approach effectively captures temporal patterns and action–state relationships. The privileged latent state space incorporates both implicit estimations and system-specific characteristics. It comprises several key parameters, including the implicitly estimated linear velocity, foot–ground contact forces, actuator latency, and mass distribution. Furthermore, this space also includes the ground friction and collision restitution coefficients as well as motor characteristics and impedance control parameters. These features collectively enhance the robustness and adaptability of the control strategy by encoding critical environmental interactions and system dynamics that are not directly observable through conventional sensors.

3.2. Reward Design

In terms of action space design, we define a 12-dimensional continuous action vector $a_t$ to represent the desired angles of individual robot joints at time t. To achieve smooth motion control, filtering methods are introduced to process the action sequences.

From the perspective of reward function design, we comprehensively consider multiple dimensions, including robot kinematics, dynamics, and control performance; specific design details are presented in Table 1. Our objective is to guide the agent in maximizing cumulative rewards while accomplishing task goals [21,22]. To this end, appropriate constraint terms are incorporated into the reward function to regulate critical indicators and ensure the naturalness and robustness of its motion behaviors.

Table 1. Reward function design, where $\exp (·)$ represents an exponential function with base e, ${(·)}_{\mathrm{cmd}}$ denotes the speed command value, h indicates the base height, $\tau$ denotes the robot joint torque, and q represents the robot joint angular velocity.

Reward Component	Equation	Weight
Lin velocity tracking	$\exp \left(-\sum {(v_{\mathrm{cmd},i}-v_i)}^2/\sigma \right)$	1.0
Ang velocity tracking	$\exp \left(-{({\omega }_{\mathrm{cmd}}-\omega )}^2/\sigma \right)$	0.6
Linear velocity (z)	$v_z^2$	−1.3
Angular velocity (xy)	$\sum ({\omega }_x^2+{\omega }_y^2)$	−0.06
Orientation	$\sum (g_x^2+g_y^2)$	−0.2
Height	${(h-h_{\mathrm{target}})}^2$	−1.0
Feet Clearance	$\sum h_i^2·v_i$	−0.01
Action rate	$\sum {(a_t-a_{t-1})}^2$	−0.01
Feet Air Time	$\sum (t_i-t_{\mathrm{target}})·c_i$	1.0
Power	$\sum |{\tau }_i·q_i|$	−2 × 10−5

To clarify the terms used in Table 1, the symbols are defined as follows. For motion tracking, $v_{cmd,i}$ and $v_i$ are the commanded and actual linear velocity components of the base, while ${\omega }_{cmd}$ and $\omega$ are the commanded and actual body angular velocities (e.g., yaw). Body stability is maintained by penalizing ${\omega }_x,{\omega }_y$ (roll and pitch angular velocities), $v_z$ (vertical velocity), and $g_x,g_y$ (projected gravity vector components indicating tilt). The terms h and $h_{target}$ represent the actual and target base height. For foot control, $h_i$ and $v_i$ are the height and vertical velocity of the foot, while $t_i$ and $t_{target}$ are its actual and target air time, flagged by $c_i$. Control smoothness is encouraged by comparing $a_t$ and $a_{t-1}$ (the current and previous actions). Finally, to manage efficiency, ${\tau }_i$ and $q_i$ (joint torque and velocity) are used to calculate instantaneous power, and $\sigma$ is a scaling coefficient for the exponential reward functions.

3.3. Domain Randomization

In reinforcement learning, domain randomization is a training paradigm designed to enhance a model’s generalization capabilities. It functions by introducing a wide range of uncertainties into the training environment. Consequently, this method significantly improves the adaptability and robustness of the model when deployed in the real world [23]. For quadruped robot control, we developed a domain randomization method (parameters listed in Table 2) that covers physical parameters, external disturbances, and control parameters across multiple dimensions to simulate locomotion under various terrain conditions. At the physical level, random perturbations were applied to factors such as center-of-mass positions and mass distributions. Regarding external disturbances, random force interventions within [0, 10] N were introduced at 15-s intervals to simulate potential impacts and variations in resistance encountered in real-world environments. On the control aspect, the robustness and generalization performance of the controller were further enhanced through randomization of PID control parameters and motor strengths, along with the incorporation of random delay effects [24].

Table 2. Domain randomization design.

Randomisation Name	Range	Unit
friction	[0.2, 3.1]	-
restitution	[0.1, 1.0]	-
base_mass	[−1.0, 3.0]	kg
com	[−0.1, 0.2]	m
push	[0, 10]	N
motor strength factor	[0.5, 1.2]	-
kp	[0.5, 1.6]	-
kd	[0.5, 1.6]	-

3.4. Curriculum Learning

We propose a dual-curriculum learning framework comprising two core modules: the terrain curriculum and command curriculum. This framework achieves progressive reinforcement of robotic learning through dynamic coupling between terrain complexity progression and motion command optimization [25]. By dynamically balancing terrain feature complexity and motion parameter constraints in three-dimensional spatial dimensions, the system enables agents to gradually evolve from low-speed control on basic terrains to high-speed maneuvering in complex scenarios. Specifically, when the robot successfully masters locomotion under current terrain conditions, the system synchronously escalates terrain complexity while expanding motion parameter ranges. Conversely, if motion performance falls below expectations, it maintains effective learning by either reducing terrain difficulty or constraining command ranges [26]. This bidirectional regulation mechanism significantly enhances training efficiency, allowing agents to progressively develop robust motion control policies in dynamically balanced environments, as shown in Figure 2.

Figure 2. Dual-curriculum learning framework.

In the terrain curriculum design, the system constructs an adaptive terrain difficulty adjustment mechanism through displacement monitoring. Specifically, when the robot’s displacement from the starting point exceeds 90% of the current terrain length, the system increases the terrain complexity level by one grade. To prevent training oscillation, we introduce a hysteresis mechanism in which the difficulty level is reduced only if the displacement falls below a lower threshold (60%). If the displacement falls within the buffer zone (60–90%), the current difficulty level is maintained. When the system detects that the terrain has reached the preset maximum complexity level, it employs a random terrain generation strategy to prevent local optima.

For the motion command curriculum design, we developed a performance-based dynamic parameter adjustment mechanism. The system calculates real-time moving averages of linear velocity tracking rewards and raises the linear velocity command when this metric consistently exceeds 75% of the historical maximum value. This framework incorporates a progressive parameter adjustment strategy to ensure smooth transitions in motion capability development.

3.5. Contrastive Learning

In this work, we integrate a contrastive learning method into our actor–critic framework. Our approach minimizes the discrepancy between a window of historical actions [t − 6, t] and a corresponding window of predicted actions [t − 5, t + 1]. This process effectively enhances the agent’s generalization capabilities and robustness. To achieve this, we designed a hybrid encoder architecture that integrates information from multiple input sources, including historical observations, terrain features, and privileged states. This hybrid encoder consists of three specialized components: a history encoder, a terrain encoder, and a privileged state encoder.

The historical encoder employs a multi-layer perceptron (MLP) architecture to extract latent representations from historical observations. Specifically, given a historical observation sequence $H=\{h_1,h_2,\dots ,h_T\}$, the historical encoder outputs the latent representation $Z_h=ML{P}_{\mathrm{history}}\left(H\right)$. The terrain encoder processes scan information to capture latent representations of environmental layouts. Given scan data S, the terrain encoder produces the latent representation $Z_s=ML{P}_{\mathrm{scan}}\left(S\right)$. The privileged encoder handles the agent’s privileged state information to extract task-relevant latent representations. For privileged state p, the privileged encoder generates the latent representation $z_p=ML{P}_{privacy}\left(p\right)$. The fusion encoder integrates these three latent representations and further optimizes the combined features. During training, the fusion encoder processes $[z_h,z_s,z_p]$. However, during real-world deployment $z_p$ is replaced by the estimated latent ${\widehat{z}}_p$ generated by the history encoder (student), ensuring that the policy operates without access to ground-truth physics parameters.

To optimize the quality of latent representations, we design the contrastive learning loss function detailed in Table 3. This loss is computed from a covariance matrix $C=(1/B)Z^{T}Z$, where B denotes the batch size and Z represents the latent representation matrix. In the table’s formulas, $\Pi (i,j)$ represents the element in the i row and j column of this covariance matrix C. The loss function consists of three components: (1) a diagonal loss ($L_{diag}$) to ensure sufficient informativeness of individual features; (2) an off-diagonal loss ($L_{off\_diag}$) to minimize redundancy among features; and (3) a privileged information loss ($L_{priv}$) to align the latent representations of the student and teacher encoders. The total loss function L weights these components; the hyperparameter $\lambda$ was set to $5\times {10}^{-3}$ in our experiments. In specific applications, defining the historical action sequence $H_{\mathrm{hist}}=\{a_{t-6},\dots ,a_t\}$ and the future action sequence $H_{\mathrm{future}}=\{a_{t-5},\dots ,a_{t+1}\}$ allows the encoder E to map them to the latent space, obtaining representations $z_{\mathrm{hist}}=E\left(H_{\mathrm{hist}}\right)$ and $z_{\mathrm{future}}=E\left(H_{\mathrm{future}}\right)$. The contrastive loss function can be expressed as $L_{\mathrm{contrastive}}=D(z_{\mathrm{hist}},z_{\mathrm{future}})$, where D is a distance metric. This contrastive loss $L_{\mathrm{contrastive}}$ is added as a regularization term to the total loss function, and its gradients are backpropagated to both the policy network $\pi$ and the encoder E. For the policy network, in order to minimize $L_{\mathrm{contrastive}}$, the action $a_{t+1}$ generated at time $t+1$ must ensure that the encoding $z_{\mathrm{future}}$ of the entire $H_{\mathrm{future}}$ sequence is close to $z_{\mathrm{hist}}$. This implies that the output distribution of the policy $\pi (·|s_{t+1})$ is not only influenced by the current state $s_{t+1}$ but also implicitly constrained by the historical action sequence $H_{\mathrm{hist}}$. This constraining effect penalizes decisions that cause large jumps in the representation vectors within the latent space, forcing the policy to generate smoother and temporally continuous action sequences. This effectively suppresses high-frequency oscillations in control commands, as detailed in Figure 3.

Table 3. Comparison of loss function formulas.

Loss	Equation
Diagonal loss	$L_{diag}=\sum {(\Pi (i,i)-1)}^2$
Off-diagonal loss	$L_{off\_diag}=\sum \Pi {(i,j)}^2$
Priv loss	$L_{priv}=\left|\right|z_{student}-z_{teacher}{\left|\right|}^2$
Temporal loss	$L_t=\parallel z_{hist}-z_{future}\parallel$
Total loss	$L=L_{diag}+\lambda L_{off\_diag}+0.01L_{priv}+L_t$

Figure 3. Architecture of the contrastive loss function.

To further evaluate the feature extraction capability of the RLCO framework, we performed a t-SNE analysis on the 128-dimensional latent vectors generated by the history encoder. As shown in Figure 4, the encoder successfully maps high-dimensional proprioceptive sequences into a structured latent space. Specifically, flat ground and rough terrain form proximal clusters, indicating that the model captures their shared kinematic characteristics. Similarly, stairs and slopes exhibit close proximity due to their related gait requirements. This structured representation validates that our contrastive loss $L_{contrastive}$ effectively organizes the latent space to enhance policy robustness while maintaining necessary sensitivity to terrain transitions.

Figure 4. t-SNE visualization of the latent representations.

Traditional policy optimization methods primarily focus on optimizing reward signals while neglecting cost control. By incorporating both reward and cost signals, proximal policy optimization (PPO) can effectively balance policy stability [27]. In robotic motion control, agents are required not only to maximize rewards but also to manage costs. Although traditional PPO algorithms demonstrate excellent performance in policy optimization, they lack effective mechanisms for processing cost signals. The integration of reward and cost signals enables PPO to control costs while optimizing policies [28]. To balance the optimization of rewards and costs, PPO introduces a dynamically adjusted weighting factor k. This factor is progressively updated according to training iterations, with its update rule defined as $k_{new}=\min (k_d·n^i,1)$, where i denotes the number of training iterations. Furthermore, we compute generalized advantage estimation (GAE) cost advantages based on the cost value function, generating cost advantage signals that provide policy update directions. This approach mitigates the risk of falling into local optima caused by single-objective reward optimization.

We adopted an approximate solution method based on dual advantage functions, treating it as a solution to the constrained Markov decision process (CMDP) problem. This method addresses the cost constraint as a dual problem, evaluating the cumulative cost of constraint violations through an independently defined cost value function $V_c$ and cost advantage function $A_c$. The advantage of this method lies in decoupling the evaluation processes of reward and cost: the reward advantage function $A_{\mathrm{reward}}$ focuses on the main task objective, while the cost advantage function $A_{\mathrm{cost}}$ concentrates on safety constraints. This architectural separation enhances stability by preventing cost signals from corrupting reward signals, lending greater clarity to the policy update direction and bolstering robustness, especially when objectives conflict. By computing a separate generalized advantage estimation (GAE) for costs, the system generates a distinct cost advantage signal, creating an independent optimization pathway for safety. This dual-gradient guidance is crucial for navigating the complex tradeoffs, steering the agent away from suboptimal local optima such as excessive caution or recklessness and enabling it to converge on a policy that is both highly performant and consistently safety-compliant.

Algorithm 1 summarizes the overall optimization process of PPO, incorporating the calculation of cost advantages.

Algorithm 1 RLCO Policy Optimization with Dual Advantages.

Require:  Policy ${\pi }_{\theta }$, Reward Value function $V_{\varphi }^r$, Cost Value function $V_{\psi }^c$, Encoder ${\mathrm{Encoder}}_{\omega }$ Ensure:  Updated Policy ${\pi }_{\theta }$   1: for iteration = 1, 2, … do   2:     $k_{new}\leftarrow \min (k_d·n^{\mathrm{iteration}},1)$   3:     for $t=1,2,\dots ,T$ do   4:         $z_t\leftarrow {\mathrm{Encoder}}_{\omega }\left(s_t\right)$   5:         $a_t\sim {\pi }_{\theta }\left(a\right|z_t)$   6:         $V_t^r\leftarrow V_{\varphi }^r\left(z_t\right),V_t^c\leftarrow V_{\psi }^c\left(z_t\right)$   7:         Execute $a_t$, receive $r_t$ (reward), $c_t$ (cost), and $s_{t+1}$   8:         Store $(s_t,a_t,r_t,c_t,V_t^r,V_t^c,\log \pi \left(a_t\right|z_t))$ into buffer D   9:     end for 10:     {Calculate Generalized Advantage Estimation (GAE) for both Reward and Cost} 11:     $V_{T+1}^r\leftarrow V_{\varphi }^r\left({\mathrm{Encoder}}_{\omega }\left(s_{T+1}\right)\right),V_{T+1}^c\leftarrow V_{\psi }^c\left({\mathrm{Encoder}}_{\omega }\left(s_{T+1}\right)\right)$ 12:     $A_T^r\leftarrow 0,A_T^c\leftarrow 0$ 13:     for $t=T,T-1,\dots ,1$ do 14:         ${\delta }_t^r\leftarrow r_t+\gamma V_{t+1}^r-V_t^r$ 15:         $A_t^r\leftarrow {\delta }_t^r+\gamma \lambda A_{t+1}^r$ 16:         ${\delta }_t^c\leftarrow c_t+\gamma V_{t+1}^c-V_t^c$ 17:         $A_t^c\leftarrow {\delta }_t^c+\gamma \lambda A_{t+1}^c$ 18:     end for 19:     for $k=1,2,\dots ,K$ do 20:         for each batch B sampled from D do 21:             $z\leftarrow {\mathrm{Encoder}}_{\omega }\left(s\right)$ 22:             $\log {\pi }_{new}\leftarrow \log {\pi }_{\theta }\left(a\right|z)$ 23:             $\mathrm{ratio}\leftarrow exp(log{\pi }_{new}-log{\pi }_{old})$ 24:             $A_{total}\leftarrow A^r-k_{new}·A^c$ 25:             $L_{policy}\leftarrow -min(\mathrm{ratio}·A_{total},\mathrm{clip}(\mathrm{ratio},1-ϵ,1+ϵ)·A_{total})$ 26:             $L_{value}\leftarrow {(V_{\varphi }^r\left(z\right)-(A^r+V^r))}^2+{(V_{\psi }^c\left(z\right)-(A^c+V^c))}^2$ 27:             Compute $z_{hist}$ and $z_{future}$ from sequence windows 28:             $L_{contrastive}\leftarrow L_{diag}+{\lambda }_{off}{L}_{off\_diag}+0.01L_{priv}+D(z_{hist},z_{future})$ 29:             $L_{total}\leftarrow \mathrm{mean}(L_{policy}+c_1{L}_{value}-c_2\mathrm{Entropy}+L_{contrastive})$ 30:             Update $\theta ,\varphi ,\psi ,\omega$ using gradient of $L_{total}$ 31:         end for 32:     end for 33: end for

3.6. Ablation Studies

For comparative evaluation, we designed ablation studies with four configurations:

Baseline: Implements standard PPO algorithm for policy training without domain adaptation mechanisms, serving as the foundational reference group.

CostNet: Builds upon the baseline by incorporating a cost constraint computation module during policy network updates, enabling explicit safety constraint optimization through cost functions [29].

ContrasNet: Integrates a contrastive learning loss function into the actor–critic architecture to enhance policy generalization via feature disentanglement in the latent space.

AsymmetricNet: Designs an actor–critic training framework with asymmetric information to enhance the model’s environmental feature learning capability.

RLCO: A composite framework combining cost-constrained optimization and contrastive representation learning, achieving adaptive policy optimization under safety constraints through dual mechanisms.

4. Training Configuration

The experiments in this study were conducted within the Isaac Gym reinforcement learning environment, as shown in Figure 5. To ensure training efficiency and stability, we adopted a distributed training architecture [30]. The implementation deployed 4096 agents for parallel training, completing a total of 6000 iteration updates. For algorithm implementation, we employed the asymmetric actor–critic framework described in Section 2. Both the actor (policy) network and the critic (value) network were implemented as independent multi-layer perceptron (MLP) networks; they did not share parameters. This separation is crucial for the asymmetric design in which the critic network processes privileged information to guide the actor, which only receives non-privileged onboard sensor data. Both the actor and critic networks contain three fully-connected hidden layers with 512, 256, and 128 neurons, respectively.

Figure 5. In the Isaac Gym simulation environment, the quadruped robot successfully completed a multi-terrain locomotion task. During the initial 0–5 s phase, the robot accomplished stair climbing; in the subsequent 5–20 s phase, the robot smoothly transitioned to flat-ground walking mode.

All computations were executed on a high-performance GPU server equipped with an RTX 4090 graphics card. For policy updates, we implemented the proximal policy optimization (PPO) algorithm with a clip range of 0.2 to balance stability and exploration capability. The Adam optimizer was employed for network parameter optimization, with the learning rate set to $1\times {10}^{-3}$ throughout the training process.

5. Real-World Experimental Setup

Experiments were conducted in real-world environments using the Unitree Go2 robot. We incorporated joint positions and velocities, historical actions, and user commands as observations, deploying a Jetson Nano embedded computer as an edge controller to execute real-time policy inference and motion planning. A PD controller with a proportional gain $k_p=30$ and derivative gain $k_d=0.75$ was implemented for joint-level trajectory tracking, ensuring a balanced dynamic response and stability. Note that this policy gains robustness to controller parameters by training on randomized PD gain factors in simulation (see Table 2). This allows the same policy to be directly deployed to the real robot (zero-shot transfer) without concern for the specific fixed gains used by the real-world hardware PD controller. Real-time data interaction among sensors, controllers, and actuators was achieved through the ROS framework.

6. Results

In the ablation study, we conducted a comparative performance analysis between our proposed method and baseline control strategies. For the evaluation illustrated in Figure 5, the robot executed stair-climbing motions in the simulated testing environment during the 0–5 s interval, followed by flat-ground walking from 5 to 20 s. Throughout this process, multiple motion parameters were recorded, including base velocity (X, Y, Z, Yaw), degree-of-freedom (DOF) position, joint velocity, vertical contact forces, torque/velocity curves, and joint torque, as shown in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. We comprehensively compare the performance of each strategy by analyzing the motion parameters under different control strategies, with the detailed results summarized in Table 4. The X, Y, Z, and yaw velocity are recorded in order to measure the time required to regain stability during the transition from stair climbing to flat-ground walking. The DOF position, joint velocity, contact forces, torque/velocity, and joint torque are recorded to capture the fluctuation range during the motion of the quadruped robot.

Figure 6. Baseline quadruped robot control strategy.

Figure 7. CostNet quadruped robot motion control strategy.

Figure 8. ContrasNet quadruped robot motion control strategy.

Figure 9. AsymmetricNet quadruped robot motion control strategy.

Figure 10. RLCO quadruped robot motion control strategy.

Table 4. Performance comparison of different motion control strategies.

Parameter	Baseline	CostNet	ContrasNet	AsymmetricNet	RLCO
Velocity X	6 s	6 s	5 s	6 s	6 s
Velocity Y	6 s	5 s	5 s	2.5 s	2.5 s
Velocity Z	6.5 s	5 s	5 s	4.5 s	4.5 s
Velocity Yaw	6 s	5.5 s	2.5 s	5.5 s	5.5 s
DOF Position	[−0.5, 2.5] rad	[−0.5, 2.0] rad	[0, 2.5] rad	[0.25, 2] rad	[0.25, 2] rad
Joint Velocity	[−30, 15] rad/s	[−25, 15] rad/s	[−15, 20] rad/s	[−15, 15] rad/s	[−15, 15] rad/s
Contact Forces	$peak$∼300 N	$peak$∼175 N	$peak$∼175 N	$peak$∼175 N	$peak$∼175 N
Torque/Velocity	$scatter$	$scatter$	$cluster$	$cluster$	$cluster$
Joint Torque	[−5, 10] Nm	[−10, 10] Nm	[−10, 5] Nm	[−5, 5] Nm	[−5, 5] Nm

Based on the ablation study results, we conducted a quantitative evaluation to analyze the experimental results from three perspectives: overall performance, training efficiency, and sensitivity to key hyperparameters.

• Analysis of Overall Performance and Training Efficiency

We first compared the performance of our proposed RLCO strategy against a series of baseline models and two ablation models, CostNet and ContrasNet. As illustrated in Figure 11, the RLCO strategy demonstrated superior performance among all comparative methods. Its learning curve exhibited a robust upward trend throughout the training process. In particular, the curve slope was steepest within the interval of 0 to 2500 training steps, indicating the highest learning efficiency, during which the mean reward increased rapidly to 10. Ultimately, the strategy converged to a maximum mean reward of approximately 14, significantly outperforming the other models. Furthermore, as shown in Figure 12, the RLCO strategy also excelled in terms of mean episode length. Its values surpassed those of the baseline and ablation models from the early stages of training and eventually stabilized, converging to 1000 steps. In contrast, the baseline model experienced substantial fluctuations between 2000 and 3500 steps, reflecting the instability of its strategy during environmental exploration. In contrast, RLCO and its ablation variants all maintained a steady upward trajectory, validating the positive contribution of our introduced modules to improving the robustness of the policy.

Figure 11. Comparative analysis of the impact of window size and model ablation on the mean reward value of the RLCO policy.

Figure 12. Comparative analysis of the impact of window size and model ablation on the mean episode length of the RLCO policy.

• Sensitivity Analysis of the Time Window Hyperparameter

To validate the rationale behind selecting the time windows [t − 6, t] and [t − 5, t + 1] for contrastive learning, we conducted additional tests with two alternative configurations: a shorter window of [t − 3, t] and [t − 2, t + 1] (denoted as RLCO_t − 3), and a longer window of [t − 9, t] and [t − 8, t + 1] (denoted as RLCO_t − 9). The RLCO_t − 3 model using the shorter window achieved a final mean reward that converged to only 10, with a mean episode length of approximately 910. Both metrics are significantly lower than the performance of the standard RLCO (mean reward of 14, episode length of 990). This demonstrates that an overly short history fails to provide the policy network with sufficient temporal context of the motion, hindering its ability to effectively suppress action oscillations and predict future trends while reducing motion stability. Conversely, the RLCO_t − 9 model’s mean reward converged to 8, far below the 14 achieved by RLCO, and its mean episode length was only 900, also substantially lower than RLCO’s 990. This decline in performance indicates that simply extending the history horizon yields diminishing returns. A distinctively long history window incorporates states from the distant past that have low correlation with the robot’s immediate future dynamics. These irrelevant data points act as noise within the input space, reducing the signal-to-noise ratio of the latent representation. Consequently, the encoder struggles to disentangle critical motion cues from outdated information, which hinders the policy’s ability to rapidly adapt to sudden environmental changes.

7. Discussion

We propose a novel locomotion control framework named RLCO that enables quadruped robots to effectively navigate complex unstructured terrain while relying solely on proprioceptive sensing through reinforcement learning. Comprehensive experimental validation was conducted through field tests on the Unitree Go2 robotic platform. We thoroughly evaluated the practical performance of our RLCO framework by designing diverse testing scenarios, including typical unstructured terrain challenges such as stair ascent and descent, slope negotiation, and soft grassland traversal, as illustrated in Figure 13. All experiments were completed in real-world environments, with complete testing procedures documented in archived video recordings (video link: https://www.youtube.com/playlist?list=PLFJjnb32o0n9yDMpd1Xt_kVLXfChyBZHD, accessed on 21 October 2025). Experimental results validate the effectiveness of the RLCO framework in enhancing stable operation capabilities for quadruped robots in complex dynamic environments, achieving successful zero-shot transfer in real-world applications without additional fine-tuning. Future research directions in this field will focus on several key areas. First, we aim to expand task diversity by applying the RLCO framework to more challenging dynamic tasks such as high-altitude jumping, complex loco-manipulation, and adaptive locomotion on highly deformable terrains, which will allow us to further explore the performance boundaries of the algorithm. Second, we will focus on online adaptation and continual learning, specifically investigating methods that enable robots to dynamically fine-tune their policies based on real-time physical feedback (e.g., payload variations or foot-end wear) during deployment. This will further enhance the robustness and stability of the system in diverse and unpredictable real-world environments.

Figure 13. Testing of the RLCO quadruped robot motion control strategy in real-world environments.