Reinforcement Learning for Bipedal Jumping: Integrating Actuator Limits and Coupled Tendon Dynamics

Abstract

In high-dynamic bipedal locomotion control, robotic systems are often constrained by motor torque limitations, particularly during explosive tasks such as jumping. One of the key challenges in reinforcement learning lies in bridging the sim-to-real gap, which mainly stems from both inaccuracies in simulation models and the limitations of motor torque output, ultimately leading to the failure of deploying learned policies in real-world systems. Traditional RL methods usually focus on peak torque limits but ignore that motor torque changes with speed. By only limiting peak torque, they prevent the torque from adjusting dynamically based on velocity, which can reduce the system’s efficiency and performance in high-speed tasks. To address these issues, this paper proposes a reinforcement learning jump-control framework tailored for tendon-driven bipedal robots, which integrates dynamic torque boundary constraints and torque error-compensation modeling. First, we developed a torque transmission coefficient model based on the tendon-driven mechanism, taking into account tendon elasticity and motor-control errors, which significantly improves the modeling accuracy. Building on this, we derived a dynamic joint torque limit that adapts to joint velocity, and designed a torque-aware reward function within the reinforcement learning environment, aimed at encouraging the policy to implicitly learn and comply with physical constraints during training, effectively bridging the gap between simulation and real-world performance. Hardware experimental results demonstrate that the proposed method effectively satisfies actuator safety limits while achieving more efficient and stable jumping behavior. This work provides a general and scalable modeling and control framework for learning high-dynamic bipedal motion under complex physical constraints.

1. Introduction

In recent years, reinforcement learning (RL) has achieved remarkable success in high-dynamic motion-control tasks for robots, particularly demonstrating excellent policy-optimization capabilities in complex actions such as bipedal walking, running, and jumping [1,2]. However, despite their strong performance in simulation environments, these methods still face significant challenges when deployed on real hardware [3,4]. The core issue lies in the persistent “sim-to-real” gap, which manifests primarily in two aspects. First, the physical limitations of real-world actuators, including peak torque and instantaneous power constraints, are frequently oversimplified or neglected during training. Many RL-based approaches still rely on static and uniform torque limits, failing to account for the speed-dependent characteristics of power, thereby increasing the risk of motor overload or failure during actual execution [3,5]. Second, complex nonlinear features such as tendon-driven transmission, joint coupling, and elastic deformations are rarely modeled explicitly, resulting in control policies that fail to generalize to real-world systems [6,7].

To address the aforementioned challenges, numerous studies have proposed solutions from different perspectives. For instance, Rajeswaran et al. [8] introduced a residual RL framework based on expert demonstrations to construct more stable and efficient policies. Peng et al. [9] enhanced policy robustness by applying large-scale domain randomization. Yu et al. [10] and Xie et al. [11] improved simulation fidelity through system identification and online model adaptation, respectively. Haarnoja et al. [12] proposed the Soft Actor–Critic (SAC) algorithm, which achieved impressive performance in terms of sample efficiency and policy stability. However, these methods commonly overlook a critical issue: the lack of systematic modeling of actuator physical boundaries and complex structural characteristics. This omission leads to policies that are unaware of physical feasibility constraints during training, making them prone to generating infeasible actions in high-power, non-periodic tasks such as jumping.

Specifically, in the context of jumping control, several studies have attempted to develop high-performance policies within RL frameworks. For example, Bellegarda et al. [13] incorporated a power-clipping mechanism during execution to enhance real-world deployability. Zhang et al. [14] proposed a goal-guided and phase-based training framework to improve convergence efficiency, but their reward function failed to guide the policy to recognize actuator limits during training, which may result in unsafe actions during deployment. Atanassov et al. [15] utilized curriculum learning to achieve robust jumping behaviors. However, these approaches generally lack an integration of dynamic torque boundaries into the training process, limiting the policy’s ability to perceive and respond to motor states in real time. In addition, methods such as the PSTO framework proposed by Zhu et al. [16] and the DDPG-based gait optimization for CyberDog by Yan et al. [17], although effective in improving energy efficiency for low-dynamic tasks, rely heavily on periodic and stable motion patterns and thus struggle to generalize to explosive, high-power tasks like jumping. More importantly, many existing studies use reward functions that primarily focus on task performance but fail to explicitly model the actuator’s torque or power constraints [18]. As a result, the policies are unable to effectively learn and adapt to the actual physical boundaries of the actuators during training, significantly limiting their robustness and safety in real-world deployment [19]. In summary, while existing methods have alleviated parts of the sim-to-real gap to a certain extent, most have not systematically incorporated actuator physical limits and structural characteristics into the policy-learning process, making them inadequate for deployment in high-power tasks involving complex drive systems.

To address these limitations, this paper presents a physically consistent RL framework for tendon-driven bipedal robot jumping control. We introduce a linear transmission mapping from joint space to motor space to accurately model torque coupling caused by tendon-driven mechanisms. On this basis, we derive a speed-dependent dynamic joint torque boundary that reflects real actuator torque constraints and integrate it into the reward function to guide the policy toward physically feasible actions during training. To further narrow the sim-to-real gap, the model also incorporates tendon elasticity deviation and motor tracking errors to better approximate real-world conditions. The overall control framework proposed in this study is illustrated in Figure 1. The main contributions are summarized as follows:

• Torque Error-Compensation Modeling: By incorporating the effects of tendon elasticity and motor actuation delay, a dynamically adjustable joint torque safety boundary is derived as a function of joint velocity. This significantly enhances the simulation model’s fidelity and stability in approximating the real-world system;
• Torque Transmission Modeling: A linear mapping model from joint space to motor space is constructed to account for the characteristics of tendon-driven bipedal robots. By defining the pulley radii of the hip, knee, and ankle joints, the model systematically captures the torque coupling effects introduced by the tendon-driven mechanism, laying a solid foundation for high-precision control;
• Physically Consistent RL Framework: In the RL training process, this study integrates a dynamic torque boundary constraint into the reward function, guiding the policy to learn and comply with velocity-dependent torque limits. This approach allows the policy to implicitly adjust its outputs, ensuring efficient jumping while respecting the physical constraints of the actuators.

All experiments were conducted on our self-developed experimental platform. The results show that the proposed method generates stable and feasible control policies within actuator safety limits, effectively optimizing jumping performance and significantly enhancing the deployability and control quality of the policy. This work provides a theoretically sound and practically viable framework for applying RL to high-dynamic robotic tasks, laying a solid foundation for future policy design that integrates physical consistency.

2. Materials and Methods

2.1. Mechanism Description for Robot

2.1.1. Overall Structural Design of the Robot

The bipedal robot platform used is illustrated in Figure 2. The robot stands 1.6 m tall, weighs approximately 28 kg, and has a total of 28 degrees of freedom (DOFs), 10 of which are allocated to the legs. Each leg has 5 DOFs, including 3 at the hip (pitch, abduction/adduction, and rotation), 1 at the knee (pitch), and 1 at the ankle (pitch). To enhance mobility and significantly reduce the load and inertia of distal limbs, tendon-driven mechanisms are employed at the robot’s key joints. This actuation method offers distinct advantages, including lightweight transmission, superior force control, and fast dynamic response. Owing to these features, tendon-driven mechanisms have been widely applied in robotics and bio-inspired manipulators. In scenarios where low mass, high compliance, and a high torque-to-weight ratio are essential, a tendon-driven mechanism enables agile, safe, and efficient motion control [20,21]. High-torque DC motors combined with tendon-driven systems are centrally mounted near the center of mass (COM) on both sides, and their output is transmitted to the knee and ankle joints via lightweight high-strength cables. This design reduces the structural mass of a single leg to below 3.6 kg.

The tendon-driven mechanism offers advantages such as compact structure, fast response, and high transmission efficiency, making it a critical foundation for achieving highly dynamic motions such as jumping [22]. The entire tendon-driven mechanism uses 1.5 mm diameter high-strength cables as the transmission medium, offering excellent tensile strength and fatigue resistance. From a mechanical perspective, a bi-directional winding configuration is employed along with a built-in tension adjustment mechanism, which effectively prevents cable slack and backlash. This ensures stable torque transmission and enables low-backlash joint control performance. The entire control system runs on a Linux-based real-time platform, with each joint equipped with an absolute encoder and operating under a unified position control mode. This setup ensures both precise joint control and robust real-time performance under complex task conditions.

2.1.2. Two-Stage Rope Drive System for the Knee Joint

The knee joint is actuated using a two-stage tendon-driven transmission with a remotely mounted motor, as shown in Figure 2. The knee motor is positioned at the upper thigh, and its torque is transmitted via a cable that passes through a guiding pulley and connects to an output pulley mounted on the knee joint. The torque transmission ratio is defined as:

$$k_{\mathrm{knee}}={\displaystyle \frac{r_{\mathrm{j}}^{\mathrm{knee}}}{r_{\mathrm{m}}^{\mathrm{knee}}}}$$

(1)

$${{\tau }_{\mathrm{j}}}^{\mathrm{knee}}=k_{\mathrm{knee}}·{\tau }_{\mathrm{m}}^{\mathrm{knee}}$$

(2)

where $r_{\mathrm{j}}^{\mathrm{knee}}$ and $r_{\mathrm{m}}^{\mathrm{knee}}$ are the radii of the output and input pulleys, respectively, and ${{\tau }_{\mathrm{j}}}^{\mathrm{knee}}$ and ${\tau }_{\mathrm{m}}^{\mathrm{knee}}$ are the corresponding torques of the knee joint and motor. This configuration enables high torque output at the joint while significantly reducing the mass and inertia of the lower leg.

2.1.3. Knee and Ankle Combined Three-Stage Rope Drive System

To further reduce distal limb mass and improve actuation efficiency, this study proposes a novel three-stage coupled tendon-driven mechanism for the knee and ankle joints. Through a cascaded pulley transmission structure, a single motor can simultaneously drive both joints, forming a compact and efficient power transmission path. Specifically, the motor output is first transmitted to the knee-joint output pulley via the first-stage cable. The knee-joint output pulley is then connected to the ankle joint via a second-stage cable. Finally, a third-stage mechanism transfers the torque from the ankle pulley to the foot, completing the three-stage actuation loop. The overall torque transmission ratio of this system is given by:

$$k_{\mathrm{ankle}}={\displaystyle \frac{r_{\mathrm{j}}^{\mathrm{knee}}·r_{\mathrm{j}}^{\mathrm{ankle}}}{r_{\mathrm{m}}^{\mathrm{knee}}·r_{\mathrm{m}}^{\mathrm{ankle}}}}$$

(3)

$${{\tau }_{\mathrm{j}}}^{\mathrm{ankle}}=k_{\mathrm{ankle}}·{\tau }_{\mathrm{m}}^{\mathrm{ankle}}$$

(4)

where $r_{\mathrm{m}}^{\mathrm{ankle}}$ and $r_{\mathrm{j}}^{\mathrm{ankle}}$ denote the radii of the motor pulley and the pulley of the ankle joint, respectively. This three-stage tendon-driven mechanism significantly reduces the load at the distal end by remotely locating the motor, thus improving joint responsiveness and landing compliance. Meanwhile, it features a compact structure, high transmission efficiency, and strong torque amplification capability, making it a key design to enable efficient and stable jumping control.

2.2. Torque Constraint Modeling

In the bipedal robot studied in this study, the leg adopts a hybrid tendon-driven mechanism. The transmission characteristics between joints are influenced by the radii of the pulley and the number of transmission stages. Unlike rigid direct-drive systems, tendon-driven joints transmit torque through multistage pulley systems, resulting in different torque transmission ratios between the motor and each joint. Taking the hip, knee, and ankle joints as examples:

• The hip joint is directly actuated by the motor, with a torque transmission ratio of $k_{\mathrm{hip}}=1$;
• The knee joint is driven via a two-stage tendon-driven mechanism, with a ratio of $k_{\mathrm{knee}}={\displaystyle \frac{r_{\mathrm{j}}^{\mathrm{knee}}}{r_{\mathrm{m}}^{\mathrm{knee}}}}$;
• The ankle joint is driven through a three-stage tendon-driven mechanism, with a ratio of $k_{\mathrm{ankle}}={\displaystyle \frac{r_{\mathrm{j}}^{\mathrm{knee}}·r_{\mathrm{j}}^{\mathrm{ankle}}}{r_{\mathrm{m}}^{\mathrm{knee}}·r_{\mathrm{m}}^{\mathrm{ankle}}}}$;

These ratios $k_i$ must be precisely considered in torque-constrained modeling, as well as in the formulation and solution of torque-constrained dynamics problems.

2.2.1. Leg Modeling Driven by Mixed Tendons

To accurately model the dynamic response characteristics of tendon-driven systems, this work systematically analyzes the torque deviations caused by tendon-driven elasticity and motor-control lag. A corresponding error-modeling framework is developed for both the knee and ankle joints to characterize their motion discrepancies.

(1)

Knee-joint two-stage tendon-driven error modeling.

The knee-joint motor is mounted at the upper thigh and drives the knee-joint output pulley through a single-stage cable transmission, forming a single-input–single-output structure. In this configuration, the elastic deformation of the cable causes a mismatch between the motor output and the actual knee-joint position, resulting in an angular deviation denoted as $\Delta j_{\mathrm{kc}}$. This deviation is converted into torque error through the relationship between deformation and angular displacement. According to the tendon-driven dynamic model, the torque error between the knee-joint motor side and joint side is given by:

$$\begin{array}{cc}\hfill & {{\tau }_{\mathrm{m}}}^{\mathrm{knee}}=k_{\mathrm{p}}\left(m_{\mathrm{kt}}-m_{\mathrm{kc}}\right)-k_{\mathrm{d}}\left({\dot{m}}_{\mathrm{kc}}\right)=K·\mathsf{\Delta }{m}_{\mathrm{kc}}·{r_{\mathrm{m}}^{\mathrm{knee}}}^2\hfill \\ \hfill & {{\tau }_{\mathrm{j}}}^{\mathrm{knee}}=k_{\mathrm{p}}\left(j_{\mathrm{kt}}-j_{\mathrm{kc}}\right)-k_{\mathrm{d}}\left({\dot{j}}_{\mathrm{kc}}\right)=K·\mathsf{\Delta }{j}_{\mathrm{kc}}·r_{\mathrm{m}}^{\mathrm{knee}}·r_{\mathrm{j}}^{\mathrm{knee}}\hfill \end{array}$$

(5)

where $m_{\mathrm{kt}}$ and $j_{\mathrm{kt}}$ represent the target positions of the knee-joint motor and joint, respectively; $m_{\mathrm{kc}}$ and $j_{\mathrm{kc}}$ denote their current positions. $k_{\mathrm{p}}$ is the proportional gain, $k_{\mathrm{d}}$ is the derivative gain. $r_{\mathrm{motor}}$ and $r_{\mathrm{knee}}$ are the radii of the motor pulley and knee-joint pulley, respectively, and K represents the tendon stiffness coefficient, which is 650 N/mm.

(2)

Ankle-joint three-stage tendon error modeling.

The control signal for the ankle joint must pass through a three-stage cascade mechanism, consisting of the knee output pulley and ankle input pulley, to reach the ankle joint. As a result, the total error originates from both the knee transmission segment and the ankle segment itself, forming a multi-stage compounded error system. The resulting torque error can be decomposed into the following two parts:

The torque error caused by elastic deformation between the ankle joint motor and the knee output pulley (denoted as ${\tau }_{\mathrm{mak}}$);

The torque error caused by elastic deformation during transmission from the knee output pulley to the ankle joint (denoted as ${\tau }_{\mathrm{jka}}$).

The total torque error at the ankle joint ${{\tau }_{\mathrm{j}}}^{\mathrm{ankle}}$ is obtained by summing the above two components:

$${\tau }_{\mathrm{mak}}=k_{\mathrm{p}}\left(m_{\mathrm{at}}-m_{\mathrm{kc}}\right)-k_{\mathrm{d}}\left({\dot{m}}_{\mathrm{kc}}\right)=K·\mathsf{\Delta }{j}_{\mathrm{kc}}·r_{\mathrm{m}}^{\mathrm{knee}}·r_{\mathrm{j}}^{\mathrm{knee}}$$

(6)

Because the radii of the knee-joint motor and the ankle-joint motor are the same, we can get:

$${\tau }_{\mathrm{mak}}=k_{\mathrm{p}}\left(m_{\mathrm{at}}-m_{\mathrm{k}\mathrm{c}}\right)-k_{\mathrm{d}}\left({\dot{m}}_{\mathrm{kc}}\right)=K·\mathsf{\Delta }{j}_{\mathrm{kc}}·r_{\mathrm{m}}^{\mathrm{ankle}}·r_{\mathrm{j}}^{\mathrm{knee}}$$

(7)

$${\tau }_{\mathrm{jka}}=k_{\mathrm{p}}\left(j_{\mathrm{at}}-j_{\mathrm{ac}}\right)-k_{\mathrm{d}}\left({\dot{j}}_{\mathrm{ac}}\right)=K·\mathsf{\Delta }{j}_{\mathrm{ac}}·r_{\mathrm{m}}^{\mathrm{ankle}}·r_{\mathrm{j}}^{\mathrm{ankle}}$$

(8)

$${{\tau }_{\mathrm{j}}}^{\mathrm{ankle}}={\tau }_{\mathrm{mak}}+{\tau }_{\mathrm{jka}}=K·r_{\mathrm{j}}^{\mathrm{ankle}}·\left(r_{\mathrm{j}}^{\mathrm{knee}}·\mathsf{\Delta }{j}_{\mathrm{kc}}+r_{\mathrm{j}}^{\mathrm{ankle}}·\mathsf{\Delta }{j}_{\mathrm{ac}}\right)$$

(9)

Here, $k_{\mathrm{p}}$ is the proportional gain, $k_{\mathrm{d}}$ is the derivative gain. $\mathsf{\Delta }{j}_{\mathrm{kc}}$ and $\mathsf{\Delta }{j}_{\mathrm{ac}}$ represent angular position deviations propagated from the knee and ankle segments.

2.2.2. Modeling of Leg-Joint Error Compensation in Hybrid Tendon-Driven Mechanism

In a hybrid tendon-driven mechanism, due to the elastic deformation of cables and motor-control delays in the transmission path, significant discrepancies often arise between the motor target position and the actual joint output. To improve the motion-control accuracy of the knee and ankle joints in bipedal robots, this section analyzes and models the such errors.

The joint position error in a hybrid tendon-driven mechanism mainly consists of two components: the motor execution error and the elastic deformation error of the tendon. As shown in Figure 3, the error of the knee joint, denoted by $e_{\mathrm{k}}$, can be expressed as:

$$e_{\mathrm{knee}}=j_{\mathrm{kt}}-j_{\mathrm{kc}}=e_{\mathrm{knee}1}+e_{\mathrm{knee}2}$$

(10)

Here, $e_{\mathrm{knee}1}$ represents the error caused by tendon elasticity, and $e_{\mathrm{knee}2}$ denotes the deviation between the motor target and its current state. The coefficient ${\displaystyle \frac{1}{3}}$ corresponds to a gear reduction ratio of 3:1 in the knee-joint drive system, used to map the motor-side position to the output side. The expressions are as follows:

$$\begin{array}{cc}\hfill e_{\mathrm{knee}1}& ={\displaystyle \frac{1}{3}}{m}_{\mathrm{kc}}-j_{\mathrm{kc}}\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill e_{\mathrm{knee}2}& ={\displaystyle \frac{1}{3}}(m_{\mathrm{kt}}-m_{\mathrm{kc}})\hfill \end{array}$$

(12)

The error model for the ankle joint is similar to that of the knee. The coefficient ${\displaystyle \frac{1}{2}}$ corresponds to a gear reduction ratio of 2:1 in the ankle-joint drive system, used to map the motor-side position to the output side. The total error $e_{\mathrm{ankle}}$ is expressed as:

$$e_{\mathrm{ankle}}=j_{\mathrm{at}}-j_{\mathrm{ac}}=e_{\mathrm{ankle}1}+e_{\mathrm{ankle}2}$$

(13)

where

$$e_{\mathrm{ankle}1}={\displaystyle \frac{1}{2}}{m}_{\mathrm{ac}}-j_{\mathrm{ac}}+e_{\mathrm{knee}1}$$

(14)

$$e_{\mathrm{ankle}2}={\displaystyle \frac{1}{2}}(m_{\mathrm{at}}-m_{\mathrm{ac}})$$

(15)

This error-modeling approach not only clarifies the structural sources of drive-system errors but also significantly enhances the stability and accuracy of joint control without increasing sensor cost.

2.3. Torque Constraint Modeling

In tendon-driven bipedal robots, actuators are constrained not only by their maximum torque limits during motion but also by their power output capabilities [23]. Compared to regular locomotion tasks, jumping demands rapid energy bursts, imposing stricter requirements on actuator power output [24]. Therefore, explicitly introducing torque constraints into the control formulation improves both the physical realism and stability of RL strategies.

For each joint $i\in \{\mathrm{hip},\mathrm{knee},\mathrm{ankle}\}$, the motor power is defined as [25]:

$$P_{\mathrm{m}}^{\mathrm{i}}={\tau }_{\mathrm{m}}^{\mathrm{i}}·{\dot{\theta }}_{\mathrm{m}}^{\mathrm{i}}$$

(16)

where

• ${\tau }_{\mathrm{m}}^{\mathrm{i}}$ is the motor output torque at joint i;
• ${\dot{\theta }}_{\mathrm{m}}^{\mathrm{i}}$ is the angular velocity of the motor;
• $P_{\mathrm{limit}}^{\mathrm{i}}$ is the maximum allowable power of the motor, as constrained by peak limits.

In a tendon-driven mechanism, motor torque is transmitted to joint torque via tendons and pulleys, yielding the following relation:

$${\tau }_{\mathrm{j}}^{\mathrm{i}}=k_{\mathrm{i}}·{\tau }_{\mathrm{m}}^{\mathrm{i}},{\dot{\theta }}_{\mathrm{m}}^{\mathrm{i}}={\displaystyle \frac{1}{k_{\mathrm{i}}}}·{\dot{q}}_{\mathrm{j}}^{\mathrm{i}}$$

(17)

Here:

• ${\tau }_{\mathrm{j}}^{\mathrm{i}}$: actual output torque at joint i;
• ${\dot{q}}_{\mathrm{j}}^{\mathrm{i}}$: joint velocity;
• $k_{\mathrm{i}}$: torque amplification factor for joint i, determined by pulley radii.

Substituting Equation (17) into the power definition yields the relationship between motor power and joint torque:

$$P_{\mathrm{m}}^{\mathrm{i}}={\displaystyle \frac{{\tau }_{\mathrm{j}}^{\mathrm{i}}·{\dot{q}}_{\mathrm{j}}^{\mathrm{i}}}{k_{\mathrm{i}}^2}}$$

(18)

Thus, under torque constraint, the theoretical upper bound of the joint torque becomes:

$${\tau }_{\mathrm{j}}^{\mathrm{i}}\le {\displaystyle \frac{P_{\mathrm{limit}}^{\mathrm{i}}·k_{\mathrm{i}}^2}{{\dot{q}}_{\mathrm{j}}^{\mathrm{i}}+\epsilon }}$$

(19)

where $\epsilon >0$ is a small positive constant to avoid division by zero. In Equation (19), a small positive constant $\epsilon$ is introduced to prevent division by zero when the joint velocity approaches zero. In this study, $\epsilon$ is set to 0.05 based on empirical observations to ensure numerical stability during policy training. Sensitivity analysis showed that values of $\epsilon$ in the range of 0.01 to 0.1 have minimal influence on torque limit behavior at moderate to high joint velocities, while providing stable bounds during near-zero velocities. This prevents the torque boundary from diverging and ensures consistent policy-learning dynamics across varying joint speeds.

In practice, due to elastic deformation in tendons and control delay in motors, there exists a deviation between target and actual torque. We therefore introduce the total torque error term ($i\in \{\mathrm{knee},\mathrm{ankle}\}$):

$$\delta {\tau }^{\mathrm{i}}=\delta {\tau }_{\mathrm{e}}^{\mathrm{i}}+\delta {\tau }_{\mathrm{m}}^{\mathrm{i}}$$

(20)

The error terms can be computed using the motor encoders and the tension model. Applying Hooke’s law to relate tendon tension with elastic deformation yields [26]:

$$\delta {\tau }_{\mathrm{e}}^{\mathrm{i}}=K·e_{\mathrm{i}1}·{r_{\mathrm{m}}^{\mathrm{i}}}^2$$

(21)

The knee joint adopts a two-stage tendon-driven mechanism. Specifically, a section of cable connects the motor to the knee-joint pulley via a guide wheel, forming a single-input–single-output structure. When the cable is subjected to load, it experiences a slight elongation, resulting in a deviation between the actual output angle of the knee joint and the target angle, denoted as $e_{\mathrm{knee}1}$. This deviation leads to a torque error. The parameter K in Equation (22) denotes the tendon elasticity coefficient, which characterizes the relationship between cable deformation and torque loss. In our formulation, K is determined based on the mechanical specifications provided by the tendon manufacturer (e.g., stiffness per unit length under load). Specifically, we use a value of K = 650 N/mm, which corresponds to the elastic compliance observed in our tendon material under typical loading conditions.

$$\delta {\tau }_{\mathrm{e}}^{\mathrm{knee}}=K·e_{\mathrm{knee}1}·{r_{\mathrm{m}}^{\mathrm{knee}}}^2$$

(22)

The ankle joint is actuated by the ankle motor through a three-stage tendon-driven mechanism, where tendons connect the knee-joint output pulley and the ankle-joint input pulley. In this configuration, two segments of elastic tendons undergo elongation, and the resulting force is ultimately transmitted to the ankle joint.

$$\delta {\tau }_{\mathrm{e}}^{\mathrm{ankle}}=K·(e_{\mathrm{knee}1}·{r_{\mathrm{m}}^{\mathrm{knee}}}^2+e_{\mathrm{ankle}1}·{r_{\mathrm{m}}^{\mathrm{ankle}}}^2)$$

(23)

$$\delta {\tau }_{\mathrm{m}}^{\mathrm{i}}=k_{\mathrm{p}}{e}_{\mathrm{i}2}-k_{\mathrm{d}}{\dot{m}}_{\mathrm{ic}}$$

(24)

The motor error is modeled as a proportion of the target torque. In our tendon-driven platform, the control signal sent to the motor is subject to execution latency and mechanical backlash. Based on the manufacturer specifications and internal calibration tests of our actuator system, we observed a tracking delay and attenuation corresponding to 5–10% deviation from the commanded torque across dynamic motions. Therefore, we introduced a conservative attenuation factor $\lambda$ to approximate the torque-reduction effect. The damping term $\lambda$ in Equation (25), reflecting residual motor-tracking error and dynamic delay, was empirically selected within the range [0.05, 0.1], after multiple hardware tests. Incorporating this factor during training improves the robustness of the learned policy, especially in real-world deployment where ideal motor responses are rarely achieved.

$$\delta {\tau }_{\mathrm{m}}^{\mathrm{i}}={\lambda }_{\mathrm{i}}·{\tau }_{\mathrm{m}}^{\mathrm{i}},{\lambda }_{\mathrm{i}}\in [0.05,0.1]$$

(25)

Through the above modeling, the total error for each joint is obtained as:

Knee joint:

$$\delta {\tau }^{\mathrm{knee}}=K·e_{\mathrm{knee}1}·{r_{\mathrm{m}}^{\mathrm{knee}}}^2+{\lambda }_{\mathrm{knee}}·{\tau }_{\mathrm{m}}^{\mathrm{knee}}$$

(26)

Ankle joint:

$$\delta {\tau }^{\mathrm{ankle}}=K·(e_{\mathrm{knee}1}·{r_{\mathrm{m}}^{\mathrm{knee}}}^2+e_{\mathrm{ankle}1}·{r_{\mathrm{m}}^{\mathrm{ankle}}}^2)+{\lambda }_{\mathrm{ankle}}·{\tau }_{\mathrm{m}}^{\mathrm{ankle}}$$

(27)

Combining the above, we define the maximum safe joint torque under torque constraint as:

$${\tau }_{\mathrm{j}}^{\mathrm{safe},\mathrm{i}}={\displaystyle \frac{P_{\mathrm{limit}}^{\mathrm{i}}·k_{\mathrm{i}}^2}{{\dot{q}}_{\mathrm{j}}^{\mathrm{i}}+\epsilon }}-k_{\mathrm{i}}·\left|\delta {\tau }^{\mathrm{i}}\right|$$

(28)

2.4. Reinforcement Learning Framework

To achieve efficient autonomous control of bipedal robot jumping, this study builds a RL training environment based on the isaacgym platform and proposes a physics-consistent control framework that explicitly incorporates torque boundaries and tendon-driven coupling constraints [27]. Compared to conventional policy-optimization methods that focus solely on trajectory performance, this work systematically integrates two key factors—actuator safety and structural coupling characteristics—into the reward function design, as shown in Figure 4.

2.4.1. Torque Mapping in Tendon-Driven Mechanism

The leg actuation structure adopted in this work employs serial tendon routing to drive multiple joints, resulting in a significant coupling effect between motor output torque and the actual torque experienced at each joint. This coupling effect interferes with torque transmission between joints, often resulting in poor multi-joint coordination. To quantitatively characterize this structural influence, we establish a linear mapping from motor torques to joint-space torques. This formulation effectively improves coordination among joints during the jumping process, ensuring smoother and more stable motion. The mapping is defined as follows:

$$J_{m2j}=\left[\begin{array}{ccc}1.0& 0.0& 0.0\\ -0.384& 0.5& -0.5\\ 0.0& 0.74& 0.74\end{array}\right]$$

(29)

Each row of the matrix represents a linear combination of the torques from the three motors that contributes to the torque of a specific joint (hip, knee, and ankle). This coupling structure maps the desired joint-space torques ${\tau }_{\mathrm{j}}$ to the motor-space torques ${\tau }_{\mathrm{m}}$, meaning that the control of a single joint will simultaneously affect the torque output of other joints. As a result, it increases the complexity of generating an effective control strategy.

2.4.2. Velocity-Dependent Torque Limit Modeling in Joint Space

In this study, although tendon elasticity inherently exhibits nonlinear characteristics, we adopt a linear mapping approach to simplify the coupling effects within the tendon-driven system. This simplification enables more efficient multi-joint control in reinforcement learning. To further address control errors caused by tendon elasticity, we incorporate a dynamic torque boundary model and an error-compensation mechanism, ensuring that nonlinear effects are adequately considered throughout the training process. The proposed dynamic torque boundary model is parameterized based on the motor’s torque-speed characteristics, effectively capturing the motor’s instantaneous output capability under varying rotational velocities. Specifically, the model adaptively adjusts the allowable joint torque according to the joint velocity, thereby dynamically constraining joint outputs. This enhances the controller’s ability to adapt to motor performance variations, particularly under highly dynamic tasks.

In high-power jumping tasks, joint velocities often exhibit rapid and abrupt variations. To prevent motor overload under such conditions, we construct a velocity-dependent dynamic torque limit model in joint space, denoted as ${\tau }_{\mathrm{j}}^{\mathrm{safe},\mathrm{i}}$. This model is parameterized based on the motor’s torque-speed characteristics, capturing its real-time torque capacity at varying speeds. Specifically, the motor’s torque output is influenced by the joint velocity, which in turn dynamically adjusts the allowable torque at each joint. This mechanism enables the system to flexibly accommodate motor performance variations during highly dynamic movements. By modeling the nonlinear coupling between joint velocity, motor power, and torque constraints, the proposed approach significantly enhances the physical feasibility of the learned policy and improves actuator safety.

To make this constraint directly applicable in RL training, we perform an inverse mapping to convert it into motor-space limits:

$${\tau }_{\mathrm{m}}^{\mathrm{safe},\mathrm{i}}={J_{m2j}}^{-1}·{\tau }_{\mathrm{j}}^{\mathrm{safe},\mathrm{i}}$$

(30)

Subsequently, after the policy network outputs the actions (i.e., target motor torques), a clipping operation is applied to constrain the actions within the aforementioned limits, thereby ensuring the physical feasibility of the policy output.

2.4.3. Reward Function Design

In this study, torque constraints and physical consistency principles are seamlessly integrated into the design of the RL reward function. The overall reward is formulated as a weighted sum of multiple sub-terms, covering aspects such as task accomplishment, posture and motion stability, energy consumption, and actuator safety [28]. Building upon classic control rewards—such as linear and angular velocity tracking, posture stabilization, and foot coordination—we introduce several torque-related terms that penalize actions exceeding dynamic torque limits. These additions significantly enhance the deployability of the learned policy on real-world robotic systems.

The weights in the reward function (as shown in Figure 4) were manually selected through iterative trial-and-error experiments. Specifically, the term base-height (with a weight of −20.0) directly incentivizes jumping height, while other components such as torques ($-1\times {10}^{-5}$), dof_pos_limit (−10.0), and feet_stance/feet_swing (−2.0 each) enforce actuator safety and proper contact patterns. The weighting scheme was manually tuned to ensure that the agent prioritizes achieving sufficient vertical lift while respecting the physical feasibility of motion and contact dynamics. Larger penalties are assigned to violating joint limits and undesired contact states to avoid physically unrealistic behaviors. To evaluate the effectiveness of the reward structure in balancing task objectives and physical constraints, we performed ablation studies by removing key penalty terms. When the torques term was removed, the learned policy frequently generated torque commands exceeding actuator safety limits, risking hardware damage. Omitting the dof_pos_limit term led to joint overextension and unrealistic postures. Additionally, removing feet_stance and feet_swing terms caused asymmetric landings and unstable COM trajectories. In contrast, the full reward structure yielded stable and symmetric jumping with compliant joint behavior, demonstrating a well-balanced optimization between performance and physical feasibility.

When the output torque of the policy exceeds the soft power limit, the torque reward function applies a linear penalty, encouraging the policy to remain within the safe operating range while still meeting task requirements, as shown in Equation (31).

$${\tau }_{\mathrm{torque}\_\mathrm{limits}}=\sum _{i=1}^{n}max\left(|{\tau }_i|-{\tau }_{\mathrm{m}}^{\mathrm{safe},\mathrm{i}}·\lambda ,0\right),\lambda =0.95$$

(31)

To facilitate reproducibility, we summarize key training configurations as follows. The policy and value networks adopt a 3-layer MLP structure with hidden layer sizes [256, 256, 128] and ReLU activation. We use the Proximal Policy Optimization (PPO) algorithm implemented with the Adam optimizer. The learning rate is set to $3\times {10}^{-4}$ for both actor and critic networks, with a batch size of 64 and a discount factor $\gamma =0.99$. Each policy is trained for 3000 iterations with 4096 environment steps per iteration, totaling approximately 12 million steps. Training converges within 1.5 million steps, as monitored by reward plateaus and stable policy entropy. Model training and evaluation were conducted using Isaac Gym with GPU acceleration.

This work is the first to systematically incorporate tendon-driven mechanisms and torque-bound dynamic coupling modeling into a reinforcement learning (RL) framework. The proposed method integrates these effects into the reward function in a fully differentiable manner. By explicitly accounting for torque errors and coupling effects during training, the learned policy eliminates the need for online adaptation during deployment. Through dynamic adjustment within the RL process, the model can effectively accommodate actuator limitations in real-world tasks, thereby ensuring both system stability and control efficiency. This approach not only significantly improves the physical feasibility of the learned jumping policy and enhances actuator safety, but also provides a generalizable solution for controlling robots with complex actuation mechanisms.

3. Results

To systematically verify the effectiveness and practicality of the proposed RL jump-control framework, this paper focuses on two core innovations: tendon-driven coupling structure modeling and dynamic power boundary control. These innovations were tested on a self-developed physical robot, and key experiments were designed. Each experiment specifically evaluates the impact of different control elements on the strategy’s performance and physical feasibility, thus forming a closed-loop from theoretical hypothesis to practical effect validation.

3.1. Torque Coupling Modeling Impact

In this experiment, we aim to investigate whether incorporating torque errors induced by the tendon-driven structure and modeling the motor–joint coupling can significantly enhance multi-joint coordination and jumping stability in RL. Specifically, the coupling effects between tendons, motors, and joints play a critical role in controlling high-dynamic tasks such as jumping in bipedal robots. Without adequate modeling of these interactions, joint-level control imbalances may arise, impairing the efficiency and stability of jumping behaviors. To this end, we design a comparative study between two control strategies: (1) direct joint torque control without modeling tendon-induced torque errors and motor–joint coupling; and (2) a control strategy that explicitly models both the torque errors and the coupling relationships during learning. Both strategies adopt the proposed velocity-dependent dynamic torque limiting. We evaluate performance across several key metrics, including joint torque variation, joint position trajectories, joint velocities, jump symmetry, foot height, and COM height. These indicators comprehensively reflect differences in jumping performance and physical feasibility across control strategies.

Experimental results are presented in Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9, where the blue solid lines represent the performance of the strategy with coupling and torque error modeling, while the orange dashed lines correspond to the baseline without such modeling. The comparison reveals that incorporating physical coupling modeling significantly improves control performance. The proposed approach enables more accurate joint trajectory tracking, smoother velocity profiles, reduced torque overshoot, enhanced postural stability, and consistently higher jump heights. These findings strongly validate the effectiveness of embedding low-level physical modeling into the RL framework, providing a robust foundation for the stable and practical deployment of control policies in complex, high-dynamic robotic tasks.

3.2. Dynamic Torque Limit vs. Static Limit

To verify whether the proposed velocity-dependent dynamic power limit strategy offers better actuator protection compared to traditional static constraints while maintaining or even enhancing jumping performance. This study designs a comparative experiment—dynamic torque limit vs. static limit. Three control strategies are tested: (1) Static limit, which imposes fixed torque bounds on joint outputs; and (2) ours, which constructs a dynamic torque bound based on joint velocity and maps it to motor space for action clipping. Evaluation metrics include the proportion of torque outputs exceeding safe bounds and the maximum instantaneous joint power during jumping. This experiment aims to assess whether the dynamic boundary can adaptively adjust with respect to joint speed, thereby ensuring actuator safety without compromising—if not improving—the control performance and stability of dynamic jumping.

The proposed RL control strategy, based on dynamic torque boundary constraints and error-compensation modeling, demonstrates superior performance in comparative experiments. Specifically, the method maintains jump amplitude while effectively regulating joint torque output, thereby reducing the risk of system overload and enhancing both control stability and physical feasibility of the policy. The experimental results are shown in Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14, where the blue solid lines represent the control performance achieved using the proposed method with dynamic torque limiting, while the orange dashed lines correspond to the baseline strategy employing fixed peak torque limits. Experimental results show that, compared to traditional static constraint strategies, the proposed approach achieves better performance in terms of jump height, control smoothness, actuator safety, and postural stability. This provides a reliable solution for deploying RL strategies in high-dynamic tasks.

4. Discussion

This paper presents a RL framework for jumping control that incorporates physical consistency constraints, addressing the challenges posed by actuator torque limitations and structural complexity in tendon-driven bipedal robots performing highly dynamic jumping tasks. By constructing a linear mapping model from joint space to motor space, deriving a dynamic torque boundary expression, and designing a power-aware reward function, the proposed approach significantly enhances the physical feasibility and deployability of learned control policies.

Compared with most existing RL studies, the primary distinction of this work lies in its systematic integration of actuator constraints and structural characteristics into the policy-learning process. In previous works, such as Kumar et al. [29] and Gehring et al. [30], improvements were made in policy generalization and trajectory tracking through fast adaptation modules and optimized control strategies. However, these methods generally did not model actuator torque or power constraints during training, making their policies vulnerable to producing infeasible actions in highly dynamic, non-periodic tasks like jumping. Similarly, while Kim et al. [31] considered power constraints or employed model-guided mechanisms at the policy execution stage to improve stability, most approaches still lacked systematic modeling of actuator limitations during training, leading to control outputs that violate physical feasibility under high-power conditions, thus undermining deployment safety and reliability. In contrast, this work embeds a differentiable motor torque-speed constraint directly into the learning process, enabling the policy to implicitly comply with real-system limitations while optimizing performance, thereby avoiding the need for post hoc clipping or corrections at deployment time.

Moreover, by introducing a torque error-compensation mechanism, the proposed framework significantly improves the fidelity of the simulation environment in approximating real hardware behavior, further mitigating the sim-to-real gap. Hardware experiments demonstrate that the learned policy not only achieves greater jumping height and enhanced stability, but also ensures that the actuators consistently operate within safe power ranges throughout the task, validating the engineering applicability of this approach in high-dynamic scenarios.

While our proposed framework demonstrates promising performance in tendon-driven bipedal jumping, several limitations remain to be addressed. First, the torque transmission model assumes ideal pulley behavior and omits mechanical non-idealities such as friction, hysteresis, and backlash, which may impact performance during prolonged use. Extending the model to include these effects could improve simulation realism and actuator safety. Second, our dynamic torque boundary is primarily defined as a function of joint velocity. However, in practice, actuator limitations are also influenced by factors such as temperature, current saturation, and mechanical wear. Incorporating these variables could enhance long-term robustness. Third, although our framework improves sim-to-real transfer, it has not yet been tested under unpredictable environmental conditions, such as varying terrain or external disturbances. Real-world deployment under such conditions will be a focus of future work. Fourth, our current analysis focuses only on the lower limb joints; integrating full-body dynamics, such as trunk and arm motion, could further improve jump efficiency and stability. Fifth, while the framework is tailored for tendon-driven robots, its modular structure allows for adaptation to other robotic systems by recalibrating the torque mapping and constraint models. Lastly, we acknowledge the absence of quantitative comparisons with prior RL frameworks. Although we qualitatively position our approach relative to existing work (e.g., torque modeling, safety handling), future studies will aim to provide direct metric-based comparisons when compatible baselines are available.

In summary, this work provides a theoretically sound and practically viable solution for deploying RL-based controllers in high-power, structurally complex robotic systems. Although the framework proposed in this study is specifically designed for the jumping task, its foundation in physical consistency and modular structure makes it readily extensible to other high-dynamic behaviors such as running and fall recovery. The dynamic torque boundary model is derived from general actuator power constraints and depends primarily on joint velocity and motor characteristics, which are also critical in other explosive tasks. Similarly, the tendon-driven torque coupling model, while tailored to the current hardware, provides a structured formulation that can be transferred to other tasks where multi-joint coordination is critical. In future work, we plan to validate the applicability of our method to these additional behaviors, further demonstrating its generalizability across dynamic scenarios. For non-tendon-driven systems, such as those using rigid gear transmissions or direct-drive actuators, the torque transmission model must be adapted or simplified to reflect the system’s kinematic and actuation properties. While the tendon-specific coupling structure may no longer be present, similar inter-joint dynamics—due to mechanical constraints or payload interactions—can still be captured through appropriately designed Jacobian-like mappings. The power-aware boundary and reward formulation, however, remains applicable, as actuator safety limits and velocity-dependent torque constraints are fundamental across most robotic systems.

5. Conclusions

This paper addresses the challenges of actuator constraints and structural complexity in high-power bipedal jumping control by proposing a RL framework integrated with physical consistency constraints. The method systematically incorporates joint torque coupling modeling for tendon-driven mechanism and a velocity-dependent dynamic torque boundary mechanism into the policy training process, effectively enhancing the physical feasibility and deployment reliability of the learned control strategy. By establishing a linear mapping from joint space to motor space, the proposed approach accurately captures the torque coupling effects inherent in tendon-driven mechanism, resolving the difficulties of multi-joint coordination. At the same time, a dynamically adjustable torque boundary model based on joint velocity is constructed to reflect actuator power limitations, and is embedded into the reward function. This enables the learning policy to implicitly perceive and respect physical constraints during training. Real-world experimental results demonstrate that the proposed method ensures actuator safety and achieves stable and feasible jumping performance without the need for online adaptation, highlighting its practical deployability. In conclusion, the proposed control framework lays a solid foundation for the practical deployment of RL in high-dynamic bipedal robots and opens up new research directions and technical pathways for control strategy design under complex actuation structures.