Energy Consumption Minimization of Quadruped Robot Based on Reinforcement Learning of DDPG Algorithm

Abstract

Energy consumption is one of the most critical factors in determining the kinematic performance of quadruped robots. However, existing research methods often encounter challenges in quickly and efficiently reducing the energy consumption associated with quadrupedal robotic locomotion. In this paper, the deep deterministic policy gradient (DDPG) algorithm was used to optimize the energy consumption of the Cyber Dog quadruped robot. Firstly, the kinematic and energy consumption models of the robot were established. Secondly, energy consumption was optimized by reinforcement learning using the DDPG algorithm. The optimized plantar trajectory was then compared with two common plantar trajectories in simulation experiments, with the same period and the number of synchronizations but varying velocities. Lastly, real experiments were conducted using a prototype machine to validate the simulation data. The analysis results show that, under the same conditions, the proposed method can reduce energy consumption by 7~9% compared with the existing optimal trajectory methods.

1. Introduction

Quadruped robots, due to their excellent working ability in complex terrain and dangerous environments [1], have been widely used in rescue operations [2], military contexts [3], exploration missions [4], and mining activities [5], offering significant advantages such as low cost and high completion rates. However, the rapid depletion of energy reserves during locomotion remains a critical issue, resulting in short operating time. Reducing the motion energy consumption without losing the motion performance has become a key challenge to achieving longer operational duration and high-performance robot service.

Various methods, including gait planning [6,7,8], trajectory planning [9,10], and reinforcement learning [11,12,13,14,15], have been used to enhance the control of four-legged robots. Wang et al. proposed a soft gait control strategy that minimized tracking error and jitter to improve the quadrupedal crawling robot’s stable passage in a sidehill environment [6]. Li et al. proposed a gait-switching control method for quadrupedal robots based on the combination of dynamic and static and improved the robot’s ability to adapt to different terrains using Matlab–Adams co-simulation [7]. Chen et al. proposed a quadrupedal robot that was able to adapt to multiple terrains based on the idea of trajectory planning by combining the Cartesian space with the joint space [8]. Chen et al. proposed a quadrupedal robot that was able to adapt to multiple terrains based on the idea of trajectory planning by combining the Cartesian space with the joint space’s bionic foot trajectory [9]. Levin et al. reported on learning hand–eye coordination for robotic grasping through deep learning and big data [11]. Haarnoja et al. proposed a sample-efficient deep reinforcement learning (RL) algorithm based on maximum entropy artificial intelligence for real robots to learn to walk [12]. Tan et al. utilized deep reinforcement learning techniques to automate quadrupedal robots’ agile locomotion [13]. Nagabandi A et al. conducted a study on learning online adaptation in the context of model-based reinforcement learning to enable robots to learn to adapt to dynamic real-world environments through reinforcement learning [15]. Notably, the robot’s autonomous movement, recognition of the terrain to plan its path, etc., were realized by reinforcement learning. This shows that reinforcement learning methods have very high potential value.

The current conventional methods for studying robot energy consumption are as follows. Yang et al. proposed a trotting gait foot trajectory based on cubic spline interpolation and investigated the variation in energy consumption due to different gait parameters of the quadrupedal robot SCalf by simulation [16]. Benotsmane et al. found that an industrial robot executed three defined reference trajectories with reduced accuracy using an MPC controller, but was always faster and required less energy [17]. G. Wang et al. proposed a quadratic planning force-allocation controller to reduce the energy consumption of a hexapod robot by optimizing the instantaneous power at each time step [18]. Mikolajczyk et al. investigated the current state of the art of bipedal walking robots that use natural bipedal locomotion (humans and birds) as well as innovative synthetic solutions and probed the energy efficiency of bipedal robots [19]. These methods have made good progress in the control of legged robots. The feasibility of the traditional [19,20,21,22,23] approach can be directly observed since it usually uses different platforms for simulation, planning, and control, and the obtained data are imported into the robot control system for experiments. However, repeated attempts are inefficient and often accompanied by problems such as damage to the body, slow progress, and unsatisfactory results. Therefore, it is very difficult to consider the energy consumption with the traditional approach, and reinforcement learning can play an important role in this regard.

This paper focuses on the energy consumption of the legs of the Cyber Dog quadruped robot during locomotion. To optimize the energy consumption, reinforcement learning was performed using the deep deterministic policy gradient (DDPG) algorithm, the optimal energy consumption data were extracted in a simulated environment, and then the energy consumption of the robot was compared with that of two common foot end trajectories [9] under the same experimental conditions. The power output of each joint of the robot’s leg was reduced while maintaining the speed of movement, thus optimizing energy consumption under the same conditions. The paper is structured as follows. Section 2 models the robot’s kinematics and energy consumption and analyzes two common end-of-foot trajectories. Section 3 describes the experimental methodology and setup to lay the research foundation for subsequent experiments. Section 4 compares the optimal end-of-foot trajectories obtained through reinforcement learning with the conventional end-of-foot trajectories and compares the energy consumption at the same speed, period, and number of synchronizations through simulation experiments. The reliability of the simulation results was confirmed by prototype experiments. The results obtained from the study are used in Section 5 to evaluate the advantages of the discussed methods. Finally, concluding remarks are made, and future research directions are outlined. The method in this paper can reduce the energy consumption in the movement of quadrupedal robots, extend the endurance of quadrupedal robots in complex and dangerous environments, and reduce the risk of accidental injuries caused by experiments with high-value robots in real environments.

2. Robot Modeling

The Cyber Dog quadrupedal robot is an in-house prototype developed by Xiaomi equipped with three degrees of freedom in each leg and high-performance servo motors. These motors provide a maximum output torque of 32 Nm and a maximum speed of 220 rpm, allowing the robot to move at speeds of 3.2 m/s. The Cyber Dog’s design ensures high-torque and high-speed performance with a fast response. The robot is further equipped with the NVIDIA JETSON XAVIER NX platform, which integrates 384 CUDA cores, 48 Tensor Cores, six Carmel ARM CPUs, and two deep learning acceleration engines for fast interaction between simulation data and real prototypes. The training platform was based on the Matlab simulation platform Simulink, and the 3D diagram of the Cyber Dog quadruped robot was imported into the Simulink simulation platform to establish the intelligent body simulation system. The system meets the simulation requirements, such as joint limitation and ground contact, and has the ability of reinforcement learning.

In order to better train for energy optimization in simulation, the following assumptions were adopted:

The study focused only on the motion of the robot on the plane and ignored the friction between the joints.

The rolling motion of the robot’s hip joints was not considered when walking on a plane.

Considering these preconditions, Matlab configured the external environment parameters for the intelligent body to interact with Simulink. The model contained 111 observations, including the Cartesian coordinates of the torso, the orientation, the joint angles, the 3D velocities, the angular velocities, the joint velocities, and the joint moments. This facilitated a more consistent setup of reward terms in the reward function, consistent with the research described in this paper. For this paper, the dimensions of the simulated robot were aligned with those of the real-world robot to improve the accuracy of the simulation. Furthermore, the fidelity of the physics simulator was enhanced by carefully tuning the parameters of the robots in the simulation, such as the joint rotation ranges and motor control ranges. Figure 1a,b illustrate the real and simulated models.

2.1. Kinematics Modeling

2.1.1. Swing Phase of Leg Mechanism

A body coordinate system (O_w, −x_w, y_w, z_w) was established on the quadruped robot; O_w is the coordinate origin at the center of the torso of the robot model. The leg mechanism can be regarded as a series of connecting rods constituting a connecting rod mechanism in each joint of the leg to establish the connecting rod coordinate system (O_i, −x_i, y_i, z_i) with i = 0, 1, 2, 3, and 4, as shown in Figure 2.

θ₁, θ₂, and θ₃ denote the joint angles of each joint in the leg mechanism; l₁, l₂, and l₃ denote the lengths of each connecting rod of the leg mechanism. The Denavit–Hartenberg (D–H) parameters for when the leg mechanism was in the swing phase are detailed in

Table 1. D–H parameters for the swing phase of the right anterior (RF) leg mechanism.

i	ai−1/mm	αi−1/(°)	θ/(°)	di/mm
1	0	0	θ1	0
2	l1	90	θ2	0
3	l2	0	θ3	0
4	l3	0	0	0

.

Using the homogeneous matrix $T_i^{i-1}$ to represent the spatial posture of each joint of the leg, the kinematic equation for the swing phase is Equation (1):

$${}_4{}^{w}T_{sw}={}_0{}^{w}T_{sw}\cdot {}_1{}^{0}T_{sw}\cdot {}_2{}^{1}T_{sw}\cdot {}_3{}^{2}T_{sw}\cdot {}_4{}^{3}T_{sw}$$

$${}_4{}^{w}T_{sw}=\left[\begin{array}{cccc}{c}_1{c}_{23}& -c_3{s}_{23}& s_1& l_3{c}_1{s}_{23}+l_2{c}_1{c}_2+l_1{c}_1+a/2\\ s_1{c}_{23}& -s_1{s}_{23}& -c_1& l_3{s}_1{c}_{23}+l_2{s}_1{c}_2+l_1{s}_1-b/2\\ s_{23}& c_{23}& 0& l_3{s}_{23}+l_2{s}_2-h\\ 0& 0& 0& 1\end{array}\right]$$

(1)

where s₂₃ = sin(θ₂ + θ₃), c₂₃ = cos(θ₂ + θ₃), s_i = sinθ_i, and c_i = cosθ_i.

P_x, P_y, and P_z, which can be obtained from Equation (1), are the origin vector elements of the position of the foot end relative to the basic coordinate system, and they are expressed in Equation (2).

$$\begin{array}{l}{p}_x^{sw}=l_3{c}_1{s}_{23}+l_2{c}_1{c}_2+l_1{c}_1+a/2\\ p_y^{sw}=l_3{s}_1{c}_{23}+l_2{s}_1{c}_2+l_1{s}_1-b/2\\ p_z^{sw}=l_3{s}_{23}+l_2{s}_2-h\end{array}$$

(2)

The inverse transformation of the chi-square matrix [7] is used to obtain the transformation equation of the foot end position with respect to the angle of the joint associated with the lateral swing θ_sw1, the angle of the thigh joint θ_sw2, and the angle of the calf joint θ_sw3, as shown in Equation (3):

$$\begin{gathered} {\theta }_{1sw}=\arctan (\frac{p_y^{sw}+\frac{b}{2}}{h-p_z^{sw}}) \\ {\theta }_{2sw}=arcsin\left[\frac{D_2}{\sqrt{{\left(p_x^{sw}-\frac{a}{2}\right)}^2+{D_1}^2}}\right]-\arccos \left[\frac{p_x^{sw}-\frac{a}{2}}{\sqrt{{\left(p_x^{sw}-\frac{a}{2}\right)}^2+{D_1}^2}}\right] \\ {\theta }_{3sw}=arccos\left(\frac{p_x^{sw}-\frac{a}{2}-l_2{s}_2}{l_3}\right)-arcsin\left[\frac{D_2}{\sqrt{{\left(p_x^{sw}-\frac{a}{2}\right)}^2+{D_1}^2}}\right]+\arccos \left[\frac{p_x^{sw}-\frac{a}{2}}{\sqrt{{\left(p_x^{sw}-\frac{a}{2}\right)}^2+{D_1}^2}}\right] \end{gathered}$$

(3)

where $\begin{array}{l}{D}_1=\sqrt{{\left(h-p_z^{sw}\right)}^2+{\left(p_y^{sw}+\frac{b}{2}\right)}^2}-l_1\\ D_2=\frac{l_3{}^2-{\left(p_x^{sw}-\frac{a}{2}\right)}^2-{\left(\sqrt{{\left(h-p_z^{sw}\right)}^2+{\left(p_y^{sw}+\frac{b}{2}\right)}^2}-l_1\right)}^2-l_2{}^2}{2l_2}\end{array}$.

2.1.2. Supporting Phase of Leg Mechanism

When the leg mechanism is in the support phase, there is no side swing motion; therefore, the θ₁ of the side swing joints is constant at zero. The spatial coordinate system of each joint of the leg in this phase is shown in Figure 3. The θ₂, θ₃, and θ₄ denote the joint angles of the shoulder, knee, and ankle joints, respectively. In addition, the D–H parameters of the supported phase are shown in

Table 2. D–H parameters for the support phase of the right front (RF) leg mechanism.

i	ai−1/mm	αi−1/(°)	θ/(°)	di/mm
1	0	0	θ4	0
2	l3	0	θ3	0
3	l2	0	θ2	0
4	0	0	θ1	0

.

Then the kinematics equations of the supported phase are expressed in Equation (4).

$$\begin{gathered} {}_3{}^{0}T_{sp}={}_1{}^{0}T_{sp}\cdot {}_2{}^{1}T_{sp}\cdot {}_3{}^{2}T_{sp} \\ {}_3{}^{0}T_{sp}=\left[\begin{array}{cccc}{c}_3{c}_4-s_3{s}_4& -c_3{s}_4-c_4{s}_3& 0& l_2\left(c_3{c}_4-s_3{s}_4\right)+l_3{c}_4\\ c_3{s}_4+c_4{s}_3& c_3{c}_4-s_3{s}_4& 0& l_2\left(c_3{s}_4+c_4{s}_3\right)+l_3{s}_4\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right] \end{gathered}$$

(4)

P_x, P_y, and P_z, derived from Equation (4), are the origin vector elements of the foot’s end phase with respect to the position of the base coordinate system when the leg mechanism is in the support phase; they are expressed in Equation (5).

The inverse transformation of the chi-square matrix was used to obtain the transformation equation θ_2sp, θ_3sp for the foot-end position with respect to the angle of the transverse support related to the knuckle, as shown in Equation (6).

$$\begin{gathered} p_x^{sp}=l_2\left(c_3{c}_4-s_3{s}_4\right)+l_3{c}_4 \\ {p}_y^{sp}=l_2\left(c_3{s}_4+c_4{s}_3\right)+l_3{s}_4 \end{gathered}$$

(5)

$$\begin{gathered} {\theta }_{2sp}=arcsin\left[\frac{D_2}{\sqrt{{\left(p_x^{sp}-\frac{a}{2}\right)}^2+{D_1}^2}}\right]-\arccos \left[\frac{p_x^{sp}-\frac{a}{2}}{\sqrt{{\left(p_x^{sp}-\frac{a}{2}\right)}^2+{D_1}^2}}\right] \\ {\theta }_{3sp}=arccos\left(\frac{p_x^{sp}-\frac{a}{2}-l_2{s}_2}{l_3}\right)-arcsin\left[\frac{D_2}{\sqrt{{\left(p_x^{sp}-\frac{a}{2}\right)}^2+{D_1}^2}}\right]+\arccos \left[\frac{p_x^{sp}-\frac{a}{2}}{\sqrt{{\left(p_x^{sp}-\frac{a}{2}\right)}^2+{D_1}^2}}\right] \end{gathered}$$

(6)

where $\begin{array}{l}{D}_1=\sqrt{h^2+{\left(p_y^{sp}+\frac{b}{2}\right)}^2}-l_1\\ D_2=\frac{l_3{}^2-{\left(p_x^{sp}-\frac{a}{2}\right)}^2-{\left(\sqrt{h^2+{\left(p_y^{sp}+\frac{b}{2}\right)}^2}-l_1\right)}^2-l_2{}^2}{2l_2}\end{array}$.

The kinematic inverse solution provided a theoretical basis for the subsequent programming of the control system in the Matlab simulation by obtaining the variables of each joint.

2.2. Energy Model

To analyze the energy consumption of the Cyber Dog, it was necessary to establish an energy consumption model. For the energy consumption of the Cyber Dog in this paper, only the mechanical power was considered, so the calculation of energy consumption was obtained by integrating the mechanical power into the gait cycle, where power is the product of joint moment and joint angular velocity. This is shown in Equation (7), where T is the gait cycle, E is the energy consumption in one gait cycle, and P is the joint power. In this paper, the dynamics of the robot are not considered; the joint moments and joint angular velocities of the robot can be observed, and the data were obtained by the oscilloscope module in Simulink.

$$\begin{gathered} E={{\displaystyle \int }}_0^{T}Pdt \\ E={{\displaystyle \int }}_0^T\left|{\tau }_{\mathrm{i},\mathrm{j}}\cdot \stackrel{\cdot }{{\theta }_{\mathrm{i},\mathrm{j}}}\right|dt \end{gathered}$$

(7)

In the second line of Equation (7), i is the i-th leg of the quadruped robot, j is the j-th joint of the quadruped robot, τ is the quadruped robot’s joint moment, and $\stackrel{\cdot }{\theta }$ is the angular velocity of the quadruped robot’s joints.

2.3. Foot Trajectory Analysis

A well-designed foot-end trajectory can reduce joint moments and angular acceleration, reduce energy consumption, and increase the energy consumption rate. The composite pendulums and linear foot trajectories, commonly used foot trajectories, and the end-of-foot trajectory of a robot walking one step read by the Simulink oscilloscope module, as shown in Figure 4, were compared with the trajectories obtained from subsequent simulations of energy consumption. The gait parameters considered were S = 0.2 m, T = 0.8 s, and H = 0.15 m, which represent the displacement of the robot’s X-axis walking in one gait cycle, the gait period, and the maximum height of the robot’s foot lift in the Z-axis direction, respectively. In this paper, the side swing motion of the leg mechanism of the quadruped robot was not explored, so the motion of the foot end in the y-axis direction was not considered.

3. Methods

RL, the experimental method of this paper, is generally regarded as a process run by a Markov decision process (MDP), which is composed of the element groups {S, A, P, R, $\gamma$}. S = S₁, S₂ … S_t are the state sets, which are the sets of all states that the MDP can have. As shown below, the state S_t at any moment is a specific state in the state set s, where s_t = s, s ϵ S. A is the set of all behaviors that the actors can perform. Finally, P (s_t+1|s_t, a_t) represents what a given agent performs at time step t, and $\gamma$ ∈ (0, 1) is the discount factor. Although P (s_t+1|s_t, a_t) is usually a complex, high-dimensional unknown task, samples can be collected through simulation or in the real world, and R (s_t, a_t) is the reward value obtained by s_t when the agent performs the action of a_t. A policy in RL is defined as the mapping from the state space to the operation space. In the strategy search, the goal is to find the optimal strategy π to maximize the long-term expected reward:

$$\eta \left(\pi \right)=E_{\rho \pi \left(s\right)\pi \left(a|s\right)}\left[Q^{\pi }\left(s,a\right)\right]$$

(8)

where

$$Q^{\pi }\left(s_t,a_t\right)=E_{s_{t+1},a_{t+1},\dots }\left[\Sigma \underset{t=0}{\overset{T}{\Sigma }}{\gamma }^{t}r\left(s_t,a_t\right)\right]$$

(9)

where Q^π(s) is the state distribution under policy π, that is, the probability of accessing state S under policy π, and Q^π (s_t, a_t) is the state action value function for calculating state action pairs. From s_t+1 to P (s_t+1|s_t, a_t) and to π(a_t|s_t) and t are terminal time steps. The reward function in reinforcement learning is as follows, which includes speed stability, straight walking, trunk vibration deviation, keeping forward and energy consumption, to encourage the robot to achieve the expected training results more quickly.

$$r_t=A{r}_v+B{r}_{\mathrm{y}\mathrm{offset}}+C{r}_{\mathrm{Fuselage}\mathrm{stable}}+D{r}_{\mathrm{Keep}\mathrm{moving}}+E{r}_E$$

(10)

where $r_v={(v_0-v_x)}^2$ is the quadruped robot’s reward for training while maintaining its speed, $v_0$ is the real-time velocity of the robot’s motion, $v_{\mathrm{x}}$ is the velocity that the robot motion needs to reach, and A is the weighting factor for the velocity term. $r_{yoffset}=y^2$ is the deviation of the fuselage in the y direction, which is related to whether the fuselage can walk in a straight line; B is the weighting factor of the y-axis offset term. $r_{Fuselagestable}={\hat{z}}^2$ represents the z-direction offset of the quadruped robot’s trunk during the movement, and C is the weight factor of the fuselage z-direction offset term. $r_{Keepmoving}=\frac{T_S}{T_f}$ in order to maintain the training time of the quadruped robot, T_s is the sampling time of the environment, T_f is the simulation time of the environment, and D is the weighting factor for the keep-moving incentive term. $r_E={\displaystyle {\int }_0^{10}{\tau }_{\mathrm{i},\mathrm{j}}\cdot \stackrel{\cdot }{{\theta }_{\mathrm{i},\mathrm{j}}}}dt$ is the energy consumption of the quadruped robot, where the environmental simulation time is 10 s, so the integral range is 0–10; the coefficient E is the weighting factor for the energy consumption incentive item. Each of the weighting coefficients A, B, C, D, and E needs to be obtained through subsequent simulations.

Considering the challenges of concurrently optimizing the quadrupedal robot’s speed of motion, straight-line walking, and energy consumption, which involve complex and difficult training, extended time requirements, and the risk of converging to the local optimal solution, this study divided the learning process into two stages. The first stage employed the DDPG algorithm to train the quadrupedal robot to maintain a stable speed and direction of motion. This was achieved by pruning the double-q learning, delaying the update of the goal and policy networks, and implementing goal policy smoothing to avoid overestimating the value function; the energy consumption term was not considered at this time, so the energy consumption term weighting factor E = 0. The second phase used the same DDPG algorithm to maintain stable motion on the basis of the linear motion of the quadruped robot while minimizing energy consumption. One notable advantage of this approach is that it provides faster simulation speeds, reduces experimental bias, and ensures that the robot is simulated with minimal energy consumption under ideal conditions. The reinforcement learning flowchart is shown in Figure 5; according to the joint moments and joint angular velocities fed back by the robot, the intelligent body unfolded its actions after intelligent decision-making and obtained feedback about the previous moment’s behavior, i.e., reward, from the changing state, which guided the subsequent behavioral decisions. Observations and actions for reinforcement of learning are shown in

Table 3. Observation and action content for reinforcement of learning.

Observation (Major)		Action
Hip joint angular velocity (RF)	Hip joint torque (RF)	Hip joint torque (RF)
Femoral joint angular velocity (RF)	Femoral joint torque (RF)	Femoral joint torque (RF)
Knee joint angular velocity (RF)	Knee joint torque (RF)	Knee joint torque (RF)
Hip joint angular velocity (RH)	Hip joint torque (RH)	Hip joint torque (RH)
Femoral joint angular velocity (RH)	Femoral joint torque (RH)	Femoral joint torque (RH)
Knee joint angular velocity (RH)	Knee joint torque (RH)	Knee joint torque (RH)
Hip joint angular velocity (LF)	Hip joint torque (LF)	Hip joint torque (LF)
Femoral joint angular velocity (LF)	Femoral joint torque (LF)	Femoral joint torque (LF)
Knee joint angular velocity (LF)	Knee joint torque (LF)	Knee joint torque (LF)
Hip joint angular velocity (LH)	Hip joint torque (LH)	Hip joint torque (LH)
Femoral joint angular velocity (LH)	Femoral joint torque (LH)	Femoral joint torque (LH)
Knee joint angular velocity (LH)	Knee joint torque (LH)	Knee joint torque (LH)

.

The training was performed on a Matlab-based training platform with the Cyber Dog quadruped robot model. The model parameters were consistent with the body and joint sizes of a real quadruped robot. Two participant networks and two critic networks were used for the training network. The participant and critic networks consisted of three fully connected hidden layers with 128 neurons each and used the ReLU activation function. Furthermore, the algorithm was implemented using 10 random seeds, and details of the other training parameters are shown in

Table 4. Training environment parameters.

Hyper Parameter	Value
Value limit for torque commands	25
Actor learning rate	0.0004
Critic learning rate	0.002
Max steps	400
Score averaging window length	250
Stop training value	190
Save agent value	200
Joint control range	−45~45°
Max episodes	20,000

.

The results of continuous reinforcement learning were used to correct the reward function parameters. This yielded weighting coefficients A, B, C, D, and E of −2, −20, −50, 25, and −0.2 for each reward term, respectively. Thus, the robot could keep walking in a straight line, and the energy consumption reached the optimal state of the reward function in Equation (11); the reward function curve is shown in Figure 6.

$$r_t=-2r_v-20r_{\mathrm{y}\mathrm{offset}}-50r_{\mathrm{Fuselage}\mathrm{stable}}+25r_{\mathrm{Keep}\mathrm{moving}}-0.2r_E$$

(11)

The training process in Figure 6 shows that the energy consumption of the quadruped robot at this time is theoretically the optimal cycle that RL thinks can consume the least energy and maintain the speed of stable walking.

A theoretical analysis of the energy-optimal motion state of a quadruped robot can be performed using reinforcement learning. However, due to the autonomous nature of the learning process, effective gait cycles cannot be established. The method used in this paper was to extract the simulation data with the minimum energy consumption from the cycle with the optimal reward function, add the transform sensor module in Simulink, and connect it to an oscilloscope to observe the changes in the x and z coordinates of the robot’s foot-end. The foot-end trajectory record with the smallest energy consumption was extracted as the foot-end trajectory of the simulated trajectory, and the obtained trajectory is shown in Figure 7.

The kinematic inverse Equations (3) and (6) were implemented within the Simulink function module by Matlab. Then, the spatial position coordinates x, y, and z of the three foot-end trajectories were imported into the simulation system for simulation. An oscilloscope was connected to the joints to record the joint moments and angular velocities, and the difference in energy consumption between the simulated foot-end trajectories and the above trajectories at the same period and velocity can be calculated using Equation (7). For the control of the position and pose of the robot, PID control was used, and the control was built using Simulink in Matlab, as shown in Figure 8. The gait planning in the PID control used a diagonal gait, as shown in Figure 9, where the left front leg, right front leg, right hind leg, and left hind leg of the robot are divided and denoted by LF, RF, RH, and LH; the white squares represent the lifting of the leg in the swing phase, and the black squares represent the landing of the leg in the support phase, where T is the gait period, which was set to 0.8 s in this experiment. The equation for the relationship between the body velocity v and the step length S and the foot-end period T_m is as follows: $v=S/T_m$, where T_m is the period of 0.2 s for one step of foot-end walking.

4. Result and Discussion

In this section, simulation and prototype experiments are performed to compare the energy of the simulated foot-end trajectories with two common plantar trajectories at the same period and velocity, and the results of the experiments are discussed.

4.1. Simulation Experiment

The experimental setup had a gait period of 0.8 s, divided into a support phase and a swing phase, each with a duration of 0.4 s. Figure 7 demonstrates the comparison of the robot movement power consumption with different plantar trajectories at three different speeds, 1 m/s, 1.2 m/s, and 1.4 m/s, within the same motion time. It can be seen that the simulation trajectory did not have obvious peak fluctuations in one gait cycle, and the change in the power was much more stable; the other two trajectories both had obvious sudden changes in the power at the moments of the interaction between the swing phase and the support phase, which exacerbated the energy consumption. Figure 10 shows that the simulated foot trajectory was more competitive than other common foot tracks, along with the power of the composite cycloid foot trajectory and the straight foot trajectory at three different speeds. The energy consumption in different cases was obtained by the calculation of Equation (7), and the energy reduction of this paper’s method over the other two common foot ends at different speeds was calculated by Equation (12), as shown in

Table 5. Energy consumption of three foot trajectories (kJ) at speed.

	1 m/s	1.2 m/s	1.4 m/s
Simulation trajectory	1.6236	1.8152	2.8261
Composite pendulum line trajectory	1.8451	2.0489	3.1021
Straight path	2.3571	2.7571	3.2871
Decrement	12%	11%	9%

. It shows that the end-of-foot trajectories with velocities of 1 m/s, 1.2 m/s, and 1.4 m/s reduced the energy consumption by 9% to 12% compared with the other two trajectories. However, the ability to reduce energy consumption diminished with increasing speed. This may be due to the need to further optimize the joint moments and joint angular velocities for high-attitude motion states as the robot moves faster.

$$D_t=\frac{E_{min}-E_S}{E_{min}}$$

(12)

In Equation (12), E_min is the energy consumption value of the other two foot-end trajectories with less energy consumption during the movement, E_s is the energy consumption value of the simulated foot-end trajectory during the movement, and D_t is the energy consumption reduction.

4.2. Prototype Experiment

In contrast to simulation environments, real-world environments have a multitude of disturbances, such as joint friction and varying environmental conditions. Hence, it is important to validate the methodology presented in this paper by testing it on a physical prototype. In order to evaluate the difference between the direct data obtained from the simulation and the data obtained in the real environment, the data obtained from the simulation was integrated into the Cyber Dog quadruped robot.

A 150-meter-long plane was chosen as the experimental area. The environmental variables were identical to the simulated environment. As shown in Figure 11, cue cards were placed every 10 m in the experimental area in order to measure the stability of the robot’s movement speed. The simulation data were transferred through the Type-C interface of the Cyber Dog. An energy consumption experiment was conducted during 100 m of robot travel using three different trajectories—a simulated trajectory, a compound pendulum trajectory, and a straight-line trajectory—and comparing the energy consumption of the robot on the basis of the level of power drop on the controller board.

The robot was placed in the experimental environment and tasked with walking 100 m at a speed of 1 m/s. Comparison experiments on power consumption were conducted using different foot trajectories, and the position variations of the robot’s left forelimb (LF) foot were recorded during a gait cycle; the results are shown in Figure 12.

As evident from the experimental results in Figure 12, the power consumption in the real experiment with a simulation trajectory at a speed of 1 m/s was only 9%, which is significantly lower than the 18% power consumption with the composite pendulum trajectory and the 25% power consumption of the linear trajectory. In comparison with the higher power consumption of the composite pendulum trajectory, the simulated trajectory reduced the energy consumption by 9%. It is worth noting that the simulation results of the experiments conducted at a speed of 1 m/s showed that the energy consumption can be reduced by 12% using the method of this paper, but the results of the real experiments showed that the method of this paper can reduce the energy consumption by 9%, and the comparison revealed that there was a 3% error amplitude between the simulation and the real experiments, which may be attributed to the omission of factors such as the robot’s thermal power and joint friction loss in the energy consumption calculations. Considering that the value of the battery’s power display is not reliable, but there may be display errors, the experiment was carried out after using a multimeter to measure the battery’s voltage at the end of each charge to make it equal to 25 V to ensure that the robot consumed the same amount of energy during the experiment. The Cyber Dog could walk 3600 m at 1 m/s using a composite pendulum foot-end trajectory in a fully charged state. Using the foot-end trajectory simulated in this paper to move at 1 m/s at full power, the Cyber Dog could walk 3924 m, which is an increase of 324 m in walking distance, and significantly improved the robot’s endurance in complex and changing environments.

Then, the walking speed of the robot was adjusted to 1.2 m/s and 1.4 m/s for 100 m walking experiments. The magnitude of energy consumption was obtained by comparing how much battery power was consumed by the operator panel for different foot trajectory movements at different speeds, calculated with Equation (13). Subsequently, experiments were conducted to compare the power consumption of the robot when moving with three different foot-end trajectories at varying speeds. The results are shown in Figure 13.

$$P_C=E_f-E_n$$

(13)

where E_f is the initial power, E_n is the remaining power after movement, P_c is the power consumption of the robot, and the initial power is 95%.

The comparative analysis of energy consumption at three different speeds revealed that the simulated trajectory consumed less energy than the other two trajectories. The energy consumption of the Cyber Dog quadrupedal robot increased with the speed of motion in a trend, consistent with the simulation results. Under the experimental conditions in this paper, the simulated trajectory achieved energy savings of 7% to 9% compared with the two common plantar trajectories, and the margin of error between the simulation and the real experiment was 3%. These simulation results are consistent with the experimental findings, thereby validating the effectiveness of the method proposed in this paper.

4.3. Discussion

Compared with the energy consumed by the two common plantar trajectories at varying velocities, the proposed method consistently resulted in lower overall energy consumption for the quadrupedal robot at the same velocity. This may be attributed to the finer control of joint motors by the RL intelligent body, which enabled the output of smaller joint moments and joint angular velocities while maintaining the velocity in order to reduce the power of the various joints of the quadrupedal robot at each moment. Further optimization of robot energy consumption using reinforcement learning can explore aspects like friction energy consumption between joints, joint onset angle, and foot lifting height. These factors can improve the energy efficiency of the legged robot and enhance the endurance of the robot. The method in this paper allows for training in a simulated environment, preventing the robot from accidental damage during training, and reducing resource wastage, which is of vital significance for high-value robots.

It is worth noting that legged and footed robots require different motion postures under different working conditions, which have been rarely studied by reinforcement learning. Particularly, the study of energy consumption in the robot’s high-posture motion is one of the key focal points for future research. Furthermore, robots with different morphologies may have different consumption for energy. It is intriguing to explore the application of reinforcement learning for optimizing the energy consumption of robots with different morphologies.

5. Conclusions

The following conclusions can be made through the findings of this paper:

• This paper proposes a reinforcement learning method for quadrupedal robots based on the DDPG algorithm aimed at minimizing energy consumption in gait patterns.
• Training was performed in two stages using the DDPG TD3 algorithm, which improved the learning efficiency and accuracy of the results.
• We compared the energy consumption obtained in simulations between this paper’s method and common foot tracks: straight line trajectories and composite pendulum trajectories.
• It was found that the plantar trajectories obtained by this paper’s methods outperformed the other two plantar trajectories at different walking speeds of the robot, consuming 7% to 9% less energy in the same movement time.
• This paper focused on the optimization of energy consumption of quadrupedal robots without considering the friction between the foot end and joints, the initial attitude of the body, and the terrain conditions.

In the future, we will take these variables into account so that the results are more closely matched to the real situation and the error is minimized. In addition, we intend to extend this approach to study other types of robots, further improve the universality of this approach, and form a kind of machine learning model adapted to most of the robots.