Development of a New Robust Stable Walking Algorithm for a Humanoid Robot Using Deep Reinforcement Learning with Multi-Sensor Data Fusion

Abstract

The difficult task of creating reliable mobility for humanoid robots has been studied for decades. Even though several different walking strategies have been put forth and walking performance has substantially increased, stability still needs to catch up to expectations. Applications for Reinforcement Learning (RL) techniques are constrained by low convergence and ineffective training. This paper develops a new robust and efficient framework based on the Robotis-OP2 humanoid robot combined with a typical trajectory-generating controller and Deep Reinforcement Learning (DRL) to overcome these limitations. This framework consists of optimizing the walking trajectory parameters and posture balancing system. Multi-sensors of the robot are used for parameter optimization. Walking parameters are optimized using the Dueling Double Deep Q Network (D3QN), one of the DRL algorithms, in the Webots simulator. The hip strategy is adopted for the posture balancing system. Experimental studies are carried out in both simulation and real environments with the proposed framework and Robotis-OP2’s walking algorithm. Experimental results show that the robot performs more stable walking with the proposed framework than Robotis-OP2’s walking algorithm. It is thought that the proposed framework will be beneficial for researchers studying in the field of humanoid robot locomotion.

1. Introduction

Today, robots are used in many areas, from automation systems to the defense industry. Artificial intelligence and learning algorithms are subject to continuous development and improvement so that robots can efficiently perform the daily tasks of humans. Thus, robotics has become an essential element and has found a place in many different actions in daily life.

Robotic systems can be classified into two main areas: manipulator robots with a fixed workspace and mobile robots with a portable workspace. Although the wheeled robots can travel quite quickly, they can only go across flat ground. While slower than wheeled robots, tracked robots can go across more difficult terrain. In challenging terrain, legged robots outperform wheeled and tracked robots, as they show greater mobility and flexibility, are highly adaptable to terrain differences, and cause less environmental damage. Considering this superiority and the fact that approximately 80% of the earth is inaccessible by conventional wheeled vehicles, legged robots are more prominent in the field of mobile robots [1].

Humanoid robots are two-legged robots made with a human-like design and have different functions, such as being able to move and speak. They have an important place among mobile robots because of their similarity to humans, ability to move in environments suitable for humans, and ability to use tools designed for humans.

A humanoid robot’s design focuses primarily on balance control. A simple way to control the balance of a bipedal walking robot is to mimic the locomotion of humans, which is typically human. An average human gait consists of all the movements that occur with the forward orientation of the Center of Gravity (CoG) through the limbs and body. During walking, displacements and load are carried out by the legs, while, by adjusting to the CoG changes brought on by the movement of the legs, the remainder of the body moves on.

In biomechanical research, understanding the bipedal locomotion stability and gait mechanism is essential for better understanding how humans move from one place to another [2]. Although it may seem simple, human locomotion is a highly complex state involving multiple Degrees of Freedom (DoF) combined with the complex nonlinear dynamics produced due to the various extensor and flexor muscle groups in the lower body. While humanoid robots are known for their ease and flexibility to move around a wide range of terrain, the main concern is stability as they have a high-dimensional and nonlinear system. Over the years, many control scientists have developed an interest in finding a solution to the stability issue with the bipedal gait system [3].

There are approaches to the development of stable walking when recently proposed gait frameworks are reviewed. The dynamic model of a robot, for which the gait planner and controller are created, forms the basis of the fundamental framework. Some restrictions are taken into account in this framework to lessen the difficulty of creating a whole-body dynamics model Another framework’s primary component is a network of signal generators that together produce intrinsically rhythmic signals [4,5]. The Central Pattern Generator (CPG) is the basis for this kind of framework. It draws inspiration from research on the neurophysiology of vertebrate and invertebrate species [6,7]. These studies have demonstrated that CPGs in the spinal cord that are coupled in a specific way are responsible for rhythmic locomotion, including walking, running, and swimming. Oscillators are often assigned to each link in this kind of framework to produce set points (torque, position, etc.). Many humanoid robots have more than 20 DoF. As a result, adjusting the oscillators’ parameters is both challenging and labor-intensive in terms of experimentation. Robots’ knowledge is typically static in these two types of frameworks mentioned above. It does not improve from past experience. For this reason, they must at least reconfigure the parameters to adapt to new environments. According to another framework, based on Reinforcement Learning (RL), walking trajectories are generated [8]. According to this framework, walking trajectories are generated following training that needs a lot of samples and takes a long time. The framework attempts to learn how to generate walking trajectories depending on a function during training. The methodologies outlined above are combined to create the final framework [9,10,11,12,13]. This kind of framework is also referred to as a hybrid walking framework. It seeks to maximize performance by utilizing the various strengths of each strategy.

Traditional control theory methods rely on complex deterministic and mathematical engineering models. One of the most widely used models, the inverted pendulum, is the source of several algorithms, including those in [14,15,16,17]. The Zero Moment Point (ZMP) is the traditional method accepted as an indicator of dynamic stability in bipedal robots [18]. The robot’s dynamic balance is preserved when it reaches the ZMP because the its foot’s response to the ground balances out the dynamics brought on by its locomotion [18]. The robot’s Center of Mass (CoM) is calculated to acquire the ZMP. Therefore, it is necessary to use simulation computations or force sensors that are mounted to the robot’s feet. However, for a small-size humanoid robot with constrained system resources and processing rates, calculating the ZMP for each step takes some time. Moreover, traditional approaches mainly rely on dynamics and mathematical models for both the robot and terrain. Therefore, it requires a vast amount of time and effort for designers. The model needs to be redesigned when either the terrain or the kind of robot changes. In addition, the past knowledge and expertise of the designers, who cannot fully explore the robot’s potential, also impact the performance of the human-designed model. Traditional control systems’ fundamental drawback is their heavy dependence on the correctness of a mathematical model, which may be affected by joint friction, ground contact force, or other uncertainties.

The RL-based control approach is an alternate method of addressing the issues mentioned above with humanoid robot walking. Due to its adaptable learning capabilities, the RL is increasingly applied in the field of bipedal robotic gait control. The RL is a branch of machine learning that may be used to train complicated control systems without using models. An agent (robot) may learn how to control itself in various circumstances by interacting with the environment, thanks to the RL. According to the environmental states and the agent’s behaviors, the environment is modeled in real life to either reward or penalize the agent. The agent works on developing the ability to utilize the past to forecast which behaviors will result in the greatest reward in the future. The development of numerous action policy learning algorithms, many of which are based on the Markov Decision Process (MDP), has been crucial to the model-free learning of bipedal walking in particular [19,20]. The RL aids in getting over the difficulties of dynamic design and computation. For walking problems, the RL is a type of intelligent learning technique. The ZMP location can be controlled using RL approaches to ensure walking stability. While motion control problems can be handled with RL with great performance, legged robot gait control is still difficult because of its complexity.

For the walking pattern, trajectories are generated that allow the robot to walk as desired. Numerous walking parameters are involved in trajectory generation. The procedure of manually adjusting the walking parameters is quite difficult and complex, especially for a sophisticated robot with more than 10 DoF. Only little distortions are permitted by the controls used to construct typical trajectories. Walking might break down due to even the smallest shift. It is possible that these controllers cannot be adjusted to diverse terrains such as slopes and stairs. As a result, it becomes necessary to alter various walking parameters, which significantly raises the cost and workload.

The feature engineering for traditional RL algorithms comes from observations. It might be difficult to extract features for complicated issues, or there may not be enough data from which to develop a strong model. The bipedal gait is substantially more difficult since it calls for a high-dimensional state and action space and demands careful control of each joint while maintaining stability. Extracting high-level features from data with a wide state space and missing observations is now possible thanks to a more recent technique called Deep Neural Networks (DNNs). Deep Reinforcement Learning (DRL) enables an agent to interact with the environment more complicatedly, thanks to recent developments in DNNs. Some end-to-end DRL techniques teach the robot model using the default reward of a simulator. Such DRL controllers, however, have the potential to produce movements that are unsuitable for robots. Utilizing a robot simulator and prior knowledge is one way to mitigate this scenario. Additionally, for RL without previous knowledge, hyperparameters are frequently sensitive. Applications are constrained by low convergence and ineffective training. In this study, an efficient new framework built on the humanoid robot model Robotis-OP2 combined with the controller that generates the traditional trajectory and DRL is proposed to get over these constraints. Most of the standard machine learning methods are suitable for supervised learning or unsupervised learning methods. The main reason for choosing DRL from the machine learning methods in this paper is its interaction with the environment and the high cost of creating datasets on robot systems. It is ensured that the robot can learn the optimum with the reward/punishment mechanism according to the status information received from the environment with the sensors during walking. In this study, the Artificial Neural Network (ANN) is trained directly with data from the robot, without any ready dataset. Thanks to the proposed framework, the optimum walking parameters of the trajectories generated with the walking pattern generator are obtained with the Dueling Double Deep Q Network (D3QN) [21]. The training of the D3QN is carried out using the robot’s multi-sensors in the Webots [22] simulator. After determining optimum gait parameters, a robot posture stabilization system in the sagittal plane is proposed. Experimental studies are performed in the Webots simulation environment and real environment with the proposed framework and Robotis-OP2’s walking algorithm by transferring the controllers to the real robot. Experimental results have shown that the Robotis-OP2 humanoid robot can walk more stably on flat ground both in the simulator and in the real environment with the proposed framework than Robotis-OP2’s walking algorithm. In addition, it is the first study in the literature to optimize walking parameters for the stable walking of humanoid robots using DRL.

There are many studies in the literature on bipedal walking using RL. In OpenAI Gym [23], many algorithms are employed to resolve bipedal challenges by applying RL to directly control the joints [24]. However, convergence and performance are degraded when these algorithms are utilized to control the gait of a robot whose dynamics are significantly more complicated. Additionally, a nonhuman gait may also be present in the trained model, which is often unsatisfactory. For RL-based gait controllers, previous knowledge-based training procedures have become a critical operational principle.

Recently, there have been several attempts to incorporate RL into bipedal robotic gait control without computing the mathematical model based on model-free RL frameworks. The pose sequence that enables an NAO robot to travel the most distance in the least amount of time while walking on a level surface without falling was discovered by Gil et al. [25] using Q-learning [26]. Liu et al. [27] used the Policy Gradient (PG) [28], which is one of the RL algorithms, to correct the gait model parameters of the NAO humanoid robot to make the gait resistant to unknown disturbances. Lin et al. [29] proposed a method for dynamic bipedal gait and balance control using Q-learning without prior knowledge of the dynamic model. The bipedal robot was able to maintain static stability thanks to the balancing learning approach, which shifts the ZMP on the robot’s soles using the movement of the arm and leg. The seesaw and bipedal walking on a level surface were subjected to balancing algorithms. According to the simulation results, the robot might learn to enhance its walking speed behavior using the proposed strategy.

Silva et al. [30] proposed a Q-learning-based method for learning the action policy that enables a robot to walk upright on a slightly inclined surface. The system’s proposed design combines a standard gait generator with an RL component on two layers. This situation allows an accelerometer to be used when the slope of the ground the robot is walking on changes to provide a gait adjustment. Experimental studies on a real robot have shown that the stability problem can be successfully solved. Silva et al. [31] aimed to optimize the parameter values of the gait model generator to provide a fast and dynamically stable gait for the DARwIn-OP humanoid robot. They achieved this by using the Temporal Difference (TD) [32] algorithm from the RL methods.

In a two-dimensional simulator, robot walking was accomplished using the Deep Deterministic Policy Gradient (DDPG) [33] technique by Kumar et al. [34]. In around 25,000 episodes, their agents received the target score. The Recurrent Deep Deterministic Policy Gradient (RDDPG) [35] algorithm was used by Song et al. [36] to address the partial observability problem of bipedal walking, and the results were superior to those of the original DDPG approach. A modular framework was presented by Kasaei et al. [37] to provide stable bipedal mobility by tightly linking the analytical walking method with the DRL. To identify the optimal parameters and learn how to increase the robot’s stability by modifying the height of the CoM, a learning framework based on evolutionary algorithms and Proximal Policy Optimization (PPO) [38] was designed.

In [39], a controller combined with conventional control and RL was trained to walk the Robotis-OP3 humanoid robot in the PyBullet simulation environment. The study is divided into two parts: pose optimization and DRL. In the first part, they used the RL algorithm, Q-learning, to obtain a combination of the walking parameters of the traditional controller and optimized it according to the robot’s state. The PPO algorithm was used to train the multi-layer neural network in the second part, where the action space was created with the first phase. Thus, they obtained which pose order would enable the bipedal robot to perform the required task.

Zhang et al. [40] proposed the LORM (Learn and Outperform the Reference Motion) method, a DRL-based framework for the bipedal robot gait, utilizing prior knowledge of reference motion. In comparison to the reference motion and other motion-based approaches, the agent trained using the proposed method performed better. The proposed method has been validated on the DARwIn-OP humanoid robot in the Webots simulator. Christiano et al. [41] compared the possible trajectories of humans and humanoid robots. They developed a reward function using the data, then improved it with RL and updated the technique to DRL.

A two-level hierarchical control system was used by DeepLoco [42]. Low-level controllers first learned to work on a predetermined time scale and achieve stable walking. Second, by requesting the required step objectives from the low-level controller, the high-level controllers learned the time-scale policy of the steps. The actor-critic DRL method was applied to both levels of the hierarchy similarly. The NAO humanoid robot’s stability was maintained while walking on static and moving platforms in the simulated environment utilizing a new hybrid RL framework that was introduced in the study [43]. An iterative actor-critic RL system was used by Wawrzynski [44] to change a humanoid robot’s initial slow walk into a quick and capable one.

Two different forms of impedance controllers, to which RL algorithms were applied, were used by Feirstein et al. [45] to the enable limit-loop walking of a straightforward bipedal gait model. Leng et al. [46] proposed a Mean-Asynchronous Advantage Actor-Critic (M-A3C) RL algorithm to directly obtain the robot’s final gait without introducing the reference gait. It has been shown that the proposed method can provide continuous and stable gait planning for the bipedal robot. To solve the emerging problems of traditional gait control methods, the DRL algorithm was used in [47,48]. A biped controller based on the DDPG algorithm was created by Liu et al. [49] that can keep stability against static and dynamic disturbances.

A novel reward-adaptive RL technique for bipedal movement was proposed by Huang et al. [50]. They ensured that the control policy was optimized by more than one criterion simultaneously using a dynamic mechanism. The proposed approach used a multi-layered critic to identify a unique value function for each reward component, resulting in hybrid policy gradients. To construct a feedback system that can handle the walking pattern problem, a feedback system that combines an Adaptive Neural Fuzzy Inference System (ANFIS) [51] and a Double Deep Q-network (DDQN) [52] was proposed in [53]. To update the walking parameters, the output of the ANFIS was utilized for training a predictive model called DDQN.

Wong et al. [54] designed an oscillator-based gait model with sinusoidal functions for generating trajectory planning and obtaining bipedal motion for a humanoid robot. To ensure that the robot walks straight, the direction of rotation is considered a parameter of the walking pattern. They used Q-learning to create a simple walking pattern. According to the findings of experiments, the proposed framework enables the humanoid robot to walk steadily and straighter.

The rest of the paper is organized as follows. Section 2 includes the walking pattern generator created for gait planning and inverse kinematics analysis to determine the angles that the robot’s leg joints should take with this pattern. To find the optimal walking parameters, calculations of the orientation angles and ZMPs that form the state space of the DRL structure using multi-sensors are given in Section 3. In Section 4, the proposed stable walking framework is detailed. In this section, the optimization system of the walking parameters with the D3QN architecture and the stabilization system of the robot pose in the sagittal plane are analyzed, respectively. Experimental results are given in Section 5. The paper is concluded with the conclusions and future work in Section 6.

2. Walking Pattern Generator

In this study, a walking pattern generator is created as given in the flowchart in Figure 1. After determining the ankle and hip trajectories, the angle trajectories of the leg joints of Robotis-OP2 are obtained using inverse kinematic analysis.

The Robotis-OP2 humanoid robot with 20 DoF weighing 3 kg and its joint placement are shown in Figure 2. The robot’s joints are all actuated by the high-torque Dynamixel MX-28T smart servo motor.

An eight-phase walking cycle is planned to perform the basic walking process, as shown in Figure 3. The first two phases are the beginning phases of walking and include the necessary preparations for trajectory planning.

Phase 1: It is the initial phase used to define the robot’s trajectory coordinates.

Phase 2: It is the preparation phase of the robot before it starts walking. In this phase, the robot is ready to walk by moving the left foot forward a certain quantity and the right foot a certain quantity back.

Phase 3: This is the phase where the robot starts walking. The robot moves its CoG to the left foot. At this time, the feet are fixed on the ground and only the hip joints are shifted to the left.

Phase 4: When the CoG is moved to the left foot, the right foot begins to step.

Phase 5: The right foot finishes stepping and returns to the support area, and the CoG is ready to move to the right foot.

Phase 6: The robot moves its CoG to the right foot. In the meantime, the feet are fixed on the ground and only the hip joints are shifted to the right.

Phase 7: When the CoG is moved to the right foot, the left foot begins to step.

Phase 8: The left foot finishes stepping and returns to the support area.

Considering these eight phases, the six phases between Phase 3 and Phase 8 are repeated continuously to obtain a continuous walking process. A walking trajectory is planned for this process.

2.1. Walking Trajectory

In the gait model, cycloid curve functions are used to generate the ankle and hip trajectories. The walking pattern is generated separately in the x–z planes for the ankle and x–y planes for the hip. The ankle joints are the starting point of trajectory planning, and the hip joints are the ending point of trajectory planning.

2.1.1. Ankle Trajectory

The swing trajectory of the right ankle joint in the x–z plane is given in Equations (1)–(3) for Phase 3, Phase 4, and Phase 5, respectively. The swing trajectory, including these three phases for the left ankle joint, is given in Equation (4).

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0\\ {\mathrm{y}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array}, 0\le \mathrm{t}\le \frac{{\mathrm{t}}_{\mathrm{d}}}{2}\right.$$

(1)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}\right)\right)\\ {\mathrm{y}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=\frac{\mathrm{h}}{2}\left(1-\cos \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}\right)\right)\end{array} ,\hspace{1em}\right.\frac{{\mathrm{t}}_{\mathrm{d}}}{2}<\mathrm{t}\le \frac{\mathrm{T}-{\mathrm{t}}_{\mathrm{d}}}{2}$$

(2)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=2\mathrm{s}\\ y_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0\\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array}\right.,\hspace{1em}\frac{\mathrm{T}-{\mathrm{t}}_{\mathrm{d}}}{2}<\mathrm{t}\le \frac{\mathrm{T}}{2}$$

(3)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=\mathrm{s}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array},\hspace{1em}0\le \mathrm{t}\le \frac{\mathrm{T}}{2}\right.$$

(4)

where $\mathrm{t}$ is the current time (s), ${\mathrm{t}}_{\mathrm{d}}$ is the time of shift of the CoG in double support in Phase 5 to Phase 6 and Phase 8 to Phase 3 (s), $\mathrm{s}$ is one step length (mm), $\mathrm{h}$ is the maximum foot lift height (mm), and $\mathrm{T}$ is the walking cycle (Phase 3–8) time (s). In addition, ${\mathrm{x}}_{\mathrm{r}\_\mathrm{a}}$, ${\mathrm{y}}_{\mathrm{r}\_\mathrm{a}}$, ${\mathrm{z}}_{\mathrm{r}\_\mathrm{a}}$ and ${\mathrm{x}}_{\mathrm{l}\_\mathrm{a}}$, ${\mathrm{y}}_{\mathrm{l}\_\mathrm{a}}$, ${\mathrm{z}}_{\mathrm{l}\_\mathrm{a}}$ (mm) represent the trajectories of the right and left ankle joints in the x, y, and, z axes, respectively.

${\mathrm{t}}_{\mathrm{d}}$ is calculated by Equation (5).

$${\mathrm{t}}_{\mathrm{d}}=\frac{\mathrm{T}}{2}\times \mathrm{DSR}$$

(5)

where $\mathrm{DSR}$ is the double support ratio.

The swing trajectory for the right ankle joint, which includes Phase 6, Phase 7, and Phase 8, is given in Equation (6). The swing trajectory of the left ankle joint is given in Equations (7)–(9) for Phase 6, Phase 7, and Phase 8, respectively.

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=2\mathrm{s}\\ {\mathrm{y}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array}, \frac{\mathrm{T}}{2}<\mathrm{t}\le \mathrm{T}\right.$$

(6)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=\mathrm{s}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array}, \frac{\mathrm{T}}{2}<\mathrm{t}\le \frac{\mathrm{T}+{\mathrm{t}}_{\mathrm{d}}}{2}\right.$$

(7)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}\right)\right)+\mathrm{s}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0\\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=\frac{\mathrm{h}}{2}\left(1-\cos \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}}{\mathrm{T}-2{\mathrm{t}}_{\mathrm{d}}}\right)\right) \end{array},\hspace{1em}\frac{\mathrm{T}+{\mathrm{t}}_{\mathrm{d}}}{2}<\mathrm{t}\le \mathrm{T}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}\right.$$

(8)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=3\mathrm{s}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0 \\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{a}}\left(\mathrm{t}\right)=0\end{array}, \mathrm{T}-\frac{{\mathrm{t}}_{\mathrm{d}}}{2}<\mathrm{t}\le \mathrm{T}\right.$$

(9)

2.1.2. Hip Trajectory

The swing trajectories of the right and left hip joints in the x–y plane are given in Equations (10) and (11) for Phase 3, Phase 4, and Phase 5, respectively.

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{2\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}\right)\right)+\frac{\mathrm{s}}{2}\\ {\mathrm{y}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)=-\frac{\mathrm{w}}{2}\left(1-\cos 4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}\right)\\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)={\mathrm{h}}_{\mathrm{ah}}-{\mathrm{h}}_{\mathrm{b}}\end{array} ,\hspace{1em}0\le \mathrm{t}\le \frac{\mathrm{T}}{2}\right.$$

(10)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{2\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}\right)\right)+\frac{\mathrm{s}}{2}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)=-\frac{\mathrm{w}}{2}\left(1-\cos 4\mathsf{\pi }\frac{\mathrm{t}}{\mathrm{T}}\right)\\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)={\mathrm{h}}_{\mathrm{ah}}-{\mathrm{h}}_{\mathrm{b}}\end{array} ,\hspace{1em}0\le \mathrm{t}\le \frac{\mathrm{T}}{2}\right.$$

(11)

The swing trajectories of the right and left hip joints are given in Equations (12) and (13) for Phase 6, Phase 7, and Phase 8, respectively.

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{2\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}\right)\right)+\frac{3\mathrm{s}}{2}\\ {\mathrm{y}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{w}}{2}\left(1-\cos 4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}\right) \\ {\mathrm{z}}_{\mathrm{r}\_\mathrm{h}}\left(\mathrm{t}\right)={\mathrm{h}}_{\mathrm{ah}}-{\mathrm{h}}_{\mathrm{b}}\end{array} ,\hspace{1em}\frac{\mathrm{T}}{2}<\mathrm{t}\le \mathrm{T}\right.$$

(12)

$$\left\{\begin{array}{c}{\mathrm{x}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{s}}{2\mathsf{\pi }}\left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}-\sin \left(4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}\right)\right)+\frac{3\mathrm{s}}{2}\\ {\mathrm{y}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)=\frac{\mathrm{w}}{2}\left(1-\cos 4\mathsf{\pi }\frac{\mathrm{t}-\frac{\mathrm{T}}{2}}{\mathrm{T}}\right) \\ {\mathrm{z}}_{\mathrm{l}\_\mathrm{h}}\left(\mathrm{t}\right)={\mathrm{h}}_{\mathrm{ah}}-{\mathrm{h}}_{\mathrm{b}}\end{array} ,\hspace{1em}\frac{\mathrm{T}}{2}<\mathrm{t}\le \mathrm{T}\right.$$

(13)

where ${\mathrm{h}}_{\mathrm{ah}}$ is the height between the ankle joint and hip joint (mm), ${\mathrm{h}}_{\mathrm{b}}$ is the bending height used to adjust the ground clearance of the hip joint (mm), and $\mathrm{w}$ is the maximum right-to-left translation when the hip is swinging (mm). In addition, ${\mathrm{x}}_{\mathrm{r}\_\mathrm{h}}$, ${\mathrm{y}}_{\mathrm{r}\_\mathrm{h}}$, ${\mathrm{z}}_{\mathrm{r}\_\mathrm{h}}$ and ${\mathrm{x}}_{\mathrm{l}\_\mathrm{h}}$, ${\mathrm{y}}_{\mathrm{l}\_\mathrm{h}}$, ${\mathrm{z}}_{\mathrm{l}\_\mathrm{h}}$ represent the trajectories of the right and left hip joints in the x, y, and z axes, respectively.

Walking parameters for the half gait cycle (Phase 3, Phase 4, and Phase 5) are shown in Figure 4. As can be seen from the equations generated for the walking trajectory and Figure 4, the gait model can be adjusted by changing three length parameters ($\mathrm{s}$, $\mathrm{h}$, $\mathrm{w}$), a time parameter (T), and a rate parameter ($\mathrm{DSR}$).

2.2. Kinematic Analysis

2.2.1. Forward Kinematics

For the forward kinematics, firstly, the rotation axes of the robot leg are determined, and axes are placed on the joints. The yaw joint in the hip of the robot is not included in the kinematic analysis as it does not affect the walking trajectory for a straight walking task. The lengths of the leg links and placement of the axes are shown in Figure 5.

In Figure 6, the representation of the leg joint angles is given.

The D-H table for the leg of the Robotis-OP2 robot is created as given in

Table 1. D-H table of Robotis-OP2 robot leg.

Axis	$\mathit{\theta }$ (rad)	d (mm)	a (mm)	$\mathit{\alpha }$ (rad)
1	${\mathsf{\theta }}_1$	0	0	−π/2
2	${\mathsf{\theta }}_2$	0	93	0
3	${\mathsf{\theta }}_3$	0	93	0
4	${\mathsf{\theta }}_4$	0	0	π/2
5	${\mathsf{\theta }}_5$	0	0	0

.

Homogeneous transformation matrices (Equation (A1)) of the leg, which determine the orientation and position of a joint relative to the previous joint, are obtained by using the parameters in the D-H table given in Table 1.

The transformation matrix according to the ankle joint, where the main coordinate system of the Robotis-OP2 hip joint is located, is obtained as in Equation (A2).

In the ${\mathrm{T}}_0^5$ matrix given in Equation (A2), ${\left(\mathrm{n}, \mathrm{o}, a\right)}_{\mathrm{x},\mathrm{y},\mathrm{z}}$ represents the orientation of the hip joint, which indicates the angle of rotation of one coordinate system with relation to another coordinate system, and ${\mathrm{p}}_{\mathrm{x}},{ \mathrm{p}}_{\mathrm{y}},{ \mathrm{p}}_{\mathrm{z}}$ represents the x, y, and z position coordinates of the hip joint relative to the ankle joint, respectively.

Equation (A3) is obtained using the homogeneous transformation matrices previously obtained.

${\mathrm{p}}_{\mathrm{x}}$, ${\mathrm{p}}_{\mathrm{y}}$, and ${\mathrm{p}}_{\mathrm{z}}$ are obtained as in Equations (14)–(16), respectively, and forward kinematic analysis is completed.

$${\mathrm{p}}_{\mathrm{x}}=\cos {\mathsf{\theta }}_1\left(93\cos {\mathsf{\theta }}_2+93\cos {\mathsf{\theta }}_2\cos {\mathsf{\theta }}_3-93\sin {\mathsf{\theta }}_2\sin {\mathsf{\theta }}_3\right)$$

(14)

$${\mathrm{p}}_{\mathrm{y}}=\sin {\mathsf{\theta }}_1\left(93\cos {\mathsf{\theta }}_2+93\cos {\mathsf{\theta }}_2\cos {\mathsf{\theta }}_3-93\sin {\mathsf{\theta }}_2\sin {\mathsf{\theta }}_3\right)$$

(15)

$${\mathrm{p}}_{\mathrm{z}}=-93\sin {\mathsf{\theta }}_2-93\cos {\mathsf{\theta }}_2\sin {\mathsf{\theta }}_3-93\cos {\mathsf{\theta }}_3\sin {\mathsf{\theta }}_2$$

(16)

2.2.2. Inverse Kinematics

When the leg structure of Robotis-OP2 is examined, the first three axes cause the displacement of the hip joint, which is the endpoint, while the last two axes change the rotation of the hip joint. Since the equations to be obtained by inverse kinematics may have more than one solution set, the angle limits are chosen considering the joint angle limits of the Robotis-OP2 legs [55] in

Table 2. Joint angle limits of Robotis-OP2 legs [55].

Joint	Limits of Joint Angle (Degree)
	Right Leg		Left Leg		Selected Limits
	min.	max.	min.	max.	min.	max.
$\mathrm{Ankle} \left(\mathrm{roll}\right) {\mathsf{\theta }}_1$	−39	60	−58	34	−58	34
$\mathrm{Ankle} \left(\mathrm{pitch}\right) {\mathsf{\theta }}_2$	−71	78	−79	70	−79	70
$\mathrm{Knee} {\mathsf{\theta }}_3$	0	128	−129	0	0	128
$\mathrm{Hip} \left(\mathrm{pitch}\right) {\mathsf{\theta }}_4$	−101	25	−28	96	−96	25
$\mathrm{Hip} \left(\mathrm{roll}\right){ \mathsf{\theta }}_5$	−57	58	−57	53	−57	53

.

For inverse kinematic analysis, firstly, both sides of the equation in Equation (A2) are multiplied first by ${\mathrm{A}}_1{}^{-1}$, and then by ${\mathrm{A}}_2{}^{-1}{\mathrm{A}}_1{}^{-1}$, and ${\mathsf{\theta }}_1$, ${\mathsf{\theta }}_3$, and ${\mathsf{\theta }}_2$ are obtained, respectively.

${\mathsf{\theta }}_1$ is calculated by Equation (17).

$$\left\{\begin{array}{c}{\mathsf{\theta }}_1={\tan }^{-1}\left(\frac{{\mathrm{p}}_{\mathrm{y}}}{{\mathrm{p}}_{\mathrm{x}}}\right) if within the limit\mathrm{s} of the angle\\ {\mathsf{\theta }}_1={\tan }^{-1}\left(-\frac{{\mathrm{p}}_{\mathrm{y}}}{{\mathrm{p}}_{\mathrm{x}}}\right) el\mathrm{s}e\end{array} \right.$$

(17)

${\mathsf{\theta }}_3$ is calculated as in Equation (20) using Equations (18) and (19).

$${\cos \mathsf{\theta }}_3=\frac{{\mathrm{p}}_{\mathrm{z}}{}^2+{({\mathrm{p}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{p}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1)}^2-{93}^2-{93}^2}{2{(93)}^2}$$

(18)

$${\sin \mathsf{\theta }}_3=\sqrt{1-{\cos }^2{\mathsf{\theta }}_3}$$

(19)

$$\left\{\begin{array}{c}{\mathsf{\theta }}_3={\tan }^{-1}\left(\frac{{\sin \mathsf{\theta }}_3}{{\cos \mathsf{\theta }}_3}\right) if within the limit\mathrm{s} of the angle\\ {\mathsf{\theta }}_3={\tan }^{-1}\left(\frac{-{\sin \mathsf{\theta }}_3}{{\cos \mathsf{\theta }}_3}\right) el\mathrm{s}e\end{array} \right.$$

(20)

${\mathsf{\theta }}_2$ is calculated as in Equation (23) using Equations (21) and (22).

$${\sin \mathsf{\theta }}_2=\frac{-({\mathrm{p}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{p}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1)93{\sin \mathsf{\theta }}_3-\left(93+93{\cos \mathsf{\theta }}_3\right){\mathrm{p}}_{\mathrm{z}}}{{\left(-93{\sin \mathsf{\theta }}_3\right)}^2-{\left(93+93{\cos \mathsf{\theta }}_3\right)}^2}$$

(21)

$${\cos \mathsf{\theta }}_2=\frac{-{\mathrm{p}}_{\mathrm{z}}93{\sin \mathsf{\theta }}_3+\left({\mathrm{p}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{p}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\left(93+93{\cos \mathsf{\theta }}_3\right)}{{\left(93+93{\cos \mathsf{\theta }}_3\right)}^2+{\left(-93{\sin \mathsf{\theta }}_3\right)}^2}$$

(22)

$$\left\{\begin{array}{c}{\mathsf{\theta }}_2={\tan }^{-1}\left(\frac{{\sin \mathsf{\theta }}_2}{{\cos \mathsf{\theta }}_2}\right) if within the limit\mathrm{s} of the angle\\ {\mathsf{\theta }}_2={\tan }^{-1}\left(-\frac{{\sin \mathsf{\theta }}_2}{{\cos \mathsf{\theta }}_2}\right) el\mathrm{s}e\end{array} \right.$$

(23)

The equation in Equation (A2) is multiplied by ${\mathrm{A}}_3{}^{-1} {\mathrm{A}}_2{}^{-1} {\mathrm{A}}_1{}^{-1}$. The rotation matrix representing pitch used in obtaining ${\mathsf{\theta }}_4$ and ${\mathsf{\theta }}_5$ angles is given in Equation (24).

$$\mathrm{R}\left(\mathsf{\theta }\right)=\left[\begin{array}{ccc}\cos \mathsf{\theta }& 0& \sin \mathsf{\theta }\\ 0& 1& 0\\ -\sin \mathsf{\theta }& 0& \cos \mathsf{\theta }\end{array}\right]=\left[\begin{array}{ccc}{\mathrm{n}}_{\mathrm{x}}& {\mathrm{o}}_{\mathrm{x}}& a_{\mathrm{x}}\\ {\mathrm{n}}_{\mathrm{y}}& {\mathrm{o}}_{\mathrm{y}}& a_{\mathrm{y}}\\ {\mathrm{n}}_{\mathrm{z}}& {\mathrm{o}}_{\mathrm{z}}& a_{\mathrm{z}}\end{array}\right]$$

(24)

${\mathsf{\theta }}_4$ is calculated as in Equation (25).

$$\left\{\begin{array}{c}{\mathsf{\theta }}_4={\tan }^{-1}\left(\frac{{\sin \mathsf{\theta }}_3\left({\mathrm{n}}_{\mathrm{z}}{\sin \mathsf{\theta }}_2-{cos\mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)-{\cos \mathsf{\theta }}_3\left({\mathrm{n}}_{\mathrm{z}}{\cos \mathsf{\theta }}_2+{\sin \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)}{{\cos \mathsf{\theta }}_3\left(-{\mathrm{n}}_{\mathrm{z}}{\sin \mathsf{\theta }}_2+{\cos \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)-{\sin \mathsf{\theta }}_3({\mathrm{n}}_{\mathrm{z}}{\cos \mathsf{\theta }}_2+{\sin \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)}\right)if within the limit\mathrm{s} of the angle\\ {\mathsf{\theta }}_4={\tan }^{-1}\left(-\frac{{\sin \mathsf{\theta }}_3\left({\mathrm{n}}_{\mathrm{z}}{\sin \mathsf{\theta }}_2-{\cos \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)-{\cos \mathsf{\theta }}_3\left({\mathrm{n}}_{\mathrm{z}}{\cos \mathsf{\theta }}_2+{\sin \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)}{{\cos \mathsf{\theta }}_3\left(-{\mathrm{n}}_{\mathrm{z}}{\sin \mathsf{\theta }}_2+{\cos \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)\right)-{\sin \mathsf{\theta }}_3({\mathrm{n}}_{\mathrm{z}}{\cos \mathsf{\theta }}_2+{\sin \mathsf{\theta }}_2\left({\mathrm{n}}_{\mathrm{x}}{\cos \mathsf{\theta }}_1+{\mathrm{n}}_{\mathrm{y}}{\sin \mathsf{\theta }}_1\right)}\right)el\mathrm{s}e\end{array} \right.$$

(25)

${\mathsf{\theta }}_5$ is calculated as in Equation (26).

$$\left\{\begin{array}{c}{\mathsf{\theta }}_5={\tan }^{-1}\left(\frac{{\mathrm{n}}_{\mathrm{y}}{\cos \mathsf{\theta }}_1-{\mathrm{n}}_{\mathrm{x}}{\sin \mathsf{\theta }}_1}{{\mathrm{o}}_{\mathrm{y}}{\cos \mathsf{\theta }}_1-{\mathrm{o}}_{\mathrm{x}}{\sin \mathsf{\theta }}_1}\right) if within the limit\mathrm{s} of the angle\\ {\mathsf{\theta }}_5={\tan }^{-1}\left(-\frac{{\mathrm{n}}_{\mathrm{y}}{\cos \mathsf{\theta }}_1-{\mathrm{n}}_{\mathrm{x}}{\sin \mathsf{\theta }}_1}{{\mathrm{o}}_{\mathrm{y}}{\cos \mathsf{\theta }}_1-{\mathrm{o}}_{\mathrm{x}}{\sin \mathsf{\theta }}_1}\right)el\mathrm{s}e\end{array} \right.$$

(26)

With the rotation matrix in Equation (24), the $\mathsf{\theta }$ value, which is the robot’s body tilt angle in the sagittal plane, is adjusted. The $\mathsf{\theta }$ shown in Figure 7 is equal to the sum of the pitch angles of the leg (${\mathsf{\theta }}_2$ + ${\mathsf{\theta }}_3$ + ${\mathsf{\theta }}_4$) and can be selected at desired values so as not to cause the robot to fall while walking.

Using the generated walking trajectory, the position coordinates ${\mathrm{p}}_{\mathrm{x}}$, ${\mathrm{p}}_{\mathrm{y}}$, and ${\mathrm{p}}_{\mathrm{z}}$ of the hip joint relative to the ankle joint are calculated for both legs. The calculation operations consist of subtracting the ankle joint’s position from the position of the hip joint at each time step.

3. Orientation Angles and Gait Stability Criterion

With an Inertial Measurement Unit (IMU), the three-dimensional orientation of an object in space can be found [56]. The ZMP is where the sum of weight and vertical inertial forces equals zero. When the robot maintains at least one flat foot on the ground while walking and has active ankle joints, the ZMP may be employed as a gait stability criterion.

3.1. Calculation of Orientation Angles

The gyroscope and accelerometer are embedded in the CM-740 controller board, which is the sub-controller of Robotis-OP2. Raw accelerometer and gyroscope data are read from the controller board. The IMU consists of a three-axis ADXL335 accelerometer with an analog output and a three-axis LYPR540AH gyroscope with an analog output.

Considering Robotis-OP2’s straight gait in the sagittal plane, the body tilt angle is the pitch angle. Regarding the coordinate framework, the positive pitch angle is counterclockwise, and the positive roll angle is clockwise. Figure 8 shows the pitch and roll angles of the body.

The gyroscope measures the angular velocities ${\mathsf{\omega }}_{\mathrm{x}}$, ${\mathsf{\omega }}_{\mathrm{y}}$, and ${\mathsf{\omega }}_{\mathrm{z}}$ along the x, y, and z axes, respectively, while the accelerometer measures the linear accelerations ${\mathrm{a}}_{\mathrm{x}}$, ${\mathrm{a}}_{\mathrm{y}}$, and ${\mathrm{a}}_{\mathrm{z}}$ along the x, y, and z axes, respectively. However, the data from the sensors are raw data. These data are values read from the 10-bit ADC unit of the CM-740 sub-controller.

Table 3. Parameters of accelerometer and gyroscope.

Parameter	Accelerometer	Gyroscope
${\mathrm{ADC}}_{\mathrm{value}}$	0~1023	0~1023
${\mathrm{V}}_{\mathrm{ref}}$	3.3 V	3.3 V
Bit Rate	1024	1024
${\mathrm{V}}_{\mathrm{zero}}$	1.65 V	1.65 V
Sensitivity	0.33 V/g	0.0008 V/(degree/s)
R	$\mathrm{a}$	$\mathsf{\omega }$

shows the parameters of the accelerometer, which can measure acceleration in the range of ± 3g in three axes, and the gyroscope, which can measure angular velocity in the range of ±1600 degrees/s in three axes.

The raw data from the accelerometer and gyroscope are converted into physical units expressing g (9.8 $\mathrm{m}/{\mathrm{s}}^2$) in terms of the magnitude of the gravity for the accelerometer and degrees/s for the gyroscope using the equation in Equation (27).

$$\mathrm{R}=\left(\frac{{\mathrm{ADC}}_{\mathrm{value}}\times {\mathrm{V}}_{\mathrm{ref}} }{\mathrm{BitRate}}-{\mathrm{V}}_{\mathrm{zero}}\right)\times \frac{1}{\mathrm{Sensitivity}}$$

(27)

The acceleration values are converted to pitch $\left(\mathsf{\theta }\right)$ and roll $\left(\mathsf{\Phi }\right)$ angles by Equation (A4). The pitch $\left(\mathsf{\theta }\right)$ and roll $\left(\mathsf{\Phi }\right)$ angles are obtained using angular velocity values by Equation (A5).

Although noisy in the short term, the accelerometer offers reliable data over the long run. Angle shifts happen over lengthy time scales, but the gyroscope is stable and delivers precise information on changing orientation in the short term. To calculate the pitch and roll angles of the humanoid robot body properly, accelerometer and gyroscope data from the IMU are combined to eliminate errors. While the Kalman filter [57] has several parameters that need to be adjusted, the complementary filter is easier to set up as the complementary filter has only one filter parameter that needs to be adjusted. However, in our experimental studies on the complementary filter and the Kalman filter, the Kalman filter gives more accurate results than the complementary filter under dynamic effects. For this reason, the use of the Kalman filter, which is widely preferred for IMUs, is used as a sensor fusion method, and is preferred in this study.

3.2. Calculation of Gait Stability Criterion

In bipedal gait analysis using force or pressure measurements, the Center of Pressure (CoP) [58] is also frequently utilized. Figure 9a illustrates how the system is stable and the foot’s CoP corresponds with the ZMP when the ZMP is inside the support polygon in the single support phase. When the instantaneous ZMP in Figure 9b is at or beyond the support polygon, it means that an imbalanced moment M has arisen and cannot be corrected by the foot response forces R. In this paper, it is proposed to create a stable walking framework with higher stability by using the ZMP of the robot to take into account the dynamic balance.

Force Sensitive Resistors (FSRs) can be used to calculate the ZMP by providing measurements of the robot’s vertical ground reaction force and the moment of the ankle joint about its axis of rotation. In this paper, the feet of the Robotis-OP2 humanoid robot are replaced with the FSR set.

The placement of FSRs on the sole in Robotis-OP2 and the numbering of forces are given in Figure 10a. The starting point is defined in the coordinate system (0,0) of the ZMP regarding the center of the left ankle. In the Webots simulation environment, ZMPs are placed according to their dimensions as in Figure 10b.

When the instantaneous measurements of the force values measured with FSRs are examined, it is seen that they have noise. Since the mass of the robot is 3 kg, the sum of the force values measured from the FSRs should be equal to the approx. weight force of the robot, 29.43 N, from the formula W = mg. Since the force values measured from the FSRs have noise, it has been seen that their sum is higher than it should be. In this paper, a one-dimensional Kalman filter [59], frequently used in scientific literature, is applied to remove the noise in the measured force values. The one-dimensional Kalman filter is a state estimation method based on a single measurement parameter.

For the walking trajectory, the forward–backward direction represents the x-axis, and the right–left direction represents the y-axis. Therefore, the ZMP coordinates are calculated for the x and y axes. During a gait that includes single and double support phases, there are three statuses of the right and left feet relative to each other. These are statuses where the feet are on a line, the left foot is forward, and the right foot is forward. When Figure 10a is examined, the distances of the FSRs to each other on the y-axis remain constant during walking. Therefore, the y-axis coordinate of the ZMP (${\mathrm{ZMP}}_{\mathrm{y}}$) is calculated by the Equation (28) in all three statuses.

$${\mathrm{ZMP}}_{\mathrm{y}}=\frac{59.5\left({\mathrm{F}}_5+{\mathrm{F}}_7\right)+109.5\left({\mathrm{F}}_6+{\mathrm{F}}_8\right)+14.5\left({\mathrm{F}}_2+{\mathrm{F}}_4\right)-35.5\left({\mathrm{F}}_1+{\mathrm{F}}_3\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^8{\mathrm{F}}_{\mathrm{i}}}$$

(28)

The x-axis coordinate of the ZMP (${\mathrm{ZMP}}_{\mathrm{x}}$) is calculated by the equation in Equation (29) when the feet are on a line.

$${\mathrm{ZMP}}_{\mathrm{x}}=\frac{41\left({\mathrm{F}}_1+{\mathrm{F}}_2\right)+41\left({\mathrm{F}}_5+{\mathrm{F}}_6\right)-41\left({\mathrm{F}}_3+{\mathrm{F}}_4\right)-41\left({\mathrm{F}}_7+{\mathrm{F}}_8\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^8{\mathrm{F}}_{\mathrm{i}}}$$

(29)

${\mathrm{ZMP}}_{\mathrm{x}}$ is calculated by the equation in Equation (30) by being affected by translation as ${\mathrm{x}}_{\mathrm{left}}-{\mathrm{x}}_{\mathrm{right}}$ when the left foot is forward.

$${\mathrm{ZMP}}_{\mathrm{x}}=\frac{41\left({\mathrm{F}}_1+{\mathrm{F}}_2\right)+\left(41-({\mathrm{x}}_{\mathrm{left}}-{\mathrm{x}}_{\mathrm{right}}\right))\left({\mathrm{F}}_5+{\mathrm{F}}_6\right)-41\left({\mathrm{F}}_3+{\mathrm{F}}_4\right)-\left(41+({\mathrm{x}}_{\mathrm{left}}-{\mathrm{x}}_{\mathrm{right}}\right))\left({\mathrm{F}}_7+{\mathrm{F}}_8\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^8{\mathrm{F}}_{\mathrm{i}}}$$

(30)

${\mathrm{ZMP}}_{\mathrm{x}}$ is calculated by the equation in Equation (31) by being affected by translation as ${\mathrm{x}}_{\mathrm{right}}-{\mathrm{x}}_{\mathrm{left}}$ when the right foot is forward.

$${\mathrm{ZMP}}_{\mathrm{x}}=\frac{41\left({\mathrm{F}}_1+{\mathrm{F}}_2\right)+\left(41+({\mathrm{x}}_{\mathrm{right}}-{\mathrm{x}}_{\mathrm{left}}\right))\left({\mathrm{F}}_5+{\mathrm{F}}_6\right)-41\left({\mathrm{F}}_3+{\mathrm{F}}_4\right)-\left(41-({\mathrm{x}}_{\mathrm{right}}-{\mathrm{x}}_{\mathrm{left}}\right))\left({\mathrm{F}}_7+{\mathrm{F}}_8\right)}{{{\displaystyle \sum }}_{\mathrm{i}=1}^8{\mathrm{F}}_{\mathrm{i}}}$$

(31)

4. Proposed Framework of Gait Parameter Optimization

The main purpose of this paper is to ensure that the Robotis-OP2 humanoid robot walks stably without falling. The most important task for this purpose is to obtain the most suitable gait parameters of the trajectories generated with the walking pattern generator. The proposed framework for stable walking is given in Figure 11.

D3QN, one of the DRL algorithms, is used for the training of walking parameters to perform the straight walking process stably.

There are six walking parameters $\left({\mathrm{h}}_{\mathrm{b}}, \mathrm{w},\mathrm{s},\mathrm{h},\mathrm{T},\mathrm{DSR}\right)$ in the hip and ankle trajectories generated for bipedal walking. Using any DRL algorithm to optimize all parameters makes convergence very difficult as the size of the action space is very large. Therefore, the action space should be simplified after determining the values that the parameters can take, taking into account the kinematic structure of the robot. The limits of the walking parameters shown in Figure 12 are approximately determined as a result of kinematic calculations and observations in Webots.

When the robot walks steadily, the CoM height is constant, as seen from the hip trajectory. However, it can be in a negligible small range of variation due to noise during movement. To decrease the robot’s energy usage without bending the knee joint too much, ${\mathrm{h}}_{\mathrm{b}}=25$ mm was determined as a constant. In addition, gravity creates unwanted torques on the robot’s joints. The effect of gravity becomes more explicit when the robot switches between a double support and a single support during walking. The robot weight, supported by feet, is suddenly supported only by the support foot. In this paper’s scope, emphasis is placed on the double support phase in the eight-phase gait cycle generated for a stable gait. The DSR parameter cannot, therefore, be selected too low. The walking speed must be low so that the inertia forces are negligible in the gait cycle, which is created similarly to quasi-static walking. Therefore, the walking cycle time was chosen as $\mathrm{T}=2$ s.

4.1. Training with Dueling Double Deep Q Network

First of all, methods such as the DQN and DDQN are examined in our studies, and the direct use of the Dueling Double Deep Q Network (D3QN) is adopted without wasting time to overcome the problem of overestimating the state values of these methods. The D3QN has two improvements to solve the problem of state value overestimation. First, the D3QN uses a dual network structure that uses a $Q_1$ evaluation network to select an action for the next step with $a_{max}=\arg \underset{a}{\max }{Q}_1\left(s^{\prime },a;\theta \right)$. Then, $a_{max}$ is evaluated with Equation (32) by a target network $Q_2$ to reduce the overestimation.

$$\left\{\begin{array}{c}{a}_{max}=\arg \underset{a}{\max }{Q}_1(s^{\prime },a;\theta )\\ y=r+\gamma Q_2\left(s^{\prime },a_{max};{\theta }^{-}\right)\end{array} \right.$$

(32)

Second, the D3QN divides the action-value function $Q_{\pi }\left(s, a\right)$ into two parts as in Equation (33): a state-value function $V_{\pi }\left(s\right)$ related to state and an advantage function $A_{\pi }\left(s,a\right)$ regarding both state and action.

$$Q_{\pi }\left(s,a;\theta ,\alpha ,\beta \right)=V_{\pi }\left(s;\theta ,\alpha \right)+A_{\pi }\left(s,a;\theta ,\beta \right)$$

(33)

where $\theta$ is the common parameter of $V_{\pi }$ and $A_{\pi }\left(s,a\right)$, $\alpha$ and $\beta$ are the parameters of $V_{\pi }\left(s\right)$ and $A_{\pi }\left(s,a\right)$, respectively.

The D3QN structure used is shown in Figure 13.

For stable humanoid walking implementation, the D3QN learns the optimal policy by performing Algorithm 1.

Algorithm 1 D3QN

Initialize a stable humanoid gait system (Environment)

Initialize $\mathcal{D}$ repeat buffer with $M$ capacity

Initialize ${\mathcal{Q}}_1$ Evaluation Network with random $\theta$ parameters

Initialize ${\mathcal{Q}}_2$ Target Network with ${\theta }^{-}\leftarrow \theta$ parameters

Refill buffer with data generated by random policy

while$True$do

Restart stable humanoid walking system

while $No termination$ do

Choose a random action $a$ with probability $ϵ$, otherwise choose action $a=\arg \underset{a}{\max }{Q}_1\left(s_t, a;\theta \right)$

Perform action $a$, get reward $r$ and get next state $s^{\prime }$

Save experience tuple 〈$\mathrm{s},a,r,s^{\prime }$〉 to $\mathcal{D}$

Sample mini batch from $\mathcal{D}$

if $there i\mathrm{s} a termination at \mathrm{s}tep j+1$ then

$Y_j=r_j$

else

$a_{max}=\arg \underset{a}{\max }{Q}_1\left(s^{\prime },a;\theta \right)$

$Y_j=\gamma \underset{a^{\prime }}{\max }{Q}_2(s^{\prime },a_{max};{\theta }^{-})$

${\widehat{Y}}_j=Q_1\left(s,a;\theta \right)$

Minimize loss to train $Q_1$ for updating $\theta$

Update Target Network at every $C$ steps: ${\theta }^{-}\leftarrow \theta$

if$Q_1$ converges then

$Q_2=Q_1$

$Q_{optimal}=Q_2$

break

4.1.1. The Architecture of D3QN

The layers, the number of neurons, and the activation functions of the D3QN, whose structure is given in Figure 13, are given in

Table 4. The layer structure of D3QN.

Layer Name	Layer Type	Neuron Numbers	Activation Type
Input	Fully connected	9	-
Common FC	Fully connected	512	ReLU
Common FC	Fully connected	256	ReLU
FC1 for Value	Fully connected	128	ReLU
FC1 for Advantage	Fully connected	128	ReLU
FC2 for Value	Fully connected	1	Linear
FC2 for Advantage	Fully connected	14,400	Linear
Output	Fully connected	14,400	-

. ANN structures with hidden layer neurons such as 128, 1024, and 2048 are tried, but the best result is obtained from the ANN designed with the number of neurons in Table 4.

The training phase includes 30 thousand episodes using the Adam optimizer, and each episode consists of seven steps. The D3QN algorithm was created in such a way that the robot’s walking for 6T time was taken into account for each training step. In each step performed in the episode, the robot takes an action and performs its movement. If the robot can take 12 steps without falling, it receives a reward and the next step of the episode is passed from where it left off. However, if the robot falls at any time during the walking or has an inverse kinematics calculation error, it cannot receive a reward and is punished directly. Next, the environment is reset, and the next episode begins.

The hyperparameters selected for training the D3QN are listed in

Table 5. The layer structure of D3QN. The selected hyperparameters for D3QN.

Hyperparameter	Value
Mini batch size	32
Replay memory size	15,000
$\mathrm{Discount} \mathrm{factor} \gamma$	0.95
Learning rate	0.0001
$\mathrm{Initial} \mathrm{exploration} \mathrm{rate} ϵ$$\mathrm{Min}. \mathrm{exploration} \mathrm{rate} {ϵ}_{min}$	1 0.01

.

4.1.2. State Space

The state space consists of a nine-dimensional array containing states that are important for the stable walking task as given in Table 6.

Table 6. State space.

State	Unit	Size
$\mathrm{Orientation} \mathrm{angles} (\mathrm{pitch} (\mathsf{\theta }$$) \mathrm{and} \mathrm{roll} (\mathsf{\Phi }$))	deg.	2
$\mathrm{Position} ({\mathrm{P}}_{\mathrm{x}},{ \mathrm{P}}_{\mathrm{y}},{ \mathrm{P}}_{\mathrm{z}}$)	mm	3
$\mathrm{Rotation} \mathrm{in} \mathrm{the} \mathrm{z}-\mathrm{axis} ({\mathrm{R}}_{\mathrm{z}}$)	rad.	1
$\mathrm{Zero} \mathrm{Moment} \mathrm{Point} ({\mathrm{ZMP}}_{\mathrm{x}}{ \mathrm{and} \mathrm{ZMP}}_{\mathrm{y}}$)	mm	2
$\mathrm{Distance} \mathrm{taken} \mathrm{in} \mathrm{the} \mathrm{forward} \mathrm{direction} ({\mathrm{D}}_{\mathrm{x}}$)	mm	1

Table 6. State space.

State	Unit	Size
$\mathrm{Orientation} \mathrm{angles} (\mathrm{pitch} (\mathsf{\theta }$$) \mathrm{and} \mathrm{roll} (\mathsf{\Phi }$))	deg.	2
$\mathrm{Position} ({\mathrm{P}}_{\mathrm{x}},{ \mathrm{P}}_{\mathrm{y}},{ \mathrm{P}}_{\mathrm{z}}$)	mm	3
$\mathrm{Rotation} \mathrm{in} \mathrm{the} \mathrm{z}-\mathrm{axis} ({\mathrm{R}}_{\mathrm{z}}$)	rad.	1
$\mathrm{Zero} \mathrm{Moment} \mathrm{Point} ({\mathrm{ZMP}}_{\mathrm{x}}{ \mathrm{and} \mathrm{ZMP}}_{\mathrm{y}}$)	mm	2
$\mathrm{Distance} \mathrm{taken} \mathrm{in} \mathrm{the} \mathrm{forward} \mathrm{direction} ({\mathrm{D}}_{\mathrm{x}}$)	mm	1

where $\mathsf{\theta }$ and $\mathsf{\Phi }$ are the pitch and roll angles of the robot body calculated using the Kalman filter, respectively; ${\mathrm{P}}_{\mathrm{x}}$, ${\mathrm{P}}_{\mathrm{y}}$, and ${\mathrm{P}}_{\mathrm{z}}$ are the positions of the robot in the x, y, and z axes, respectively; ${\mathrm{R}}_{\mathrm{z}}$ is the rotation of the robot in the z-axis; ${\mathrm{ZMP}}_{\mathrm{x}}$ and ${\mathrm{ZMP}}_{\mathrm{y}}$ are the ZMPs in the x and y axes, respectively; and ${\mathrm{D}}_{\mathrm{x}}$ is the forward displacement distance of the robot.

4.1.3. Action Space

In the rotation matrix of inverse kinematics equations, the range of ${\mathsf{\theta }}_{\mathrm{r}}$, which determines the pitch angle $\mathsf{\theta }$ of the robot, shown in Figure 12, is not selected at high values to be close to an upright walk. ${\mathsf{\theta }}_{\mathrm{r}}$ is added to the action space as the fifth walking parameter. The ranges of the five walking parameters are discretized as in

Table 7. Discrete values of parameters in the action space.

Parameter	Discrete Values
$\mathrm{w}$	[25 30 35 40 45 50 55 60 65 70]
$\mathrm{s}$	[25 30 35 40 45 50 55 60 65 70 75 80]
$\mathrm{h}$	[25 30 35 40 45 50]
DSR	[0.3 0.4 0.5 0.6]
${\mathsf{\theta }}_{\mathrm{r}}$	[−3 −8 −13 −18 −23]

, which can be meaningful for stable walking.

As can be seen in Table 7, the action space has a total dimension of 10 × 12 × 6 × 4 × 5 = 14,400 consisting of $\mathrm{w}$, $\mathrm{s}$, $\mathrm{h}$, $\mathrm{DSR}$, and ${\mathsf{\theta }}_{\mathrm{r}}$ combinations.

4.1.4. Reward Function

R used to optimize walking parameters is expressed by the reward function (34).

$$\mathrm{R}=10{\mathrm{D}}_{\mathrm{x}}+3{\mathrm{S}}_{\mathrm{d}}+6{\mathrm{S}}_{\mathrm{b}}-5{\mathrm{S}}_{\mathrm{h}}$$

(34)

where ${\mathrm{D}}_{\mathrm{x}}$ is the distance the robot takes in the forward direction (mm), ${\mathrm{S}}_{\mathrm{d}}$ is the counter that determines the scale of the robot in walking straight without deviating (the counter increases as the robot walks under the given threshold deviation (±0.01 rad)), ${\mathrm{S}}_{\mathrm{b}}$ is the counter that expresses the robot’s stable walking (the counter increases when the ZMP on the x and y axes is within the support polygon of the robot), and ${\mathrm{S}}_{\mathrm{h}}$ is the counter that measures the error in the ZMP calculation (when the robot’s feet do not touch the ground due to disruptive effects, ZMP cannot be calculated because information cannot be obtained from the FSRs).

The reward is determined as −100 when the robot falls or when an inverse kinematics calculation error is encountered during the robot’s movement.

4.2. Body Posture Balancing

Depending on the circumstance, maintaining balance after a walking impairment can be a challenging procedure, including many solutions. Three fundamental strategies—the ankle, hip, and stepping strategies—have been identified as a result of experimental research on this process in humans. These three balancing strategies, shown in Figure 14, are also adopted in humanoid robots. In this paper, a balancing method consisting of a PID controller is proposed so that the Robotis-OP2 robot can cope with robotic uncertainties and walk while maintaining balance. The use of the hip strategy is preferred for balancing. The main reason for choosing the hip strategy is that it directly affects body orientation, as the hip is the connecting piece between the legs and the body. In addition, it is more resistant to disruptive effects than the ankle strategy, and the angle of inclination of the body can be controlled with a single parameter because the hip is chosen as the end joint when creating kinematic equations for walking.

The studies have been conducted on a real-time closed-loop control of the Robotis-OP2’s pitch orientation for body stabilization control during walking. The block diagram of the proposed structure based on the feedback of the IMU is given in Figure 15.

5. Experimental Results

First, experimental studies were carried out to determine the parameters used to calculate the orientation angles and gait stability criterion. The accelerometer, bias, and measurement noise variances of the Kalman filter were determined as 0.001, 0.003, and 0.03, respectively, by taking a time step of 16 ms. Initial values of the process noise, measurement noise, and estimated error variances of the one-dimensional Kalman filter were determined as 0.001, 0.25, and 1, respectively. Moreover, by default, only the P controller with a gain coefficient ${\mathrm{K}}_{\mathrm{p}}$ of 10 is used for the position control of the servo motor in the Webots. In the experimental studies, proportional gain ${\mathrm{K}}_{\mathrm{p}}$ and integral gain ${\mathrm{K}}_{\mathrm{i}}$ coefficients were determined as 30 and 0.9, respectively, by trial and error.

As shown in Figure 16, experimental studies were carried out on a desktop computer with NVIDIA GTX Titan X Pascal GPU, Intel i5 4th generation 3.4 GHz processor, and 8 GB RAM. The training of the D3QN was carried out in the Webots simulator with the controller written using Tensorflow 2.2.0, Keras 2.3.1, and Python 3.8 versions by utilizing the power of parallel computing thanks to the CUDA support of the GPU.

During the training process, the rewards received in each episode were recorded. Weights were stored in every 100 episodes. After the training was completed, the average reward obtained by taking the average of the last 40 episodes was drawn, and it was observed that a highly fluctuating graph appeared. This situation is because training includes too many episodes. The graph obtained by taking the average of the last 200 episodes instead of the last 40 episodes is obtained in Figure 17.

When the training results in Figure 17 are examined, it is seen that the average reward increases exponentially up to the roughly 16,000th episode. Afterwards, the average reward remained within a specific range. During the training, it has been observed that it converges to the parameters that can perform many successful walking processes. However, action sequences consisting of the actions in

Table 8. Optimal gait parameters obtained as a result of training.

Parameter	Value	Unit
$\mathrm{w}$	50	mm
$\mathrm{s}$	60	mm
$\mathrm{h}$	30	mm
DSR	0.6	-
${\mathsf{\theta }}_{\mathrm{r}}$	−13	degree

were obtained repeatedly in at least the last 9000 episodes.

With the values in Table 8, ${\mathrm{D}}_{\mathrm{x}}$, ${\mathrm{S}}_{\mathrm{d}}$, ${\mathrm{S}}_{\mathrm{b}}$, and ${\mathrm{S}}_{\mathrm{h}}$ for six walking cycles (6T) were obtained as 709.2 mm, 711, 581, and 0, respectively, and the reward value calculated by Equation (34) was obtained as 12,711. The saved weights were loaded and tested in the Webots R2020b simulator. The walking trajectories obtained with the optimized gait parameters were visualized with graphs in MATLAB R2021b for the gait cycle. The three-dimensional representation of the walking model is given in Figure 18.

When Figure 18 is examined, the robot’s speed is 60 mm/s in the x-axis, and the robot’s hip performs a swinging movement of a maximum of 50 mm. In addition, while walking, the robot’s foot rises a maximum of 30 mm from the ground.

The ankle and hip joint trajectories for the right and left legs, along with their phases, are shown in Figure 19. The position coordinates of the hip joint relative to the ankle joint ${\mathrm{p}}_{\mathrm{x}}$, ${\mathrm{p}}_{\mathrm{y}}$, and ${\mathrm{p}}_{\mathrm{z}}$ were calculated for both legs by the walking trajectory. The calculation process consists of subtracting the ankle joint’s position from the hip joint’s position at each time step.

Although there was no disturbance on the ground, the results of straight walking were obtained using the same walking parameters (Table 8) for the four walking cycles with the closed loop. The gain coefficients ${\mathrm{K}}_{\mathrm{p}}$, ${\mathrm{K}}_{\mathrm{i}}$, and ${\mathrm{K}}_{\mathrm{d}}$ obtained by trial and error for the PID controller for body posture balancing were determined as 0.8, 7.1, and 0.01, respectively.

According to Figure 19a, although the hip trajectory in the x-axis is the same for the right and left legs, it has mirrored oscillations for the ankle. According to Figure 19b, the ankle and hip trajectories are the same for both legs. According to Figure 19c, although the hip trajectory is the same for the right and left legs, the ankle trajectory differs.

The walking algorithm inside the Robotis-OP2 uses a method to construct a gait model based on CPG-based paired oscillators that perform sinusoidal trajectories [60]. Many parameters are available in the algorithm to adjust the gait. However, it has been decided that only a subset of these parameters can be set to facilitate its use by the user [55]. Other parameters are set to default values that are known to work well. The robot’s gait can be adjusted if the user wishes by changing the default values stored in the “config.ini” configuration file shown in Figure 20 [55].

The results for walking obtained with the proposed framework and Robotis-OP2’s walking algorithm [60] are compared as in Figure 21 and Figure 22.

When Figure 21a is observed, at the beginning of the walking, an instantaneous jumping movement occurred in the time it took for the system response with the PID controller to settle on the reference input. However, since this jumping is in the first approx. 0.18 s of walking, it is observed that it did not adversely affect the walking stability of the robot. When the oscillations of the orientation angles in Figure 21 are examined, it is seen that the proposed framework has very little angle change compared to the other. In

Table 9. The ranges of orientation angles for two methods.

Method	Pitch Angle (deg.)		Roll Angle (deg.)
	min.	max.	min.	max.
Proposed framework	−13.446	−12.995	−1.964	2.332
Robot’s walking algorithm	−13.457	−7.126	−10.033	12.233

, the ranges of the robot orientation angles given in Figure 21 are compared.

With the proposed framework according to Table 9, the robot has very little swinging during walking compared to the robot’s walking algorithm for both pitch and roll orientation angles.

When Figure 22a is examined, it is seen that the ${\mathrm{ZMP}}_{\mathrm{x}}$ increases by oscillating more smoothly with the proposed frame compared to the other. Considering the step length is 60 mm, the fact that the robot’s ${\mathrm{ZMP}}_{\mathrm{x}}$ moves on 474 mm in the forward direction during four gait cycles indicates quite stable walking. When Figure 22b is examined, it is seen that, with the proposed framework, ${\mathrm{ZMP}}_{\mathrm{y}}$ oscillates regularly in a certain period, and there is only about 12 mm of vertical misalignment at the end of the walk. The other method deviates considerably from the vertical alignment compared to the proposed framework. These experimental results showed that very stable walking was obtained with the proposed framework. In addition, when all the experimental results were evaluated, it was seen that the proposed framework was quite successful in terms of both the balanced walking and stable walking of the robot. The proposed framework is also very useful for optimizing gait parameters for different state and action spaces of the DRL. The user can optimize the robot’s gait using the proposed framework for the robot’s desired speed and walking pattern generator.

The controller, which was developed after the experimental studies at Webots, was transferred to the real Robotis-OP2 humanoid robot with the remote control tool located in the robot window of Webots. Figure 23 shows the robot’s initial states of walking.

A comparison of the orientation angles of the walking performed with the proposed framework and Robotis-OP2’s walking algorithm is given in Figure 24.

In Figure 24a, it was observed that the oscillation amplitude of the pitch angle was lower. The results in Figure 24b showed that the roll angles of the robot oscillate quite close to each other. Experimental results on the real robot also proved that the robot is more stable during walking.

6. Conclusions and Future Work

In this paper, a robust algorithm is developed with the proposed framework for the stable walking of a humanoid robot. The proposed framework consists of a traditional trajectory generator controller and DRL structure. This study is the first study in the literature on the walking of a humanoid robot using the DRL algorithm, D3QN.

Experimental studies were carried out on the Robotis-OP2 humanoid robot in both simulation and real environments. For experimental studies, the walking trajectory of the robot was created using cycloid curves. Leg kinematic analyses were performed to obtain the angles of the leg joints so that the robot could reach the desired trajectories. The data obtained from multi-sensors on the robot were made sense of. The training of the D3QN for the optimization of walking parameters was carried out in the Webots simulator. With the system based on the training of the D3QN, the optimum parameters of the walking trajectory were obtained for stable walking. It converged to the optimum parameters approximately in the 21,000th episode.

For the robot body to maintain its balance in the sagittal plane, a stabilization system was created using the hip strategy and PID controller. Experimental results were obtained by performing walking tests by combining the obtained values and the balancing system. When the robot body was asked to walk with a pitch angle of −13° in the sagittal plane, the robot successfully walked with a swing between −13.446° and −12.995°. The body roll angle was between −1.964° and 2.332°. It was observed that the ZMP values in the x and y axes, which are the stable walking criteria of the robot, also have a very smooth oscillation. For walking where the robot has a step of 60 mm per second, over four walking periods, the ${\mathrm{ZMP}}_{\mathrm{x}}$ moved on 474 mm. Moreover, the ${\mathrm{ZMP}}_{\mathrm{y}}$ shifted only 12 mm from the starting coordinate. These experimental results proved that the robot has a stable gait. Experimental results were also compared with the Robotis-OP2 robot’s walking algorithm, and it was seen that the proposed framework had a more stable gait.

Based on the study in this paper, many future studies are planned. Firstly, using the proposed framework in this paper, the robot will be able to walk stably on an unknown inclined ground. The robot’s posture will be adjusted in real-time to ensure that the body pitch angle is at the desired reference value as if it were walking on flat ground. Then, studies will be carried out to enable the robot to walk longer with lower energy consumption. Next, more detailed research will be conducted to understand better the natural bipedal gait mechanisms, including in humans. In addition, it is aimed to develop a gait model for applications to the lower limbs’ exoskeleton for both healthy and disabled people.