An Admittance Parameter Optimization Method Based on Reinforcement Learning for Robot Force Control

Abstract

When a robot performs tasks such as assembly or human–robot interaction, it is inevitable for it to collide with the unknown environment, resulting in potential safety hazards. In order to improve the compliance of robots to cope with unknown environments and enhance their intelligence in contact force-sensitive tasks, this paper proposes an improved admittance force control method, which combines classical adaptive control and machine learning methods to make them use their respective advantages in different stages of training and, ultimately, achieve better performance. In addition, this paper proposes an improved Deep Deterministic Policy Gradient (DDPG)-based optimizer, which is combined with the Gaussian process (GP) model to optimize the admittance parameters. In order to verify the feasibility of the algorithm, simulations and experiments are carried out in MATLAB and on a UR10e robot, respectively. The experimental results show that the algorithm improves the convergence speed by 33% in comparison to the general model-free learning method, and has better control performance and robustness. Finally, the adjustment time required by the algorithm is 44% shorter than that of classical adaptive admittance control.

1. Introduction

As industrial robot applications continue to expand [1,2,3], the operational process of industrial robots has become increasingly independent of human intervention. It is imperative that robots possess the capacity to interact with their environments in a stable and reliable manner. However, in environments with high complexity and uncertainty, it remains challenging to ensure the reliable and accurate interactive control of the robot [4]. The models of commercial robots are highly precise [5], but the precise modeling of the environment is usually not available. The occurrence of sudden changes in the environment can result in the generation of unpredictable deviations in the interaction.

Compliance control is widely used in robot interaction control [6]. This kind of method also pays attention to contact force tracking while performing position control. Common compliance control methods can be roughly classified as hybrid position/force control [7] and impedance control [8]. Position-based impedance control is also known as admittance control. The accuracy of hybrid position/force control is reduced because it ignores the dynamic coupling between the robot and the environment [9]. Impedance control establishes the dynamic relationship between robot motion and contact force, and can achieve very accurate control with simple modeling.

Adaptive control can effectively deal with the complex situation of uncertainty [10]. Adaptive impedance control is an extension of classical impedance control. It continuously adjusts the impedance parameters to adapt to dynamic changes through feedback information, showing higher flexibility and robustness. So far, there have been many research results on such algorithms [11,12,13,14].

With the rapid development of artificial intelligence technology, machine learning methods are gradually being applied to robot force control technology. Compared with classical control, intelligent control is more inclusive of the unknowns in the model. In the early days, imitation learning method was used to build the model by collecting the demonstration experience of human experts [15,16,17,18,19]. A neural network was used for parameter estimation [20], contact force prediction [21] and system unknowns prediction [22,23]. Imitation learning was used to improve the portability of the controller [24]. In recent years, neural networks have received more attention, and their application to variable impedance control has also been widely developed [25,26,27,28].

Reinforcement learning (RL) is another machine learning technique. Unlike deep learning, which relies on a large amount of prior knowledge, reinforcement learning is semi-supervised learning. It accumulates experience through repeated attempts to achieve autonomous learning. In [29], the prior knowledge of the operator is introduced, and the reinforcement learning, fuzzy technology, and neural network are combined to propose a reinforcement learning fuzzy neural network architecture based on the actor–critic architecture, which is used to learn the parameters of the variable admittance controller. In [30], the artificial neural network is used to establish the dynamics of human–computer interaction, and the model is used for reinforcement learning, combining a cross-entropy method and model prediction control. In [31], a policy iteration with constraint embedding is proposed to ensure the semi-definiteness of the policy evaluation function and to simplify its form. In [32], the admittance parameters are directly updated through Q-learning. In [33], a reinforcement learning variable impedance control with stability guarantee is proposed. The extended cross-entropy reinforcement learning method is trained to make direct decisions on the state variables. Its stability comes from the conservative design based on the stability of the constant impedance controller. In [34,35], model-based reinforcement learning methods are used to approximate the system’s environmental change model to predict the evolution of the system state in the future.

Usually, the law of environmental change is unknown to the optimizer. In practice, environmental changes are usually estimated by designing dynamic models or designing model-free reinforcement learning to directly approximate the state results after environmental changes. The difficulty in model-based reinforcement learning is that, due to uncertainty, a good dynamics model is usually difficult to design [36,37]. In model-free methods, the value function is directly synthesized, which requires a lot of training time and data.

Deep Deterministic Policy Gradient (DDPG) [38] is one of the reinforcement learning methods that is based on actor–critic architecture. It combines the advantages of the DQN algorithm and DPG algorithm. It not only solves the problem that random policy is difficult to sample in complex action space, but also makes up for the defect that DQN cannot deal with high-dimensional continuous action space. It is very suitable for applications with high dimensions and large task space such as robot operation.

The Gaussian process (GP) model is another kind of learner that has been combined with reinforcement learning [34,35,39,40]. Compared with other regression models, the GP model has a better approximating effect when the data set is smaller and contains the uncertainty of the model. Such characteristics are more suitable for relatively small changes during industrial robot operations. We use it to approximate the policy function to achieve the purpose of reducing the amount of data and fast convergence.

In this paper, an intelligent variable admittance control method, combining classical control theory and reinforcement learning theory, is proposed and applied to the interactive compliance control of industrial robots in a changing environment. By fully considering the results of classical control and an intelligent optimizer, the parameters of impedance control are adjusted, which improves the limitation of the model-free reinforcement learning method. This method does not need to model the complex physical environment and robot, and does not require long-term experience accumulation, which improves safety and learning efficiency. The GP model is combined with the DDPG in the reinforcement learning system and applied to the approximation policy function. The improved algorithm requires fewer data sets and hyperparameters, and has higher learning efficiency and faster convergence.

The rest of this paper is as follows: Section 2 introduces the classical admittance control and states the problems to be studied. Section 3 describes the variable admittance control algorithm proposed in this paper in detail. Section 4 shows the results of simulations and experiments in detail. Finally, in Section 5, we summarize the research results of this paper.

2. Problem Statement

2.1. Classical Admittance Control

Hogan [8] proposed impedance control. The core idea of impedance control is to model the control object as a system with spring–mass–damping characteristics, and the relationship between contact force and motion state is characterized by the dynamic equation, so as to realize the control of the dynamic relationship between the two. In Cartesian space, the typical impedance model of a robot is expressed as

$$M_d({\ddot{X}}_c-{\ddot{X}}_d)+B_d({\dot{X}}_c-{\dot{X}}_d)+K_d(X_c-X_d)=F_d-F_e$$

(1)

$M_d,B_d,K_d\in {\mathbb{R}}_{n\times n}$ (n is the dimension of the minimal coordinate set) represent the mass, damping, and stiffness of the assumed system, which are positive definite. $X_d,X_c\in {\mathbb{R}}_{n\times 1}$ are the reference trajectory and the corrected trajectory; ${\dot{X}}_d,{\dot{X}}_c$, ${\ddot{X}}_d,{\ddot{X}}_c$ are their velocity and acceleration, respectively. $F_e\in {\mathbb{R}}_{n\times 1}$ is the actual contact force / torque, and $F_d\in {\mathbb{R}}_{n\times 1}$ is the expected one.

Admittance control is impedance control based on motion, which realizes the mapping from generalized force to generalized position. In general, the structure consists of a force control outer loop and a position control inner loop, as shown in Figure 1. The force controller receives the feedback from the force sensor, and outputs the corrected trajectory $X_c$. After inverse kinematics mapping, the joint trajectory $q_c$ is tracked by the joint controller inner loop. Nowadays, compared with the constant parameter admittance control method, the admittance control of variable parameters must be developed to meet the increasingly complex work requirements.

2.2. Evolution of Contact Force

We model the robot as an admittance model with a multi-dimensional spring–mass–damping system. In general, the contact between the robot and the environment is divided into three phases, as shown in Figure 2.

On the left side of Figure 2, a schematic diagram of three phases is shown, and, on the right side, the evolution of contact force in different phases is shown. In the free movement phase, the robot has not been in contact with the environment. At this time, the contact force $F_e=0$; the admittance model changes the trajectory under the guidance of $F_d$ and gradually drives the robot to move in the direction of $F_e-F_d\to 0$. In the oscillation phase, the robot has a strong nonlinear collision with the environment because the location and characteristics of the environment are unknown to the control system. However, due to the adjustment of the trajectory by the admittance control, such oscillations will gradually converge. In the stable contact phase, the contact force between the two tends to be stable, and $F_e-F_d=0$.

In the oscillation phase of contact, the high force overshoot is not expected. At the same time, we hope that the contact force can quickly stabilize and make the steady-state error tend to zero. The following part of the paper will introduce the research on achieving this goal.

3. The Proposed Method

In this paper, an intelligent variable admittance control method is designed. In this method, the classical adaptive admittance controller is used as the initial policy to generate the previous experience (similar to the experience of human experts), in combination with a variable admittance controller trained by a reinforcement learning method (intelligent policy).

3.1. Intelligent Admittance Control System

As shown in Figure 3, the selector outputs variable admittance parameters by selecting two policies. For security, in the early phase of training, it tends to select the action generated by the initial policy and record the generated data set; after a period, the transformation from selecting the initial policy to selecting the learning policy is realized. In Figure 3, $M_d,B_d,K_d,X_c,X_d,F_d$, and $F_e$ are defined as Equation (1). $B_{\theta },K_{\theta }$ are the variable admittance parameters designed in this paper. ${\pi }_{\theta }$, ${\pi }_I$ are the intelligent policy and initial policy, respectively, and $q_c$ is the joint variable of $X_c$ solved by inverse kinematics. After the adjustment of the robot joint controller, the real joint position of the robot is $q_m$, while $X_m$ is the forward kinematics solution of $q_m$.

3.2. The Establishment of Robot Model and Environment Model

The task environment is modeled as a first-order linear system (a linear spring):

$$F_e=K_e(X_c-X_e)$$

(2)

where $X_e$ is the environmental position; the definition of $X_c$ is the same as Equation (1).

The robot is modeled as a classical admittance model, as shown in Equation (1). Suppose that the trajectory tracking error of the joint controller is zero, i.e., $X_c=X_m$. Let $E=X_c-X_d$ and $\Delta F=F_d-F_e$; then, the admittance formula of the robot can be written as

$$M_d\ddot{E}+B_d\dot{E}+K_{d}E=\Delta F$$

(3)

With E as the output and $\Delta F$ as the input, the second-order system shown in Equation (3) has a transfer function $H\left(s\right)=E\left(s\right)/\Delta F\left(s\right)=1/(M_d·s^2+B_d·s+K_d)$. $\Delta F$ is expected to gradually approach 0; for $\Delta F$, there is

$$\begin{array}{c}\Delta F=F_d-F_e=F_d-K_e(X_c-X_e)\hfill \\ =F_d-K_e(X_d+\Delta F·H\left(s\right)-X_e)\hfill \\ {\displaystyle =\frac{(M_d·s^2+B_d·s+K_d)}{[K_e+(M_d·s^2+B_d·s+K_d)]}[F_d-K_e(X_c-X_e)]}\hfill \end{array}$$

(4)

When the system is in contact steady state, it has $s\to 0$. The force error $\Delta F_{ss}$ is expressed as

$$\Delta F_{ss}={\displaystyle \frac{K_d}{K_d+K_e}}[F_d-K_e(X_c-X_e)]$$

(5)

To make $\Delta F_{ss}\to 0$, we need to satisfy

$$X_c\to X_e+{\displaystyle \frac{F_d}{K_e}}$$

(6)

Under the uncertainty conditions considered in this paper, Equation (6) cannot be satisfied directly due to the unknown values of $X_e$ and $K_e$, and the steady state of the force error will always exist. To reduce the error, the disturbance caused by environmental uncertainty must be considered. It is assumed that the disturbance caused by environmental changes, including deformation caused by contact force, is $\delta E$ and the disturbance of interaction force caused by $\delta E$ is $\delta F=K_e·\delta E$. Let $\widehat{E}=E+\delta E$ and $\widehat{F}=\Delta F+\delta F$; then, it becomes

$$M_d\ddot{\widehat{E}}+B_d\dot{\widehat{E}}+K_d\widehat{E}=\widehat{F}$$

(7)

$M_d$, $B_d$, and $K_d$ are classical admittance parameters that are constant. Next, variable admittance parameters are designed to compensate for the disturbance. It should be noted that the change in $M_d$ is not conducive to the stability of the system [13]. Thus, in this paper, we do not consider changing it. Let the variable admittance parameters be $B_{\theta }$ and $K_{\theta }$; then, it becomes

$$M_d\ddot{E}+B_d\dot{E}+K_{d}E=\Delta F-B_{\theta }\dot{\widehat{E}}-K_{\theta }\widehat{E}+\delta F-(M_d·\delta \ddot{E}-B_d\dot{E}-K_{d}E)$$

(8)

Let $B_{\theta }\dot{\widehat{E}}+K_{\theta }\widehat{E}\to \delta F-(M_d·\delta \ddot{E}-B_d\dot{E}-K_{d}E)$, then $\Delta F_{ss}\to 0$ holds when $t\to \infty$.

3.3. Intelligent Policy

The intelligent policy ${\pi }_{\theta }$ is a modeled as a learner of the damping parameter $B_{\theta }$, which is modeled as a Gaussian process model. Suppose that the data set $D=\{X,Y\}$, where $X,Y$ are the optimal/sub-optimal state–action pairs in the experience pool H, $X=\{S_1,\cdots ,S_{Ne}\}$, $Y=\{A_1,\cdots ,A_{Ne}\}$, and $A_i=B_{\theta i}$ is the action corresponding to $s_i$. The Gaussian process is an exploration of $f\left(X\right)$. It is generally modeled as a normal distribution with zero mean, which is uniquely determined by the covariance function. Considering the independent and identically distributed noise $\mathcal{N}(0,{{\sigma }_n}^2)$, Y is denoted by

$$Y=f\left(X\right)+\mathcal{N}(0,{{\sigma }_n}^2)\sim \mathcal{N}(0,K(X,X)+{{\sigma }_n}^{2}I)$$

(9)

where $K(·)$ is the covariance matrix and I is the identity matrix.

For $X^{\prime }$, the noiseless prediction function $f^{\prime }\left(X^{\prime }\right)$ can be expressed as

$$f^{\prime }\sim \mathcal{N}(0,K(X^{\prime },X^{\prime }))$$

(10)

Under the condition of D, Y and $f^{\prime }$ satisfy the multivariate joint Gaussian distribution. From the Bayes Rule, we can obtain

$$\begin{array}{c}p(f^{\prime }\mid X^{\prime },D)\sim \mathcal{N}(u,\sum );\\ u=k(X^{\prime },X){(k(X,X)+{\sigma }^{2}I)}^{-1}Y\\ \sum =K(X^{\prime },X^{\prime })-K(X^{\prime },X){(K(X,X)+{\sigma }^{2}I)}^{-1}K(X,X^{\prime })\end{array}$$

(11)

At time t, for ${\pi }_{\theta }$, $X^{\prime }$ is the state $S_t$ and the corresponding output of GP is $A_t=B_{\theta t}$. Then, $A_t$ takes the mean of the posterior distribution:

$$A_t=u=K(S_t,X){(K(X,X)+{{\sigma }_n}^{2}I)}^{-1}Y$$

(12)

For each element $k(x_i,x_j^{\prime })$ in the covariance matrix $K(X,X^{\prime })$, we select the squared exponential kernel function:

$$k(x_i,x_j^{\prime })={\sigma }_{f}exp(-{\displaystyle \frac{d{(x_i,x_j^{\prime })}^2}{2{l_{i,j}}^2}})$$

(13)

where $x_i,x_j^{\prime }$ are one-dimensional elements in X and $X^{\prime }$, respectively. ${\sigma }_f$, $l_{i,j}$ are the hyperparameters of the kernel function. ${\sigma }_f$ is the sample variance, and $d(·)$ represents the Euclidean distance measure between features.

To ensure the exploration in the early stage, noise $\eta$ is introduced; then, ${\pi }_{\theta }$ is denoted by

$$\begin{array}{c}{\pi }_{\theta }=\{K_{\theta },B_{\theta }\}\hfill \\ B_{\theta }=u(S_t,{\theta }^{\pi })+\eta \hfill \\ =K(S_t,X){(K(X,X)+{{\sigma }_n}^{2}I)}^{-1}Y+\eta \hfill \\ =B_{\theta }\hfill \end{array}$$

(14)

where $K_{\theta }$ is a constant. In order to meet the stability conditions of the admittance system described in [41], $K_{\theta }$ is also limited as $K_{\theta }\succ 0$. ${\theta }^{\pi }$ is the hyperparameter of ${\pi }_{\theta }$, which includes $D=\{X,Y\}$ and the hyperparameter $\{{\sigma }_f,L,{\sigma }_n\}$ of the kernel function itself. ${\theta }^{\pi }=\{{\theta }^s,{\theta }^k\}=\{X,Y,{\sigma }_f,L,{\sigma }_n\}$, ${\theta }^s$=$\{X,Y\}$, ${\theta }^k$=$\{{\sigma }_f,L,{\sigma }_n\}$, and $L=\{l_{1,1},\dots ,l_{i,j},\dots \}$.

3.4. Initial Policy

The initial policy ${\pi }_I$ of this paper adopts the adaptive control law designed in [13]. In that paper, it is assumed that the robot is decoupled; the single-dimensional control law is expressed as follows:

$$\begin{array}{c}{\pi }_I=\{k_I,b_I\};\hfill \\ k_I=0;\hfill \\ b_I=b_{first}+\Delta b;\hfill \\ \left\{\begin{array}{c}\Delta b\left(t\right)={\displaystyle \frac{b_{first}}{\stackrel{.}{\widehat{e}}\left(t\right)}}\mathrm{\Phi }\left(t\right)\hfill \\ \mathrm{\Phi }\left(t\right)=\mathrm{\Phi }(t-\lambda )+\sigma {\displaystyle \frac{f_d(t-\lambda )-f_e(t-\lambda )}{b_{first}}}\hfill \end{array}\right.\hfill \end{array}$$

(15)

where the definition of $\stackrel{.}{\widehat{e}}\left(t\right)$, $f_d$, $f_e$ are as in Section 3.2, i.e., of a one-dimensional form. $b_{first}$ is the one-dimensional damping initial value, $\Delta b$ is a time-varying quantity, t is the real time, $\lambda$ is the sampling rate, and $\sigma$ is the learning rate.

Through the compensation of the adaptive variable damping control law, $\Delta f_{ss}\to 0$ is established at steady state. The stability of the control law has been proven by the author of that article. Although the classical adaptive admittance control law is more robust than the constant admittance control, the experiment shows that its effect is still limited.

3.5. Intelligent Admittance Policy Integrated with Classical Control

Finally, the policy $\pi$ of the intelligent admittance controller is defined as $\pi :x\mapsto u=\pi (x,t,\theta )$:

$$\pi =\left\{\begin{array}{c}{\pi }_I;{\alpha }_1\ge \epsilon or\Delta F_{ss}\in \mathbb{U}(0,\delta )\hfill \\ {\pi }_{\theta };else\hfill \end{array}\right.;\epsilon \ge 0$$

(16)

$\epsilon$ is a constant positive real number, and ${\alpha }_1$ is a time-negative correlation function. When the training begins, ${\alpha }_1$ is larger and $\pi$ has a large probability to choose the decision of ${\pi }_I$. With the increase in training time, ${\alpha }_1$ gradually decreases until $\pi$ completely tends to choose ${\pi }_{\theta }$ after the accumulation of training.

$\mathbb{U}(0,\delta )$ denotes the neighborhood of $\Delta f=0$. Since the randomness of the GP model always exists, when the output of the intelligent policy enters the neighborhood of the expected value, a small probability of jitter events will inevitably occur. When the robot system tends to the steady state of contact, its acceleration and force error should be very close to 0. At this time, the classical adaptive admittance control is selected as a policy to ensure accuracy.

3.6. Policy Optimization: Modified DDPG Algorithm

In the actor–critic architecture of the classic DDPG, the actor is designed as a deterministic policy function $A_t={\mu }_{\theta }\left(S_t\right|{\theta }^{\mu })$, where $A_t$, $S_t$ are the actions of agent and state of environmental at time $t+1$, ${\mu }_{\theta }$ is the policy function, and ${\theta }^{\mu }$ is the parameter of ${\mu }_{\theta }$. The critic is a value function $Q_{\mu }(S_t,A_t)$ used to evaluate the policy ${\mu }_{\theta }$, and both actor and critic are fitted by a neural network. To correct the deviation caused by the self-iteration of a single network, DDPG sets the dual neural network model architecture of the target network ${\mu }_{target}$, $Q_{target}$ and online network ${\mu }_{online}$, $Q_{online}$.

For ${\pi }_{\theta }$, a modified DDPG algorithm is designed to optimize it, which is called the GP-DDPG algorithm. Specifically, ${\pi }_{\theta }$ is used as the actor in the DDPG, and the numerical method adapted to the GP model is used. The following describes the adaptive changes to DDPG.

As shown in Figure 4, the actor in this paper is ${\pi }_{\theta }$, which outputs a uniquely determined action based only on $\theta$ and S. The critic, that is, the Q function is modeled as a neural network. The discrete time is $t=0,\dots ,n,\dots$, the state is $S_t=\left(X_{c,t},{\dot{X}}_{c,t},F_{e,t}\right)$, $X_{c,t}\in {\mathbb{R}}^{n\times 1}$ is the position vector of the end-effector in the Cartesian coordinates, ${\dot{X}}_{c,t}$ is the velocity vector, and $F_{e,t}\in {\mathbb{R}}^{n\times 1}$ is the force/torque. $A_t=B_{\theta ,t}$, where $B_{\theta ,t}\in {\mathbb{R}}^{n\times 1}$ is a simplified form of the output damping parameter, in which each element is composed of diagonals in the decoupled $B_{\theta ,t}\in {\mathbb{R}}^{n\times n}$ matrix form. For each state–action pair, the action value $q_t$ is calculated. The experience pool stores all historical data, which are used by the classical DDPG for experience playback, and, in this paper, they are also used to collect the hyperparameter ${\theta }^s$.

The single-dimensional $R_{t+1}$ considers error consumption $c{e}_{t+1}$ and energy consumption $c{d}_{t+1}$. It is expressed as follows:

$$\begin{array}{c}{R}_{t+1}=c{e}_{t+1}+c{d}_{t+1};\hfill \\ c{e}_{t+1}=\beta \Delta F^{\top }\Delta F+\left(1-\beta \right)\ast {\dot{E}}^{\top }\dot{E};\beta \in [0,1]\hfill \\ c{d}_{t+1}=-{\displaystyle \sum _i}\xi {({\displaystyle \frac{{\pi }_{t,i}}{{\pi }_{MAX,i}}})}^2.\hfill \end{array}$$

(17)

where i in Equation (17) is the element of each dimension in the vector, ${\pi }_{MAX}$ denotes the restriction on the output, which has the same form as ${\pi }_t$, and $\xi$ is a constant coefficient.

The optimization process of the objective function J is to find the optimal hyperparameter ${{\theta }^{\pi }}^{*}$ of ${\pi }_{\theta }$, so as to obtain the optimal solution ${{\pi }_{\theta }}^{*}$:

$$\begin{array}{c}{{\pi }_{\theta }}^{*}=\underset{{\pi }_{\theta }}{argmax}J\left({\pi }_{\theta }\right)\hfill \\ J\left({\pi }_{\theta }\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\displaystyle \sum _{t=0}^n}{\gamma }^t{R}_{t+1}\right]={\mathbb{E}}_{{\pi }_{\theta }}\left[R_1+\gamma R_2+{\gamma }^2{R}_3+\dots +{\gamma }^n{R}_{n+1}\right]\hfill \end{array}$$

(18)

Different from the classical DDPG, the numerical method is used in this paper. As explained in Section 3.4, the hyperparameters of the policy function in this article are ${\theta }^{\pi }=\{{\theta }^s,{\theta }^k\}$. The elite method is used for ${\pi }_{\theta }$, and the maximum likelihood estimation (MLE) is used to obtain ${{\theta }^{\pi }}^{*}$.

For ${\theta }^s$, different from the traditional sampling method, the prior data of the GP in this paper uses the elite method to select the $Ne$ term with the greatest value from the experience pool H. Since ${\pi }_{\theta }$ has the exploratory property in multiple iterations, the sampling method in this paper still satisfies the random sampling condition. The choice of ${\theta }^s$ takes into account the value $q_n$ of the action $A_n$ in each experience $h_n$, as well as the similarity between the state $S_n$ in which the experience is recorded and the current state $S_t$.

$$\begin{array}{c}{\theta }_{t+1}^s=\left\{(S_n,A_n)\right|(S_n,A_n,q_n)\in h_n\subset H,\mathbb{I}\left(h_n\right)=1\};n=1,\dots ,Ne\le t\hfill \\ \mathbb{I}\left(h_n\right)=\left\{\begin{array}{c}1;|S_t-S_n|<{\rho }_{1}and{q}_n>{\rho }_2\hfill \\ 0;else\hfill \end{array}\right.\hfill \end{array}$$

(19)

$\mathbb{I}(·)$ is the indicator function, which selects $Ne$ elite samples with the best action value from H, $|S_t-S_n|$ is the Euclidean distance between the current state and the historical state, ${\rho }_1$, ${\rho }_2$ are positive real numbers, and h is the historical experience in H. It should be noted that, to maximize the exploration rate, we update the $q_n$ value of the earlier data in H dynamically and randomly with the minibatch update of the critic network.

For ${\theta }^k$, it is randomly initialized, and the optimization is completed by maximum likelihood estimation. In fact, the MLE is also a common method to deal with the hyperparameter optimization of Gaussian distribution.

$${\theta }_{t+1}^k=\underset{\theta }{argmax}logp\left(A\right|S,{\theta }_t^k);(S,A)\in {\theta }_{t+1}^s$$

(20)

The critic is used to estimate the $q_t$ of each $A_t$ in H to select elites as the hyperparameter set ${\theta }^s$ of the actor. In this way, the actor will be optimized along the direction of increasing Q. The optimization of the critic is similar to that of the classical DDPG using the gradient descent method:

$${\nabla }_{{\theta }^Q}={\mathbb{E}}_{S_t,A_t,R_{t+1}\sim R}\left[(y_t-Q(S_t,A_t;{\theta }^Q)){\nabla }_{{\theta }^{\pi }}{Q}_{{\pi }_{\theta }}(S_t,A_t;{\theta }^{{\pi }_{\theta }})\right]$$

(21)

$$y_t=R_t+\gamma Q_{target}(S_{t+1},{{\pi }_{\theta }}_{target}\left(S_{t+1}\right|{\theta }^{{\pi }_{target}})|{\theta }^{Q_{target}})$$

(22)

Finally, the algorithm we proposed can be expressed as in Algorithm 1:

Algorithm 1 GP-DDPG Algorithm Combined with Classical Control

1: Initialize $t=0$, ${\theta }_0^{\pi }$, ${\theta }_0^Q$, ${\pi }_{\theta }^{{\theta }_0^{\pi }}$     Initialize action value network $Q^{{\theta }_0^Q}$     Initialize experience pool H 2: for episode = 1: M do 3:         Receive initial observation state $S_0$ 4:         for t = 0: T do 5:               if ${\alpha }_1\ge \epsilon$ or $\Delta F$ is in the neighborhood of zero 6:                   $A_t$=${\pi }_I^t\left(S_t\right)$ 7:                else 8:                   $A_t$=${\pi }_{\theta }^{{\theta }_{\mathrm{t}}^{\pi }}\left(S_t\right)$ 9:               end if 10:                Calculate $R_{t+1}$, observe $S_{t+1}$ 11:                Store transition $h=\{S_t,A_t,R_{t+1},S_{t+1},q_t\}$ in H 12:                Get TD error with random minibatch of N transitions $h^n=\{S_t^n,A_t^n,R_{t+1}^n,S_{t+1}^n\}$;                   $n=1,\dots ,N$ and soft update ${\theta }_t^Q$ to ${\theta }_{t+1}^Q$ 13:                Calculate $q_t^{n^{\prime }}$ use $Q^{{\theta }_{t+1}^Q}$ for each and update $q_t^n=q_t^{n^{\prime }}$ use $Q^{{\theta }_{t+1}^Q}$ 14:                Extract $Ne$ samples from H based on Equation (19) as ${\theta }_{t+1}^s$ 15:                Get ${\theta }_{t+1}^k$ using MLE based on ${\theta }_{t+1}^s$ 16:         end for 17: end for

4. Results

The algorithm is simulated in MATLAB 2022b. In the simulation, the error of the position controller is ignored. Without any loss of generality, only one dimension of force control effect is verified in this paper. The single-dimension damping parameter output is limited to $B_{\pi }\in (0,300)$; $\lambda$ in ${\pi }_I$ is $\lambda =1\times {10}^{-8}$. For ${\pi }_{\theta }$, there are $\xi =1$, $\beta =0.1$. $M_d=1$, $K_{\theta }=1\times {10}^{-6}$ are single-dimensional mass parameter and spring parameter design, respectively.

4.1. Simulation Results

4.1.1. The Performance of the GP-DDPG Algorithm

Firstly, the learning performance of the proposed method is verified and compared with the method using only GP-DDPG (without initial policy); the necessity of the initial policy is proven. The robot task environment is shown in Figure 5. The base coordinate system is located on the robot base, and the end of effector is located on the X-O-Z plane, moving at a constant speed of 0.3 m/s in the Y direction. The environment is a plane $y_e=1$, perpendicular to the motion direction of the robot. The stiffness of the plane is $K_e$, the position of environmental is completely unknown to the robot, and the expected force in the Y direction is set to $F_d=20$ N.

For ${\pi }_I$, there are $K_I=0$, $B_{first}=100$. The Q network is designed as a full-connect neural network with three hidden layers; each layer has 64 neurons, and the initial weight is sampled from a Gaussian distribution.

Assuming that $K_e=1000$, the duration of a task is set to 15 s, and $N=10$ learnings are performed. The obtained learning results and evolution process are shown in Figure 6, Figure 7 and Figure 8. The addition of the classical algorithm provides experience for the learner. The whole algorithm shows a convergence trend. When $N=5$, the output of ${\pi }_{\theta }$ tends to be stable. At this time, ${\pi }_{\theta }$ is still exploratory, and the fluctuation caused by bad trials is quickly corrected, which indicates that the robustness of the controller is gradually improved. After that, the performance of ${\pi }_{\theta }$ is further improved. When the contact force reaches the neighborhood of the expected steady-state error, it is switched to the classical adaptive controller, so that $\Delta F_{ss,y}\to 0$. After 10 instances of learning, the model can stabilize the contact force at $F_d=20$.

The GP-DDPG algorithm without initial policy is used for simulations, and the other conditions remain unchanged. Figure 9 and Figure 10 show the experimental results. It can be seen from the figures that, due to the lack of prior experience, the contact force exceeds 100 N in the first training, and the task fails. After a period of training, the algorithm can still converge and reach $\Delta F_{ss,y}\to 0$ at $N=15$, but it is delayed by about 33% compared with the algorithm with initial policy. It only converges to the locally optimal solution.

4.1.2. Performance of Intelligent Admittance Control in an Uncertain Environment

In the following experiments, the rough initial position of the environment is known, and the speed and acceleration of the actual environmental change is not zero, which is the environmental uncertainty for which the controller has to compensate. The simulation results of our algorithm are compared with the constant admittance control and the classical adaptive admittance control shown in Equation (15).

Firstly, the force-tracking performance under the variable stiffness plane is evaluated. The task is the same as in Section 4.1.1, and the variable stiffness plane is $y_e=0.8$. The environmental stiffness is set to

$$K_e=\left\{\begin{array}{c}5000;0\le t\le 1000 \mathrm{ms}\hfill \\ 3000;1000<t\le 2000 \mathrm{ms}\hfill \\ 5000;2000<t\le 3000 \mathrm{ms}\hfill \end{array}\right.$$

(23)

The constant admittance control parameter is set to $B=100$, $K=1\times {10}^{-6}$; for ${\pi }_I$, there is $\sigma =0.001$. Figure 11 illustrates the trajectory-tracking and force-tracking performance of the three methods. It can be seen from the figure that all three methods can achieve force tracking. Among them, the intelligent admittance controller contacts the environment within 3–4 s, which is faster than the other two methods. When the environmental stiffness changes, the abrupt force causes a short overshoot and then recovers quickly.

From Figure 11, it can be seen that the evolution trend of the damping parameter B of classical adaptive admittance control and intelligent admittance control is obviously different. Overall, the damping parameter of intelligent admittance control is significantly smaller than that of classical adaptive admittance control, because it takes into account the energy consumption term a. When the robot is not in contact with the environment, the damping of classical adaptive admittance control increases with time, which reduces the acceleration of the robot and, thus, alleviates the unknown collision. When the contact force appears, it becomes smaller immediately. In the learning process, intelligent admittance control takes into account the speed error $\dot{E}$, force error $\Delta F$, and energy consumption $c_d$. When the robot is not in contact with the environment, the damping parameter is reduced so that the robot can reach the environment quickly. However, small damping also leads to an increase in overshoot.

Next, the performance in a complex environment (the speed and acceleration of the environment are not constant) is evaluated. In this paper, a complex surface that changes according to exponential sine curves is designed.

The contact surface changes with time as follows:

$$\left\{\begin{array}{c}{y}_e=a·sin\left(0.8t\right)+0.005t\hfill \\ a=0.5exp(-0.2t)\hfill \end{array}\right.$$

(24)

The initial position of the origin of the robot end is set to $y_0=0$; for ${\pi }_I$, there is $\sigma =0.01$ and $K_e=1000$, and the other conditions remain unchanged. Compared with the sinusoidal surface, the change in the class-exponential sinusoidal surface is irregular, so it can better reflect the robustness of the algorithm. Figure 12 illustrates the performance of the algorithm. The constant admittance control cannot converge. The classical adaptive admittance control converges after a period of adjustment. After training, the robustness of the intelligent admittance control is obviously better than the former, and the response time is faster.

From Figure 12, it can be seen that the evolution trend of the damping parameter B of classical adaptive admittance control is similar to that in the variable stiffness simulation. It is worth noting that, when the system has not yet reached the steady state, our algorithm obtains a smaller damping value, which makes the robot reach and adapt to the environment faster, but also causes high-frequency oscillation, which may have a negative impact on the stability of the system in the application. In order to reduce the high-frequency oscillation of the system, the weight parameters in the reward can be adjusted appropriately to reduce the consideration of energy consumption $c_d$ and reduce the speed of the decrease of the damping parameter value.

Based on the results of the three simulations, the algorithm we designed has the best performance. The constant admittance control cannot guarantee the force tracking in all environmental stiffnesses, while the adaptive admittance algorithm has poor tracking effect in low-stiffness conditions, and there is a large jitter in the face of complex surfaces. Our algorithm can stably control the force error at the micron level, and the resistance to environmental stiffness changes is better than the first two.

4.2. Experimental Results

To further verify the proposed algorithm, the UR10e manipulator is used as the experimental object for experimental verification. Figure 13 shows the environment of this experiment. UR10e integrates the force/torque sensor. The sensor has a Z direction measurement range of 100 N and an accuracy of 5.5 N (in the experiment, the read force/torque data are filtered). Using python, a software system based on ROS is built in the Linux environment. The computer communicates with the manipulator control box through the TCP/IP protocol, sends motion instructions, and obtains the real-time status of the manipulator with a ommunication frequency of 250 HZ.

The experimental scene is shown in Figure 14. This paper shows three experiments, in which the environment is a slope. In the first experiment, the slope of the slope was steeper, and in the second experiment, the slope of the slope was gentler. The third experiment changed the environmental characteristics and covered the soft rubber material at one end of the slope. Combined with the accuracy of the force sensor and the force drift when the end is suspended, the allowable force error is set to ±2.5 N in this paper. The experiments compare three algorithms of constant admittance, classical adaptive admittance, and intelligent variable admittance, which are the same as the algorithms compared in the simulation.

The desired force of 20 N is set on the Z axis. The constant-admittance control parameter is set to $B=50$, $K=1\times {10}^{-6}$. $M_d$, $K_{\theta }$, and $\lambda$ are the same as the simulation. For ${\pi }_I$, there is $\sigma =0.0001$ and $B_{first}=50$. The characteristic is unknown.

In the steep slope experiment, the end-effector is hovered above a hard slope, that is, the position of the base coordinate system Z = 0.115 m, and moves forward along the y-axis on a fixed slope.

The results are shown in Figure 15, Figure 16 and Figure 17. All three methods have a certain compliance effect. Under the constant admittance control, the end-effector of the robot collides with the slope after about 23 s and generates overshoot. After a period of adjustment, it still cannot track the required force, and the force error always stays at about 7.5 N.

Under the classical adaptive admittance control, the end-effector of the robot collides with the inclined plane after about 13 s. After adjustment, it slowly converges to the expected force. The evolution trend of the damping parameter is the same as that in the simulation. When the environment is not contacted, the damping parameter increases continuously and decreases rapidly after contacting the environment. However, due to the measurement error in the equipment and environment and the influence of noise, the evolution of the damping parameter also has large jitter.

Under the control of intelligent admittance, the robot touches the slope and generates overshoots after about 10 s, and the desired force can be tracked smoothly in about 80 s. On average, the damping parameter of intelligent admittance control is still smaller than that of classical adaptive-admittance control. Compared with the latter, the damping change of intelligent admittance control is more obvious, which is also the reason why it reaches the environment faster and realizes force tracking earlier.

The overshoot may cause damage to the robot. When other conditions remain unchanged, the overshoot will increase with the increase in the expected contact force because the force error will affect the change in the motion state. Collision is unavoidable. In industrial assembly applications, when the expected force is small (which is also the purpose of force control), the overshoots of the three algorithms shown in the experiments are not enough to cause harm to the workpiece and robot in the assembly of high-strength metal materials, which is acceptable.

In the gentle slope experiment, the position of the base coordinate system Z = 0.113 m, and it moves forward along the y-axis on a fixed slope.

The results are shown in Figure 18, Figure 19 and Figure 20. Under the constant-admittance control, the robot collides with the slope after about 16.7 s and generates overshoot. After a period of adjustment, it still cannot track the required force, and the force error always stays at about 4.8 N. Under the classical adaptive-admittance control, the robot collides with the inclined plane after about 20.5 s, and then is in a state of overshoot for some time. After adjustment, it converges to the expected force at 104.9 s. For the intelligent-admittance control, the robot touches the slope and generates overshoots after about 10.4 s, and the desired force can be tracked smoothly in about 63.9 s. It can be seen that there is no prominent change in the force-tracking performance of each algorithm.

It is worth noting that all algorithms are slower to reach the environment because the initial position of the robot is farther away from the environment. In addition, the force overshoot is reduced compared to the force overshoot of the steep slope, and the convergence speed of the variable-admittance algorithm is improved.

In the third experiment, the characteristics and location of the environment changed, and the setting of the slope was the same as in the first experiment. The position of the base coordinate system Z = 0.115 m, and we set the Z axis force to 5 N.

The results are shown in Figure 21, Figure 22 and Figure 23. Under the constant-admittance control, the robot is in contact with the environment at about 12.2 s, while classical adaptive-admittance control is about 24.2 s and intelligent-admittance control is about 8.6 s. When passing through the rubber material, none of the three can stably track the expected force, which may be caused by the overly soft material. However, it is gratifying that, after a small amount of overshoot caused by the switching of materials, all three algorithms can track the desired force. Among them, the constant-admittance control needs to experience 22.2 s, classical adaptive-admittance control needs to experience 18.9 s, and intelligent-admittance control needs to experience 12.1 s.

The experimental results further prove the feasibility of the intelligent control algorithm designed in this paper. The overshoots of the three algorithms of constant admittance, classical adaptive admittance, and intelligent variable admittance are within the allowable range. The adjustment time of intelligent variable-admittance is better than that of classical adaptive-admittance control, and its advantages rise with the dramatic changes in the environment. When the experimental environment is a steep slope, it is improved by about 44%. When the environment is a gentle slope, it is improved by 39%. When there is a flexible material in the environment as a buffer for collision and the expected force is smaller, it is improved by 35.9%.

5. Discussion and Conclusions

To improve the adaptability and safety of a robot working in an uncertain environment, an intelligent-admittance control algorithm based on DDPG combined with classical control is proposed. Simulations and experiments are set up to verify the performance of the algorithm. The algorithm is based on reinforcement learning, combines the GP model with DDPG, and can converge to the expected value by using fewer than 15 instances of learning in the simulation.

To verify the superiority of the algorithm, which combines classical control theory and reinforcement learning theory, the control experiments of the algorithm without initial policy and the algorithm with initial policy are set up. From the simulation results, the convergence speed of the algorithm with initial policy is 33% higher than that of the algorithm without initial policy. And the algorithm without initial policy has a task failure due to excessive contact force in the early stage of training. The algorithm with initial policy avoids this failure well, thus improving security.

To verify the performance of the algorithm in an uncertain environment, the simulation of a variable-stiffness environment and complex environment are set up, and the results are compared with those of constant-admittance control and adaptive-admittance control. In the variable-stiffness environment experiment, all three algorithms can adapt to environmental changes and track the desired force. Among them, the speed of the intelligent-admittance algorithm to contact the environment is about 48% higher than that of other algorithms. In the complex environment experiment, assuming that the force control accuracy is ±0.2 N, the constant admittance control cannot track the contact force. The adaptive admittance control can track the contact force at about 26 s, while the intelligent admittance control can track the contact force at about 12 s and the speed is increased by about 53.84%.

Simulation represents an ideal world. To further verify the performance of the proposed algorithm, three experiments with a slope as the environment were set up, and the experiment was completed with a 6-DOF manipulator. In the slope surface experiments, the constant-admittance control cannot track the desired force when the expected force is large. The adaptive-admittance control can track the contact force, but its convergence speed is not as fast as that of intelligent admittance control, and this gap will become more obvious with the change in environment. In the experiments in this paper, the performance of intelligent admittance control can be improved by 44% compared with the traditional adaptive admittance control. The experimental results are highly similar to the simulation results, which can prove the effectiveness of the simulation.

The disadvantage is that, in the simulations and experiments, this paper only focuses on the tracking performance of the algorithm in a single dimension, and verifies the algorithm under a simple model. In the future, we plan to apply it to specific assembly tasks, consider more environmental interference, and conduct research on dual-robot collaborative assembly tasks.