Reinforcement-Learning-Based Visual Servoing of Underwater Vehicle Dual-Manipulator System

Abstract

As a substitute for human arms, underwater vehicle dual-manipulator systems (UVDMSs) have attracted the interest of global researchers. Visual servoing is an important tool for the positioning and tracking control of UVDMSs. In this paper, a reinforcement-learning-based adaptive control strategy for the UVDMS visual servo, considering the model uncertainties, is proposed. Initially, the kinematic control is designed by developing a hybrid visual servo approach using the information from multi-cameras. The command velocity of the whole system is produced through a task priority method. Then, the reinforcement-learning-based velocity tracking control is developed with a dynamic inversion approach. The hybrid visual servoing uses sensors equipped with UVDMSs while requiring fewer image features. Model uncertainties of the coupled nonlinear system are compensated by the actor–critic neural network for better control performances. Moreover, the stability analysis using the Lyapunov theory proves that the system error is ultimately uniformly bounded (UUB). At last, the simulation shows that the proposed control strategy performs well in the task of dynamical positioning.

1. Introduction

Unmanned underwater vehicles (UUVs) play an important role in the exploration of oceans. With the increasing demand for ocean development, the UUV, lacking the intervention ability, is no longer capable of handling certain tasks [1]. Meanwhile, the underwater vehicle manipulator system (UVMS) has been widely used in the marine energy and underwater architecture industry, and significant results have been achieved [2,3]. The application scenarios of the UVMS include but are not limited to, underwater pipeline inspection [4], ship maintenance [5], underwater rescue [6], terrain exploration [7], marine biology research [8,9], and marine archaeology research [10,11].

At present, these industrial UVMS are often fixed on the seabed during operation, forming a stable working environment, which ensures a certain level of security and sustained stable output while the scope and flexibility are limited. Furthermore, many underwater tasks cannot provide a reliable ground support environment. This makes it necessary to develop the floating-based UVMS, especially for the scenes of the bridge and the pipeline maintenance that require continuous movement, as well as the scenes such as marine archaeology and biological research that need to avoid damage to the environment during the operation. Based on UVMSs, underwater vehicle dual-manipulator systems (UVDMSs) have received a lot of research attention in recent years.

In terms of UVMS, the Girona 500 UVMS [12] of Girona University has dealt with tasks such as dynamic positioning and object grasping. The Dexrov project [13] has reduced the dependence on the work environment with the concept of remote operation. The Ocean One project [14,15] of Stanford University has developed a dual-arm humanoid robot with strong operational capabilities and perception capabilities. By remote operation, it has successfully carried out an archaeological task. Ref. [16] studied the task of moving an object to a precisely positioned peg while considering the impacts due to contact and provided a multiple impedance manipulation method that exhibits a smooth performance in the simulation. A whole-body control strategy of UVDMSs proposed in the MARIS Italian research project [17] has extended the task priority framework to deal with coordination manipulation and transportation problems. An alternative dual-arm configuration of UVDMSs is designed in [18] with only one manipulator for task operation and another for pose maintenance. This unique concept is proven to perform well in the presence of currents. Ref. [19] recently showed another different style of UVDMS similar to an underwater glider and provided a moving strategy making use of drag, which provides a low-power consumption solution for UVDMSs.

All the applications and research of the UVDMS above require reliable control methods, and the primary objective of the UVMS control is the dynamic positioning that ensures a stable working environment for other tasks. Due to the limitations of underwater signal transmission, visual positioning is commonly used for operations in the local space. The most widely used visual servo method is the image-based visual servo (IBVS) [20,21], which does not require calculating the 3D information of the targets. Refs. [22,23] propose a visual servo strategy for dynamic positioning, which utilizes the measuring sensors equipped with the UUV and achieves significant results. In [24], the visual servo method has been applied to the UVMS system, while the redundancy problem is solved through model prediction, resulting in a good positioning performance. However, few studies focus on visual positioning for the UVDMS. Considering that the manipulators of the UVDMS are usually installed away from the vehicle center, unlike the UVMS, in which the manipulator is always fixed along the center of gravity and buoyancy. This configuration has a significant impact on the stability of the system, and greater torques are required during the positioning task. Additionally, dual manipulators bring more difficulties for motion planning since the joint limits and collision risks increase.

Modeling the UVMS dynamics is challenging, as it requires consideration of both the impact of the water and coupling effects between the manipulators and the vehicle. A brief description of the kinematic and dynamic models of the UVMS is provided in [25]. Considering the model in [25], the numerical simulation is executed for the dynamic model in [26], and the coupling between the dual manipulators and the vehicle is analyzed in this work. It is impossible to develop a mathematical model that accurately represents the physical system. Until now, control research of the UVMS mostly uses robust and adaptive tools to solve the model uncertainties. In [24], the model coupling information is estimated by the EKF filter, and the model predictive-based control provides an optimal kinematic solution. In [27], the sliding mode control of the UVMS, with certain anti-interference ability, has been provided for a tracking task. To deal with uncertainties and disturbance, Refs. [28,29] provide the adaptive control strategy and observer-based method, respectively, and both of them show good performance while handling uncertainties.

Based on the research above, this work studies the visual servo control of the UVDMS, considering the model uncertainties. A hybrid visual servo method is proposed based on the multiple cameras and attitude sensors equipped with the UVDMS. The command velocity is produced by the kinematic controller using the task priority scheme. In addition, the reinforcement-learning-based speed tracking controller was designed, and the system model error is compensated by the designed actor neural network. Meanwhile, the error system is proved to be ultimately uniformly bounded by the Lyapunov method. Finally, a UVDMS model is simulated, and the results prove the effectiveness of the proposed control.

This paper is organized as follows. Section 2 describes the mathematical model of the kinematics and dynamics of the UVDMS, as well as the hybrid visual servo model of the UVDMS. Section 3 formulates the kinematic control that produces the command velocity to be tracked. In Section 3, the reinforcement-learning-based adaptive control is formulated, while the actor–critic networks are designed to compensate for the system uncertainties. And the stability analysis is down in this section. Then, the simulation work using an 18-dof UVDMS is shown in Section 4. At last, in Section 5, a brief summary of this work is provided.

2. Problem Formulation

This section introduces the kinematics and dynamics of the UVDMS by providing the necessary coordinates and variables. A visual servo model considering the camera configuration of the UVDMS is established. In addition, a task priority strategy for redundancy control and a universal reinforcement learning method based on the actor–critic algorithm are provided. At last, the objective of this study is described.

2.1. UVMS Model

To illustrate the modeling of UVDMS, a 3D model consisting of a fully actuated UUV and two 6-dof manipulators is shown in Figure 1. Obviously, this is a redundant system with 18 degrees of freedom. According to [25,30], the underwater rigid body model can be described in several coordinate frames defined as ${\mathsf{\Sigma }}_i$ (the inertial frame), ${\mathsf{\Sigma }}_b$ (the vehicle body fixed frame with origin at the center of the mass), ${\mathsf{\Sigma }}_c$ (the main camera frame), ${\mathsf{\Sigma }}_{0i}$ (the manipulator base frame, $i=1,2$), ${\mathit{\Sigma }}_{ci}$ (the camera frame fixed with the end effector, $i=1,2$), and ${\mathsf{\Sigma }}_{ei}$ (the end-effector frame attached at the end of the manipulator, $i=1,2$). In frame ${\mathsf{\Sigma }}_i$, the pose vector of the base vehicle is defined as $\mathit{\eta }={[x,y,z,\phi ,\theta ,\psi ]}^T$, which contains the global position ${\mathit{\eta }}_1={[x,y,z]}^T$ and Euler angles ${\mathit{\eta }}_2={[\phi ,\theta ,\psi ]}^T$. The vehicle’s velocity $\mathit{v}={[u,v,w,p,q,r]}^T$ containing the linear velocity ${\mathit{v}}_1={[u,v,w]}^T$ and the angular velocity ${\mathit{v}}_2={[p,q,r]}^T$ is defined in frame ${\mathsf{\Sigma }}_b$.

In terms of the underwater manipulators, the states are generally described as the joint angles $\mathit{q}={[q_l,q_r]}^T$, with ${\mathit{q}}_{\mathit{l}}={[q_{i1},q_{i2},q_{i3},q_{i4},q_{i5},q_{i6}]}^T$, and the corresponding velocity $\dot{\mathit{q}}={[{\dot{q}}_l,{\dot{q}}_r]}^T$, with ${\dot{\mathit{q}}}_i={[{\dot{q}}_{i1},{\dot{q}}_{i2},{\dot{q}}_{i3},{\dot{q}}_{i4},{\dot{q}}_{i5},{\dot{q}}_{i6}]}^T$, where $i=l,r$. Combining both the velocities of the UUV and the manipulators, one obtains the motion transformation from the vehicle body frame to the inertial frame as

$$\left[\begin{array}{c}{\dot{\mathit{\eta }}}_1\\ {\dot{\mathit{\eta }}}_2\\ \dot{\mathit{q}}\end{array}\right]=\left[\begin{array}{ccc}{\mathit{R}}_{\mathit{B}}^{\mathit{I}}& {\mathsf{0}}_3& {\mathsf{0}}_{3\times 12}\\ {\mathsf{0}}_3& J_{k,o}^{-1}(\mathit{\eta })& {\mathsf{0}}_{3\times 12}\\ {\mathsf{0}}_{12\times 3}& {\mathsf{0}}_{12\times 3}& {\mathit{I}}_{12}\end{array}\right]\mathit{\zeta }=J_k^{-1}\mathit{\zeta },\mathit{\zeta }=\left[\begin{array}{c}{\mathit{v}}_1\\ v_2\\ \dot{q}\end{array}\right]\in {\mathit{\Re }}^{18\times 1},$$

(1)

where $R_B^I\in {\mathbf{\Re }}^{3\times 3}$ is the rotation matrix from the vehicle body frame to the inertial frame, and $J_{k,o}^{-1}\left(\mathit{\eta }\right)\in {\mathbf{\Re }}^{3\times 3}$ is the Jacobian matrix of the vehicle angular velocity. Then, the relationship between the system velocity and the end effector velocity can be formulated as

$${\dot{\mathit{\eta }}}_{\mathit{e}\mathit{e}}=\left[\begin{array}{c}{\dot{\mathit{\eta }}}_{\mathit{e}\mathit{e},1}\\ {\dot{\mathit{\eta }}}_{\mathit{e}\mathit{e},2}\end{array}\right]=J({\mathit{R}}_{\mathit{B}}^{\mathit{I}},q)\mathit{\zeta }=\left[\begin{array}{c}{J}_1({\mathit{R}}_{\mathit{B}}^{\mathit{I}},q)\\ J_2({\mathit{R}}_{\mathit{B}}^{\mathit{I}},\mathit{q})\end{array}\right]\mathit{\zeta }.$$

(2)

The equation above is a brief description of the direct kinematic of the UVMS with the Jacobian $J\in {\mathbf{\Re }}^{18\times 18}$ including $J_1\in {\mathbf{\Re }}^{6\times 18}$ (Jacobian of the left end effector) and $J_2\in {\mathbf{\Re }}^{6\times 18}$ (Jacobian of the right end effector), which can be computed by the DH parameters and the rotation matrix. ${\mathit{\eta }}_{ee,1}={[x_{e1},y_{e1},z_{e1},{\phi }_{e1},{\theta }_{e1},{\psi }_{e1}]}^T$ and ${\mathit{\eta }}_{ee,2}={[x_{e2},y_{e2},z_{e2},{\phi }_{e2},{\theta }_{e2},{\psi }_{e2}]}^T$ present poses of the left and right end effectors respectively.

Most of the system uncertainties come from the UVDMS dynamics since it is difficult to obtain the hydrodynamic and internal couplings precisely. For simplicity, this work formulates a concise Lagrangian dynamic model without considering the water velocity as follows:

$$M(q)\dot{\mathit{\zeta }}+C(q,\mathit{\zeta })\mathit{\zeta }+D(q,\mathit{\zeta })\mathit{\zeta }+g(q,{\mathit{\eta }}_2)=\tau ,$$

(3)

where $M(q)\in {\mathbf{\Re }}^{18\times 18}$ represents the inertial matrix consisting of both the vehicle’s inertia and the manipulators’ inertia on the diagonal, while other matrix elements represent the coupling factors. Similarly, the added Coriolis and centripetal matrix $C(q,\mathit{\zeta })\in {\mathbf{\Re }}^{18\times 18}$ is developed in a compact manner, as well as the damping and hydrodynamic lift matrix $D(q,\mathit{\zeta })\in {\mathbf{\Re }}^{18\times 18}$. $\tau ={[{\tau }_v,{\tau }_q]}^T$ and $g(q,{\mathit{\eta }}_2)$ denote the input vector of the vehicle and joint torques and the restoring forces by gravity buoyancy, respectively. As a rigid body system, these matrices have the following properties: the symmetric matrix $M(q)$ is positively defined with $m_1(q)\le \Vert M(q)\Vert \le m_2(q)$, where $m_1$ and $m_2$ are positive definite functions, and $C(q,\mathit{\zeta })$ can be chosen to satisfy $x^T(\dot{M}-2C)x=0$. The dynamic model can be written as a general nonlinear differential equation $\dot{\mathit{\zeta }}=f(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)$ to simplify the subsequent derivation.

2.2. Visual Servo Model

Traditional image-based visual servo requires multiple feature points or specially shaped patterns, which are not easy to obtain or deploy underwater. Fortunately, the UUVs, especially the UVMS, are always equipped with several cameras so that fewer feature points are needed by developing a semi-stereoscopic visual system.

In this work, we consider that the UVDMS has two cameras fixed with both end effectors and a main camera fixed with the vehicle body. As long as the object feature (i.e., a single point) is within all the cameras’ field of view, the depth information can be computed by taking advantage of the transformation between the hand cameras and the body camera. Then, the visual servo model is formulated briefly to provide a command velocity for the dynamic control.

The transformation from the camera velocity in the camera frame to the image feature velocity in the image plane is described as

$$\dot{s}=L(s,z)v_c,$$

(4)

where

$$L(s,z)=\left[\begin{array}{cccccc}-\frac{{\rho }_m}{z}& 0& \frac{m}{z}& \frac{mn}{{\rho }_n}& -\frac{{\rho }_m^2+m^2}{{\rho }_m}& \frac{{\rho }_{m}n}{{\rho }_n}\\ 0& -\frac{{\rho }_n}{z}& \frac{n}{z}& \frac{{\rho }_n^2+n^2}{{\rho }_m}& -\frac{mn}{{\rho }_m}& -\frac{{\rho }_{n}m}{{\rho }_m}\end{array}\right].$$

(5)

This equation shows that the feature point $s=(m,n)$ in the image plane is driven by the camera velocity $v_c$ with the image Jacobian $L(s,z)\in {\mathbf{\Re }}^{2\times 6}$. The scalar variable $z$, z-position of the object in the camera frame, will be calculated with the Jacobian of the main camera and the camera’s internal parameters in $L$ are obtained by camera calibration. To make full use of the sensors on the UUV (i.e. IMU), an augmented feature vector (including the image features, the image depth, and the Euler angles of the end effector) and its desired form are defined as

$${\chi }_1={[m_{e1},n_{e1},z_{e1},{\varphi }_{e1},{\theta }_{e1},{\psi }_{e1}]}^T,{\chi }_1^{\ast }={[m_{e1}^{\ast },n_{e1}^{\ast },z_{e1}^{\ast },{\varphi }_{e1}^{\ast },{\theta }_{e1}^{\ast },{\psi }_{e1}^{\ast }]}^T.$$

(6)

3. Control Strategy Developments

3.1. Hybrid Visual Servo

Without loss of generality, the left manipulator is taken as an example, and it is assumed that the camera frame is fixed with the end effector frame with no translation and rotation (shown in Figure 1). By transforming the object position [x_e1,y_e1,z_e1]^T  into the end effector frame, the homogeneous transformation is formulated to obtain $z_0$.

$$\left[\begin{array}{c}{x}_{c1}\\ y_{c1}\\ z_{c1}\end{array}\right]=T_{0_1}^C{T}_{e1}^{0_1}(q_l)\left(\left[\begin{array}{c}{x}_{e1}\\ y_{e1}\\ z_{e1}\end{array}\right]\right).$$

(7)

With the image Jacobians $L_{c1}$ and $L_{e1}$ defined above, $z_{e1}$ can be calculated by the scalar function $z_{e1}(m_{e1},n_{e1},m_{c1},n_{c1},q_l)$. At the same time, we can obtain ${[{\phi }_{e1},{\theta }_{e1},{\psi }_{e1}]}^T$ using the equation ${[{\phi }_{e1},{\theta }_{e1},{\psi }_{e1}]}^T=R_I^{e1}(R_B^I,q_l){[\phi ,\theta ,\psi ]}^T$, where $R_I^{e1}$ is the rotation matrix from the inertial frame to the left end effector frame. Thus, the augmented visual servo model is developed as

$${\dot{\chi }}_1=\left[\begin{array}{c}L(s,z_{e1})\\ \begin{array}{cc}-R_I^{e1}{({\mathit{\eta }}_{ee2,1})}_3& {\mathsf{0}}_{1\times 3}\end{array}\\ \begin{array}{cc}{\mathsf{0}}_{3\times 3}& J_{k,o1}({\mathit{\eta }}_{ee2,1})\end{array}\end{array}\right]{\dot{\mathit{\eta }}}_{ee,1}=J_{{\chi }_1}{\dot{\mathit{\eta }}}_{ee,1}.$$

(8)

where ${\mathit{\eta }}_{ee2,1}$ is the orientation of the left end effector, ${R_I^{e1}}_3$ represents the third component of $R_I^{e1}$, and $J_{k,o1}$ is the Jacobian from the inertial frame to the let end effector frame. Combining Equation (8) with Equation (2), together with the right manipulator visual servo model ${\dot{\chi }}_2=J_{{\chi }_2}{\dot{\mathit{\eta }}}_{ee,2}$ and the combined image feature error $e_s={\chi }^{\ast }-\chi$, one has the transformation from the system velocity to the image error change rate.

$${\dot{e}}_s=\dot{\chi }=\left[\begin{array}{c}{\dot{\chi }}_1\\ {\dot{\chi }}_2\end{array}\right]=\left[\begin{array}{cc}{J}_{{\chi }_1}& {\mathsf{0}}_{6\times 6}\\ {\mathsf{0}}_{6\times 6}& J_{{\chi }_2}\end{array}\right]\left[\begin{array}{c}{\dot{\mathit{\eta }}}_{ee,1}\\ {\dot{\mathit{\eta }}}_{ee,2}\end{array}\right]=J_{\chi }J\mathit{\zeta }=J_{vs}\mathit{\zeta }.$$

(9)

A candidate desired velocity can be chosen as ${\mathit{\zeta }}_d=-\lambda J_{vs}^{+}{e}_s$ to drive the feature points converge to the target point exponentially, where $J_{vs}^{+}=(J_{vs}^T{J}_{vs}{)}^{-1}{J}_{vs}^T\in {\mathbf{\Re }}^{18\times 12}$ is the pseudo-inverse Jacobian matrix and $\lambda$ is a positively defined diagonal matrix. In terms of a single UUV, this velocity command is sufficient for a positioning task. However, directly using the command velocity of IBVS will not stabilize the UVDMS system for its high redundancy and coupling characteristic. The task priority solution is an effective way to deal with system redundancy by decoupling the UVDMS motions as several tasks. As the end effector visual servo model contains the image depth and Euler angles in the closed loop, that means the position and orientation of the end effector are already configured. Using the remaining freedoms for the positioning of the UUV body can solve the redundancy problem. Then, a similar formulation for the vehicle body is given by ${\dot{e}}_b={\dot{\chi }}_b=J_b{\mathit{\zeta }}_b$, where $e_b={\chi }_b^{\ast }-{\chi }_b$ is the UUV image error and ${\chi }_b={[m_b,n_b,z_b,\phi ,\theta ,\psi ]}^T$ is the augmented image features (with the vehicle Euler angles and the image depth) of the target obtained by the camera at the bottom of the vehicle. ${\chi }_b^{\ast }={[m_b^{\ast },n_b^{\ast },z_b^{\ast },{\phi }^{\ast },{\theta }^{\ast },{\psi }^{\ast }]}^T$ is the desired UUV target image feature and $J_b\in {\mathbf{\Re }}^{6\times 18}$ is the Jacobian of the vehicle positioning. Similarly, the desired velocity for UUV positioning is defined as ${\mathit{\zeta }}_{bd}=-\lambda J_b^{+}{e}_b$. It should be pointed out that $z_b$ is measured by the depth sensor (i.e., the DVL or laser sensors) instead of computing. Thus, the command velocity is described as follows.

$$\begin{array}{l}{\mathit{\zeta }}_{cmd}=J_{vs}^{+}{\mathit{\zeta }}_d+(I_{18\times 18}-J_{vs}^{+}{J}_{vs})J_b^{+}{\mathit{\zeta }}_{bd}\\ J_{vs}^{+}=W^{-1}{J}_{vs}^T{(J_{vs}{W}^{-1}{J}_{vs}^T)}^{-1}\\ J_b^{+}=W^{-1}{J}_b^T{(J_b{W}^{-1}{J}_b^T)}^{-1}\end{array},$$

(10)

In this equation, the pseudo-inverse of both Jacobian matrices is modified by a weight matrix to avoid joint limits, and the operation above projects the desired vehicle velocity onto the zero space of $J_{vs}$ as the secondary task so that the manipulator visual servo will not be affected. The command velocity will be tracked by the control method proved in the next section.

3.2. Velocity Tracking Control

As it is hard to obtain the exact dynamics of the UVDMS, the control of this kind of strong nonlinear coupled system is challenging. To achieve a good control performance, an adaptive control scheme, shown in Figure 2, is proposed to deal with the system uncertainties.

Given the measured system elements $\overline{M}(q)$, $\overline{C}(q,\mathit{\zeta })$, $\overline{D}(q,\mathit{\zeta })\dot{\mathit{\zeta }}$, and $\overline{g}(q,{\mathit{\eta }}_2)$, the control law is designed as

$$\overline{M}(q)a+\overline{C}(q,\mathit{\zeta })\mathit{\zeta }+\overline{D}(q,\mathit{\zeta })\mathit{\zeta }+\overline{g}(q,{\mathit{\eta }}_2)=\tau ,$$

(11)

where $a$ is regarded as the virtual control to be designed later. Multiplying by ${\overline{M}}^{-1}(q)$ on each side of the equation, we obtain

$$\mathit{a}={\overline{M}}^{-1}(q)\tau -\overline{C}(q,\mathit{\zeta })\mathit{\zeta }-\overline{D}(q,\mathit{\zeta })\mathit{\zeta }-\overline{g}(q,{\mathit{\eta }}_2)=\overline{f}(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2).$$

(12)

By defining the model error as $\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)=f(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)-\overline{f}(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)$, the dynamic system can be written as the feedback linearization form

$$\dot{\mathit{\zeta }}=\mathit{a}+\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2).$$

(13)

It can be seen that a properly designed $\mathit{a}$ can stabilize the system with compensation for $\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)$. Therefore, the virtual control design is given below using the reference model adaptive strategy.

$$\mathit{a}={\mathit{a}}_{\mathit{r}\mathit{m}}+{\mathit{a}}_{\mathit{l}\mathit{c}}+{\mathit{a}}_{\mathit{r}\mathit{l}},$$

(14)

where ${\mathit{a}}_{\mathit{r}\mathit{m}}$ is the dynamic of the first-order reference model ${\mathit{a}}_{\mathit{r}\mathit{m}}=K_{rm}({\mathit{\zeta }}_{cmd}-{\mathit{\zeta }}_{rm})$, with $K_{rm}$ being positively defined and ${\mathit{\zeta }}_{rm}$ as the desired tracking velocity with the same initial conditions as the system. The tracking error and its dynamic is defined as

$$\begin{array}{l}e=\left[\begin{array}{c}{\displaystyle \underset{0}{\overset{t}{\int }}({\mathit{\zeta }}_{rm}-\mathit{\zeta })d_{\tau }}\\ {\mathit{\zeta }}_{rm}-\mathit{\zeta }\end{array}\right]\\ \dot{e}=\left[\begin{array}{c}{\mathit{\zeta }}_{rm}-\mathit{\zeta }\\ a_{rm}-\dot{\mathit{\zeta }}\end{array}\right]=\left[\begin{array}{cc}\mathsf{0}& I\\ -Ki& -Kp\end{array}\right]e+\left[\begin{array}{c}\mathsf{0}\\ I\end{array}\right](a_{rl}-\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2))\\ = \Lambda e+B(a_{rl}-\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2))\end{array},$$

(15)

where $Kp$ and $Ki$ are the control parameters from the reference velocity tracking control ${\mathit{a}}_{\mathsf{l}\mathsf{c}}$, and ${\mathit{a}}_{\mathit{l}\mathit{c}}$ is the compensation signal from the actor neural network.

3.3. Actor–Critic Network

The deterministic policy gradient (DPG) method is introduced to deal with the system uncertainties. Typically, this strategy includes two neural networks, $\pi (x_a,\theta )={\theta }^T{\phi }_a(x_a)+{\epsilon }_a$ and $V_c=q(x_v,W)=W^T{\phi }_v(x_v)+{\epsilon }_v$, named the actor network and the critic network, respectively. In this work, the actor network aims to compensate $\mathrm{\Delta }(\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2)$ with $x_a={[\tau ,\mathit{\zeta },q,{\mathit{\eta }}_2]}^T$ as the action with its parameters as the states. On the other hand, the quality of the actor is evaluated by the value function, which is approximated by the critic network. For the continuous dynamic system, the penalty function in an integral form is developed as follows:

$$V_c={\displaystyle \underset{t}{\overset{\infty }{\int }}P}{d}_{\tau },P=e^T(t)Qe(t),$$

(16)

where $Q$ is a positive definite matrix, and $P$ is regarded as the reward function. The critic network ${\widehat{V}}_c(x_v,\widehat{W})={\widehat{W}}^T{\phi }_v(x_v)$ is used to approximate $V_c$. To avoid the complexity of the process, the variables of the networks are abbreviated later. According to the TD target update strategy [31], a critic error function is formulated as

$$e_v=P(t)+{\dot{\widehat{V}}}_c(x_v,\widehat{W}).$$

(17)

The goal of updating $\widehat{W}$ is to minimize the target error $E_v=1/2e_v^T{e}_v$ so that the update law based on its gradient is provided in a brief form as

$$\dot{\widehat{W}}=-{\Gamma }_v({\widehat{W}}^T{\dot{\phi }}_v+P){\dot{\phi }}_v-\beta {\Gamma }_v\widehat{W},$$

(18)

where ${\Gamma }_v$ (the learning rate) and $\beta$ are positive definite. Similarly, another error function that contains both information of ${\widehat{V}}_c$ and $\widehat{\pi }$ is described as

$$\begin{array}{l}{e}_a=\varpi {\widehat{V}}_c+({\widehat{\theta }}^T-{\theta }^T){\phi }_a\\ E_a=\frac{1}{2}{e}_a^T{e}_a\end{array}.$$

(19)

The actor error $E_a$ will reduce as long as ${\widehat{V}}_c$ reduces and $\widehat{\theta }$ is close to $\theta$. Then, the update law can be easily derived from the gradient of $E_a$ as

$$\dot{\widehat{\theta }}=-{\Gamma }_a{\phi }_a{(\varpi {\widehat{V}}_c-{\widehat{\theta }}^T{\phi }_a)}^T-\alpha {\Gamma }_a\widehat{\theta },$$

(20)

where α and ${\Gamma }_a$ have the same definition as the critic network. For further study, the parameters of the neural networks must satisfy

$$\begin{array}{l}\Vert {\phi }_a\Vert \le {\phi }_{ma},\Vert {\phi }_v\Vert \le {\phi }_{mv},\Vert {\epsilon }_a\Vert \le {\epsilon }_{ma},\Vert {\epsilon }_v\Vert \le {\epsilon }_{mv}\\ \Vert {\dot{\phi }}_a\Vert \le {{\phi }^{\prime }}_{dma},\Vert {\dot{\phi }}_v\Vert \le {{\phi }^{\prime }}_{\mathrm{d}mv},\Vert W\Vert \le W_m,\Vert \theta \Vert \le {\theta }_m\end{array}.$$

(21)

Lemma 1. 

Considering the update law given in (18) and (20), the neural network weight errors $\tilde{W}=\widehat{W}-W$ and  $\tilde{\theta }=\widehat{\theta }-\theta$ are bounded by the compact sets

$$\begin{array}{l}{\Phi }_v=\left\{\tilde{W}\right|\Vert \tilde{W}\Vert \le {\delta }_c\}\\ {\Phi }_a=\left\{\tilde{\theta }\right|\Vert \tilde{\theta }\Vert \le {\delta }_a\}\\ {\delta }_c\ge \frac{W_m\left({\phi }_{dmv}^2+{\phi }_{dmv}{\phi }_{mv}\right)}{\sqrt{2({\phi }_{dmv}^2+\beta -1)}}\\ {\delta }_a\ge \sqrt{\frac{2{\varpi }^T\varpi ({\delta }_c^2+W_m^2){\phi }_{mv}^2+{\theta }_m^2({\phi }_{am}^2+1/2\alpha )}{{\phi }_{ma}^2+\alpha }}\end{array}.$$

(22)

The proof of Lemma 1 using the assumptions in Equation (21) and the two Lyapunov functions in Equation (23) is similar to that in [32]. This work will directly use this conclusion.

$$\begin{array}{l}{V}_1=\mathrm{tr}({\tilde{W}}^T{\Gamma }_v^{-1}\tilde{W})\\ V_2=\mathrm{tr}({\tilde{\theta }}^T{\Gamma }_a^{-1}\tilde{\theta })\end{array}.$$

(23)

3.4. Stability Analysis

By substituting the actor network $\pi (x_a,\theta )={\theta }^T{\varphi }_a(x_a)+{\epsilon }_a$ into Equation (15), the error dynamic can be written as

$$\begin{array}{l}\dot{e}=\Lambda e+B(\widehat{\pi }-\pi -{\epsilon }_a)\\ =\Lambda e+B({\widehat{\theta }}^T{\phi }_a-{\theta }^T{\phi }_a-{\epsilon }_a)\\ =\Lambda e+B({\tilde{\theta }}^T{\phi }_a-{\epsilon }_a)\\ =\Lambda e+\Omega \\ \tilde{\theta }=\widehat{\theta }-\theta \end{array}.$$

(24)

A Lyapunov candidate of the system error can be defined as

$$V_L=\frac{1}{2}{e}^T{P}_{r}e,$$

(25)

where $P_r$ is a positive definite matrix satisfying

$${\Lambda }^T{P}_r+P_r\Lambda =-Q_r,Q_r>0.$$

(26)

Then, we obtain the derivative of $V_L$ as

$$\begin{array}{l}{\dot{V}}_L=\frac{1}{2}{\dot{e}}^T{P}_{r}e+\frac{1}{2}{e}^T{P}_r\dot{e}\\ =\frac{1}{2}{e}^T({\Lambda }^T{P}_r+P\Lambda )e+\frac{1}{2}{\Omega }^T{P}_{r}e+\frac{1}{2}{e}^T{P}_r\Omega \\ =-\frac{1}{2}{e}^T(Q_r)e+e^T{P}_r\Omega \\ =-\frac{1}{2}{e}^T(Q_r)e+e^T{P}_{r}B\tilde{\theta }{\phi }_a-e^T{P}_{r}B{\epsilon }_a\\ \le -\frac{1}{2}{\lambda }_{\min }(Q_r){\Vert e\Vert }^2+{\lambda }_{\max }(P_r)\Vert e\Vert ({\epsilon }_{ma}+{\delta }_a)\\ =-\frac{1}{2}\Vert e\Vert ({\lambda }_{\min }(Q_r)\Vert e\Vert +{\lambda }_{\max }(P_r)({\epsilon }_{ma}+{\delta }_a))\end{array}.$$

(27)

In the inequality above, Lemma 1 is used. Then we have ${\dot{V}}_L\le 0$ when

$$\Vert e\Vert \le \frac{2{\lambda }_{\max }(P_r)({\epsilon }_{ma}+{\delta }_a)}{{\lambda }_{\min }(Q_r)}.$$

(28)

This means that $e$ is uniformly ultimately bounded, and the convergence speed depends on the minimum eigenvalue of $Q_r$. Moreover, the more accurate the neural network’s approximations are, the better the velocity-tracking controller performs.

4. Simulation Experiments

This section shows the simulation experiments to test the performance of the proposed visual servo control method for UVDMSs using Matlab R2023b and Unity 2022 The dynamic parameters of the UUV model shown in Figure 1 are listed as follows:

$$\begin{array}{l}{M}_{RB}=diag\{106,106,106,9.5,10.2,12\}\\ M_{AM}=diag\{63,32.2,31.3,5.6,3.7,3.4\}\\ D(v)=diag\{110+90|v|,90+90|v|,140+110|v|,20+8|v|,13+10|v|,17+12|v\left|\right\}\end{array}$$

The weight of the vehicle is 106 kg, while the buoyant force is 1058 N. The centers of the gravity and the buoyancy are $r_G=[0,0,0.1]$ and $r_B=[0,0,-0.1]$. The manipulator’s geometrical parameters are shown in Figure 3 and

Table 1. DH parameters of the underwater manipulator.

i	a	alpha	d	θ
1	0	−90	0.193	q1
2	0	90	0	q2
3	0	−90	0.2805	q3
4	0	90	0	q4
5	0	−90	0.2805	q5
6	0	90	0	q6

.

Inspired by simurv 4.0 [17], the dynamics of each single rigid body link and the UUV are projected into the generalized velocity coordinates to be added together as the generalized forces of 18 dimensions, which formulates the dynamic model of the UVDMS. Using the Jacobians and DH functions from simurv 4.0, the transformation between different frames can be easily achieved when developing the kinematics. The model parameters of the UUV are computed through Creo 10.0 (designing all models and measuring moment of inertial of the rigid bodies) and Ansys 2019 R3 (Computed Fluid Dynamics module of the Ansys Workbench is used for the identification of UUV hydrodynamic parameters).

$\mathit{\eta }=[0.5,0.2,0.9,0,0,0]$ and $q=[0,30,5,30,0,0,0,30,5,30,0,0]$ are the initial conditions of the states. The target coordinates in the inertial frame are $t_{a1}=[1.5,0.3,3]$ and $t_{a2}=[1.5,0.7,3]$. The desired position of the vehicle is ${\mathit{\eta }}_d=[1,0.5,2.6,0,0,0]$, and the desired orientations of both end effectors are towards the x-direction of the inertial coordinate with their z-axis.

The simulation results of the UVDMS position and orientation are shown in Figure 4, Figure 5, Figure 6 and Figure 7, including the pose of the vehicle in Figure 4, the angles of both manipulators in Figure 5, the position of both end effectors in Figure 6, the orientation(in the form of Euler angles) of both end effectors in Figure 7a. It can be seen that all states tend to be stable eventually. Comparing the convergence time of the vehicle position with that of the end effectors, we see that the vehicle arrived at its target earlier. That is, before the visual servoing, the vehicle should be driven to the working space by changing the task’s priority. The vehicle Euler angles show that the angles of the roll and pitch change more significantly than the yaw angle, which is caused by the floating-based operation with gravity changing during the movement. Similarly, this phenomenon also occurred in the figure of the velocities and torques below since the controller is trying to restore the orientation. In Figure 5, the angles of all joints change smoothly without exceeding the joint limits. In addition, Figure 6 indicates that the task priority strategy results in smoother curves of the end effector than that of the UUV.

The UVDMS velocity are shown in Figure 7b, Figure 8 and Figure 9, where Figure 7b shows the linear velocities of the end effectors, Figure 8 shows the linear and angular velocities of the vehicle, and the angular velocity of all manipulator joints are in Figure 8. From the velocity figures, it can be observed that most velocities are within the limits and running safely. Then, the command torques of the vehicle and manipulators are shown in Figure 10 and Figure 11. It can be seen that the command torques are always working until the simulation stops. That is, during the dynamic positioning, the open loop system is far away from its equilibrium with two manipulators forward. This is also proved by the pitch velocity and y-torque curves, which have larger values than others. The joint torques of both manipulators show that the joints closer to the base require more torques for moving. It should be noted that to compensate for gravity, several torques (joints 2–4) maintain non-zero values in the end.

At last, in Figure 12a, the compensation signals, divided into three parts, from the actor neural network are plotted, which indicates the system error (including the disturbance). It is assumed that the system uncertainty is bounded and does not change drastically when the estimated model is close enough to the real system model. So, we use RBF neural networks as the actor and critic networks for their infinite approximation ability and simple configurations. The base functions are Gaussian basis functions with the number of neurons as 100. A direct influence of the actor–critic networks on the control performance is that the parameter adjustment of the linear controller is easier. The simulation data have been displayed in Unity 2022 with the lightweight UVDMS designed by the author to test the control strategy as well as to avoid joint collisions and singularities. As shown in Figure 12b, the UVDMS successfully tracked the target features and positioned itself by the side of the shipwrecks. The purple trajectories show that the vehicle center and both end effectors move smoothly.

5. Conclusions

A reinforcement-learning-based adaptive control for the visual servoing of a UVDMS equipped with two six-dof manipulators is developed in this work: (a) Different from the classical IBVS schemes, the proposed hybrid visual servo takes advantage of the multiple cameras and other sensors to obtain the image depth such that fewer target image features are needed. (b) The command velocity is computed through a kinematic controller using the task priority method, considering both the positioning of the vehicle and the visual servo command. (c) In addition, a DPG strategy is used to design an actor–critic method to deal with the system uncertainties. The model uncertainty is compensated by the actor neural network, while the critic neural network evaluates the performance of the actor. The critic network is updated by the gradient of the velocity tracking error, and the actor is adjusted by the critic network. (d) Also, it is proved that the tracking error of the velocity is ultimately bounded using the Lyapunov method. (e) At last, the simulation of the UVDMS using Matlab and Unity shows a good performance of the proposed strategy.

This approach aims to provide a dynamic positioning method for UVDMSs’ tasks in a relatively stable local environment. However, the effects of currents, joint dynamics, thruster dynamics, low-quality underwater images, unknown environments, and low-precision sensor measurements (especially the linear velocity of UUV) are not taken into account. Additionally, collision avoidance (especially the mutual manipulators’ collision avoidance problem) and cooperative control are not involved.

Control and planning of the UVMS, especially the UVDMS, are challenging. Due to the coupling and uncertainty problems, designing a universal high-performance controller is far more difficult than for manipulators on land or UUVs. At the same time, the demand for ocean exploration makes the research of UVDMS and UVMS promising. Therefore, the follow-up study of this work will continue by focusing on the motion planning and visual control of the UVDMS.