Autonomous Dam Surveillance Robot System Based on Multi-Sensor Fusion ^†

Abstract

Dams are important engineering facilities in the water conservancy industry. They have many functions, such as flood control, electric power generation, irrigation, water supply, shipping, etc. Therefore, their long-term safety is crucial to operational stability. Because of the complexity of the dam environment, robots with various kinds of sensors are a good choice to replace humans to perform a surveillance job. In this paper, an autonomous system design is proposed for dam ground surveillance robots, which includes general solution, electromechanical layout, sensors scheme, and navigation method. A strong and agile skid-steered mobile robot body platform is designed and created, which can be controlled accurately based on an MCU and an onboard IMU. A novel low-cost LiDAR is adopted for odometry estimation. To realize more robust localization results, two Kalman filter loops are used with the robot kinematic model to fuse wheel encoder, IMU, LiDAR odometry, and a low-cost GNSS receiver data. Besides, a recognition network based on YOLO v3 is deployed to realize real-time recognition of cracks and people during surveillance. As a system, by connecting the robot, the cloud server and the users with IOT technology, the proposed solution could be more robust and practical.

1. Introduction

In the water conservancy industry, there are many fundamental engineering facilities, such as dams, water and soil conservation, water transfer project, shipping project, water supply project, hydraulic power plants, irrigation facilities, etc. Big water dams are the most comprehensive facilities. They always have many functions, such as flood control, electric power generation, irrigation, water supply, shipping, etc. [1]. Therefore, the safety of big dams is crucial to cities and people around.

In the past, staff must check the environment, structure, and electromechanical facilities of dams regularly every day. This is a necessary way of keeping the dam running safely. However, it cannot realize all-weather all-day monitor and sometimes staffs are at risk since most dams are constructed in remote rural areas. Nowadays, with the great improvement of robot technology, robots have been used in public safety, security check, disaster rescue, and high-voltage lines inspection [2,3,4,5]. Many researchers and engineers are also paying attention to utilize underwater robots, unmanned aerial vehicles, unmanned surface vehicles for dams’ surveillance and inspection.

However, applications of ground mobile robots are very rare. It is not easy for a ground robot to do the dam surveillance job autonomously. Firstly, the working places for most dams are off-road terrains, which needs a small, powerful, and agile robot mechanics design. Secondly, the GNSS satellite signal cannot fully cover all working area. The robot is difficult to continue path planning without accurate position information. Besides, to make a comprehensive check for a dam, the robot needs carry kinds of sensors. The robot must be capable of real-time data analyzing. In addition to the above, the robot system has to be low cost and robust.

In the autonomous robot area, simultaneous localization and mapping (SLAM) technology has been applied in many practical robot applications, especially laser SLAM. Researchers have proposed many laser-SLAM algorithms to build 2D/3D maps and realize localization [6,7]. Visual odometer (VO) is another hot topic in this area [8,9]. Cameras are relatively cheap and visual algorithms could show better performance in some environments. Many path planning methods are also being proposed to solve the obstacle avoiding problem [10,11]. The navigation technology now is ready for mobile robot design. Developing an autonomous dam surveillance robot system presents a wide variety of challenges. The robot should move on all kinds of ground surfaces and sometimes should move on narrow spaces. GNSS signal cannot cover all the work areas and it is difficult to acquire an accurate position. Online real-time inspection should be applied to the robot and the inspection algorithms should work with small computing resources.

This paper presents a general system for the mobile robot, which includes the robot, the cloud server, and terminals. The system can watch surveillance jobs status and can control the robot remotely when an emergency. As the robot body itself, we design a small powerful wheeled skid-steering mobile platform. Compared to alternative wheel configurations such as Ackerman or axle-articulated, the skid-steering platform shows two major advantages. First, it is simple and robust in mechanical terms. Second, it provides better maneuverability, including zero-radius turning. After analyzing the kinematic model of the skid-steered robot, we can estimate wheel odometry well with an onboard IMU (inertial measurement unit).

In this paper, a low-cost LiDAR sensor is used for laser odometry. This Livox MID40 LiDAR is a high-performance low-cost LiDAR sensor easily available. However, the LiDAR only has a 38.4° FOV and has a novel non-repeat scan mode, which differs a lot with common laser rangefinders. The method LiDAR odometry and mapping (LOAM) is adopted here for odometry estimation from its point cloud data. The method here avoids high computing burden since we only use the data for odometry calculation, not for real-time mapping. Besides, the robot was equipped with a GNSS receiver and an IMU. Therefore, we have position data from different sources: the LiDAR odometry, the longitude and latitude data, the wheel odometry and IMU measured data. To build a robust and fast localization system, we designed two extended Kalman filter (EKF) nodes. The first EKF node fuses IMU data, wheel odometry, LiDAR odometry to get robot transform in the inertial frame. By running it on the robot STM32 MCU board, we can ensure the fusion has real-time performance with high frequency. The second fuses results of first node, IMU data, longitude and latitude data to get robot transform in the world frame. The second node is running on the upper PC. The longitude and latitude data would be the observer position when the GNSS signal is good. After the position is acquired, we use a dynamic window approach (DWA) algorithm to do motion planning.

The paper is organized as follows. Some related work on robot control, fusion localization, object detection methods is summarized in Section 2. The general robot design is shown in Section 3. The mechanical design and control of the mobile robot can be seen in Section 4. All mounted sensors would be introduced in Section 5. Localization solution and some practical results could be found in Section 6. The online detection method is described in Section 7. Finally, a short conclusion is given in Section 8.

2. Related Work

To solve the dam surveillance and inspecting the problem, many robotics and sensor technologies have been deployed. A GPS-based surveillance robot is proposed in [12] to monitor the exterior deformation of the dam. A mechanically scanned imaging sonar (MSIS) is used for locating the underwater robot relative position for dam wall surveillance [13]. Many researchers utilized UAVs for dam image collecting and modeling, which is low-cost and convenient [14,15].

The wheeled skid-steering mobile robot is agile and strong, but its kinematics is complex since the drift always occurs and difficult to measure [16]. In [16], an IMU-based EKF fusion method is also proposed to improve the wheel odometry. To improve real-time motion control and dead-reckoning, a method to experimentally obtain an optimized kinematic model of the robot is presented in [17]. A detail model analysis and a back-stepping controller design are proposed in [18], simulation results for trajectory tracking are good. To find the correct model parameters [19], designed analysis and experimental kinematic scheme for skid-steering wheeled robot based-on a laser scanner sensor. The practical experiments show improvement in dead-reckoning performance.

A real-time kinematic (RTK) GPS sensor can help the robot to get an accurate position. It is commonly used in wide-open space [20]. Laser SLAM is now a relatively mature way to do mapping and localization (LiDAR odometry and mapping). LOAM proposed in [21] is one of the best LiDAR SLAM methods. The LOAM was running on the Velodyne Puck LiDAR which has 16 laser lines. Recently, a novel version of LOAM is proposed in [22], which is running on a low-cost novel Livox LiDAR and shows good performance in experiments. For land robots, dead reckoning is another reliable source of odometry. A robust AHRS could improve the wheel dead reckoning well [23].

Localization is one of the most important elements in the robot system. When the GPS/GNSS signal is too weak to use, a framework combines keyframe-based visual-inertial odometry with novel geometric image-based localization is built to provide a real-time estimate of a UAV’s pose. Weak GPS feeds are used as a weak prior for suggesting loop closures [24]. A Monte Carlo Localization system that fuses wheel and visual odometry for the prediction are designed for sewer inspection robots. The update step takes into account the network topology and other active methods. The system works well in practical experiments [25]. To address existing positioning systems do not perform well in urban canyons, tunnels, and indoor parking lots for driverless cars, an EKF based multi-sensor positioning system that combines a GPS sensor, a camera, and in-vehicle sensors assisted by kinematic and dynamic vehicle models [26]. Among all the localization system, EKF is widely used and shows good performances [27,28].

For dam surveillance, people and dam crack detection are important to ensure people and dam safety. Vision-based approaches to detect, track and identify people on a mobile robot in real-time has been used for many years. Thermal and grey images are successfully used to do the people recognition job on a mobile security robot [29]. Deep Learning shows great potential for crack damage detection. A deep architecture of convolutional neural networks (CNNs) method is proposed in [30] to find concrete cracks for civil infrastructures.

Therefore, a wheeled skid-steering mobile robot is built for the dam surveillance, which carries an onboard IMU, wheel encoders, a Livox LiDAR sensor, a GNSS antenna, a monocular industrial camera, and a customized MCU controller. We proposed the robot platform controller design, odometry methods, fusion localization, crack detection network to make the whole system low-cost and robust.

3. General Design

In dam surveillance applications, there are three main modules, which are the robot itself, the cloud server and the front-end terminals. The robot is responsible for moving in the workplace, collecting monitor data and handling some online detection applications. The cloud server takes charge of data processing and transferring. The users can watch the robot status, do manual analysis, and control the robot remotely based on real-time video streaming when necessary.

The robot and the server are connected by mobile internet and WIFI. The server and the terminal users are connected by the Internet and LAN. The general structure diagram is shown in Figure 1. The robot itself and the cloud server are connected via message queuing telemetry transport protocol (MQTT). Robot navigation data can transfer to the server and the server can send job command and global navigation plan to the robot. This paper mainly focuses on the robot module.

A state machine is designed here for robot control, which includes self-check, autonomous navigation, remote control, idle, and emergency states. The robot would get into self-check state once the power is turned on. All sensors, motors, localization data, the high-level computer will then be checked. The robot would get into an idle state and wait for task command from the cloud server if the self-check is passed. Autonomous navigation would be triggered once the task and waypoints are received. The emergency state would be triggered when the robot is stuck or some sensors are running wrong. The logic of the state machine is shown in Figure 2.

4. Robot Platform

To realize high robustness and agile motion performance, the robot body is built with an all-aluminum chassis which about a total of 10 Kg weight. Two 200W DC brushless high torque motors are deployed to realize four-wheel drive. A 24 V lithium battery with 40 AH capacity is used as the power supply.

All motors are controlled by a customized MCU board, which connects the motors and drivers by CAN bus. The maximal speed is 2 m/s; the maximum payload is 40 Kg; the maximum runtime is more than 4 hours. It is suitable for rugged all-terrain operation with four off-road tires. Figure 3 shows the body design.

As shown in Figure 4, for the robot body, we will use the velocity and the angular velocity in the robot axis as state variables, i.e., ${\left[v_{x}w\right]}^T$. After the kinematic analysis of the robot, we can find the relationship between robot velocities and wheel speed.

$$\left[\begin{array}{c}{v}_x\\ w\end{array}\right]=r\frac{\pi d}{60n}\left[\begin{array}{cc}\frac{1}{2}& \frac{1}{2}\\ -\frac{1}{2c}& -\frac{1}{2c}\end{array}\right]\left[\begin{array}{c}{W}_l\\ W_r\end{array}\right]$$

(1)

where r is so called effective radius of wheels, n is the reduction ratio, 2c is a spacing wheel track.

For the robot body, it will get velocity and angular velocity command ${\left[v_x^d{w}^d\right]}^T$ from upper PC. We should control the velocities to track the command. From (1), we have

$$\{\begin{array}{c}{W}_l+W_r=\frac{2v_x^d}{r\frac{\pi d}{60n}}\\ -W_l+W_r=\frac{2w_{}^{d}c}{r\frac{\pi d}{60n}}\end{array}$$

(2)

From (2), we could calculate the desired wheel velocities easily, but the direct calculation does not work well since the model is a realistic model and the angular velocity always has a large lag. Therefore, we add an onboard IMU and use the following equation to set the wheel velocities.

$$\{\begin{array}{c}{v}_x=v_x^d\\ w=k_p{e}_w+k_i{{\displaystyle \int }}^{}{e}_w+k_d{\dot{e}}_w\end{array}$$

(3)

where $k_p,k_i,k_d$ are constants for the simple PID controller, $e_w=w^d-w^i$,$w^i$ is the feedback from the onboard IMU. The practical response could be seen in Figure 5.

It is simple to calculate the wheel odometry in the inertial frame. We drive the robot moving a square and back to the origin, the dead reckoning result is shown in Figure 6 and final position value is [0.1229–0.2654 0.03] (should be [0 0 0]^T). The whole distance is about 70 m and we did experiments 10 times, the origin RMSE (root mean square errors) is 0.3135.

$$\{\begin{array}{c}\left[\begin{array}{c}\dot{X}\\ \dot{Y}\end{array}\right]=\left[\begin{array}{cc}\cos \theta & \sin \theta \\ \sin \theta & \cos \theta \end{array}\right]\left[\begin{array}{c}{v}_x\\ v_y\end{array}\right]\\ \\ \dot{\theta =w^i}\\ v_y=-X_{ICR}·w^i\end{array}\mathrm{Z}$$

(4)

5. Sensors

The experimental robot with sensors is shown in Figure 7. From the front to the back, there are 1 GNSS receiver, a stereo camera, a Livox LiDAR, a monocular camera, a 4G LTE wireless router, an industrial computer, a GNSS antenna. An onboard MCU controller with IMU is located inside the body box. Motion control is handled by the onboard MCU which also sends odometry data to the high-level industrial computer. The navigation is handled by the computer which is connected to the cloud server.

A navigation map is built mainly by the LiDAR and odometry data using the LOAM SLAM method. The map would be regarded as prior knowledge. The monocular camera in the middle is used to transfer video streaming and do the people and crack detection jobs. The GNSS receiver with antenna can acquire accurate position data in the wide-open area. On the other hand, the LiDAR’s point cloud and cameras’ depth data are designed to be used in obstacles perception as well in the future.

6. Localization and Navigation

The navigation includes perception, localization, path planning, and a remote controller. The whole diagram is depicted in Figure 8. The mature cost grid map is used for general environment representation. The filtered point cloud and depth data would be obstacles shown on the map.

The path planning method is divided into two parts here. The global waypoints are sent to the robot by the cloud server, which is pre-defined by different jobs. The local motion planning is achieved by the DWA algorithm on the grid map.

Localization is one of the most difficult problems for dam robots since the environment is changing and the robot has to move indoor and outdoor. Using one single sensor to acquire a positon is an impossible task. Many sensors would introduce fusion problem. Extended Kalman filter (EKF) is a good way to solve this problem and has been deployed in many robot localization applications.

A two EKF node structure is proposed here to build a robust localization system. The GNSS data is transformed based on the local origin and then input to EKF global node update process. The yaw angle from the LiDAR odometry is input to the EKF update process. The result of the EKF local node is input to the EKF global predict process. The LiDAR odometry data are input to the EKF local update process. The wheel odometry data and the IMU measurement is input to the EKF local predict process. It is noted here that the local EKF node is running on the embedded MCU at 200Hz and it handles in 3D space with 16 states.

It is important to get the 3D pose information for the rubber ground surface. However, the global EKF node is running at 30Hz on the computer and it only produces in 2D space. The robot position and orientation in the map are acquired with the EKF global updating. The fusion localization structure diagram is shown in Figure 9. Where ${\mathrm{W}}_l^d,{\mathrm{W}}_l^d$ are the wheel velocity command, ${\mathrm{W}}_l^{},{\mathrm{W}}_l^{}$ are the feedback wheel velocity from encoders, ${\mathrm{V}}_x^d,{\mathsf{\omega }}_{}^d$ are the velocity command generated from motion planning, ${\mathrm{V}}_x^{},{\mathsf{\omega }\mathrm{z}}_{}^{}$ are the wheel odometry outputs, $x,y,yaw$ are the LiDAR odometry, $X,Y,Yaw$ are the final global pose estimation result.

With the kinematic models and zero-velocity constraints, we are now ready to design a kinematic-model-based robot positioning scheme for the local EKF node. The general state update equation is

$$x_t=F{x}_{t-1}+G{u}_t$$

(5)

where $x_t={[q_0{q}_1{q}_2{q}_{3}xyz{V}_x{V}_y{V}_z\delta {\phi }_{bias}\delta {\theta }_{bias}\delta {\psi }_{bias}\delta V_{xbias}\delta V_{ybias}\delta V_{zbias}]}^T$ is the state variables, $q_0{q}_1{q}_2{q}_3$ are the quaternion, $xyz$ are the position in the inertial frame, $V_x{V}_y{V}_z$ are the corresponding velocities, $\delta {\phi }_{bias}\delta {\theta }_{bias}\delta {\psi }_{bias}$ are the rotation zero-offset, $\delta V_{xbias}\delta V_{ybias}\delta V_{zbias}$ are the acceleration zero-offset. We have the following detail state update equations.

$$\left[\begin{array}{c}{q}_0\left(k+1\right)\\ q_1\left(k+1\right)\\ q_2\left(k+1\right)\\ q_3\left(k+1\right)\end{array}\right]=q(h)•\left[\begin{array}{c}{q}_0\left(k\right)\\ q_1\left(k\right)\\ q_2\left(k\right)\\ q_3\left(k\right)\end{array}\right]$$

(6)

$$q(h)=\left[\begin{array}{c}1\\ \frac{\delta \varphi }{2}-\frac{\delta \varphi {}_{bias}}{2}\\ \frac{\delta \theta }{2}-\frac{\delta {\theta }_{bias}}{2}\\ \frac{\delta \psi }{2}-\frac{\delta {\psi }_{bias}}{2}\end{array}\right]$$

(7)

where $\delta \varphi$, $\delta \theta$, $\delta \psi$ are the radian variations during $dt$.

$$\left[\begin{array}{c}{V}_x\left(k+1\right)\\ V_y\left(k+1\right)\\ V_z\left(k+1\right)\end{array}\right]=\left[\begin{array}{c}{V}_x\left(k\right)\\ V_y\left(k\right)\\ V_z\left(k\right)\end{array}\right]+\left[\begin{array}{c}{g}_x\left(k\right)\\ g_y\left(k\right)\\ g_z\left(k\right)\end{array}\right]•dt+T_b^n\delta V$$

(8)

where $g_x\left(k\right)$$g_y\left(k\right)$$g_z\left(k\right)$ are the components of gravity.

$$\delta V=\left[\begin{array}{c}\delta V_x\\ \delta V_y\\ \delta V_z\end{array}\right]-\left[\begin{array}{c}\delta V_{xbias}\\ \delta V_{ybias}\\ \delta V_{zbias}\end{array}\right]$$

(9)

where $\delta V_{xbias}$, $\delta V_{ybias}$, $\delta V_{zbias}$ are the velocity variations during $dt$.

$$\left[\begin{array}{c}x\left(k+1\right)\\ y\left(k+1\right)\\ z\left(k+1\right)\end{array}\right]=\left[\begin{array}{c}x\left(k\right)\\ y\left(k\right)\\ z\left(k\right)\end{array}\right]+\left[\begin{array}{c}{V}_x\left(k\right)\\ V_y\left(k\right)\\ V_z\left(k\right)\end{array}\right]•dt$$

(10)

$$\left[\begin{array}{c}\delta {\varphi }_{bias}(k+1)\\ \delta {\theta }_{bias}(k+1)\\ \delta {\psi }_{bias}(k+1)\end{array}\right]=\left[\begin{array}{c}\delta {\varphi }_{bias}(k)\\ \delta {\theta }_{bias}(k)\\ \delta {\psi }_{bias}(k)\end{array}\right]$$

(11)

$$\left[\begin{array}{c}\delta V_{xbias}(k+1)\\ \delta V_{ybias}(k+1)\\ \delta V_{zbias}(k+1)\end{array}\right]=\left[\begin{array}{c}\delta V_{xbias}(k)\\ \delta V_{ybias}(k)\\ \delta V_{zbias}(k)\end{array}\right]$$

(12)

The general time update equation is

$$P_k=F_{k-1}{P}_{k-1}{F}^T{}_{k-1}+G_{k-1}{Q}_{k-1}{G}^T{}_{k-1}+Q_s$$

(13)

where, $P_k$ is the covariance matrix, $F_k={\left(\frac{\partial f}{\partial x}\right)}_k$ is state Jacobian, $G_k={\left(\frac{\partial f}{\partial u}\right)}_k$ is control input Jacobian, $Q_{k-1}$ is sensor process noise, $Q_s$ is an additional process noise. The state transform matrix have

$$F_k=\left[\begin{array}{cccccccccccccccc}1& F_{0,1}& F_{0,2}& F_{0,3}& 0& 0& 0& 0& 0& 0& F_{0,10}& F_{0,11}& F_{0,12}& 0& 0& 0\\ F_{1,0}& 1& F_{1,2}& F_{1,3}& 0& 0& 0& 0& 0& 0& F_{1,10}& F_{1,11}& F_{1,12}& 0& 0& 0\\ F_{2,0}& F_{2,1}& 1& F_{2,3}& 0& 0& 0& 0& 0& 0& F_{2,10}& F_{2,11}& F_{2,12}& 0& 0& 0\\ F_{3,0}& F_{3,1}& F_{3,2}& 1& 0& 0& 0& 0& 0& 0& F_{3,10}& F_{3,11}& F_{3,12}& 0& 0& 0\\ F_{4,0}& F_{4,1}& F_{4,2}& F_{4,3}& 1& 0& 0& 0& 0& 0& 0& 0& 0& F_{4,13}& F_{4,14}& F_{4,15}\\ F_{5,0}& F_{5,1}& F_{5,2}& F_{5,3}& 0& 1& 0& 0& 0& 0& 0& 0& 0& F_{5,13}& F_{5,14}& F_{5,15}\\ F_{6,0}& F_{6,1}& F_{6,2}& F_{6,3}& 0& 0& 1& 0& 0& 0& 0& 0& 0& F_{6,13}& F_{6,14}& F_{6,15}\\ 0& 0& 0& 0& dt& 0& 0& 1& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& dt& 0& 0& 1& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& dt& 0& 0& 1& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right]$$

(14)

Therefore, we can acquire

$$F_{0,1}=\frac{\partial q_0(t+\Delta t)}{\partial q_1}=\delta \varphi {}_{bias}/2-\delta \varphi /2;$$

(15)

$$F_{0,2}=\frac{\partial q_0(t+\Delta t)}{\partial q_2}=\delta {\theta }_{bias}/2-\delta \theta /2;$$

(16)

$$F_{0,3}=\frac{\partial q_0(t+\Delta t)}{\partial q_3}=\delta {\psi }_{bias}/2-\delta \psi /2;$$

(17)

$$F_{0,10}=\frac{\partial q_0(t+\Delta t)}{\partial \delta \varphi {}_{bias}}=\frac{q_1}{2};$$

(18)

$$F_{0,11}=\frac{\partial q_0(t+\Delta t)}{\partial \delta {\theta }_{bias}}=\frac{q_2}{2};$$

(19)

$$F_{0,12}=\frac{\partial q_0(t+\Delta t)}{\partial \delta {\psi }_{bias}}=\frac{q_3}{2};$$

(20)

$$F_{1,0}=\frac{\partial q_1(t+\Delta t)}{\partial q_0}=\delta \varphi /2-\delta \varphi {}_{bias}/2;$$

(21)

$$F_{1,2}=\frac{\partial q_1(t+\Delta t)}{\partial q_2}=\delta \psi /2-\delta {\psi }_{bias}/2;$$

(22)

$$F_{1,3}=\frac{\partial q_1(t+\Delta t)}{\partial q_3}=\delta {\theta }_{bias}/2-\delta \theta /2;$$

(23)

$$F_{1,10}=\frac{\partial q_1(t+\Delta t)}{\partial \delta \varphi {}_{bias}}=-\frac{q_0}{2}$$

(24)

$$F_{1,11}=\frac{\partial q_1(t+\Delta t)}{\partial \delta {\theta }_{bias}}=\frac{q_3}{2};$$

(25)

$$F_{1,12}=\frac{\partial q_1(t+\Delta t)}{\partial \delta {\psi }_{bias}}=-\frac{q_2}{2};$$

(26)

$$F_{2,0}=\frac{\partial q_2(t+\Delta t)}{\partial q_0}=\delta \theta /2-\delta \theta {}_{bias}/2;$$

(27)

$$F_{2,1}=\frac{\partial q_2(t+\Delta t)}{\partial q_1}=\delta {\psi }_{bias}/2-\delta \psi /2;$$

(28)

$$F_{2,3}=\frac{\partial q_2(t+\Delta t)}{\partial q_3}=\delta \varphi /2-\delta {\varphi }_{bias}/2;$$

(29)

$$F_{2,10}=\frac{\partial q_2(t+\Delta t)}{\partial \delta \varphi {}_{bias}}=-\frac{q_3}{2};$$

(30)

$$F_{2,11}=\frac{\partial q_2(t+\Delta t)}{\partial \delta {\theta }_{bias}}=-\frac{q_0}{2};$$

(31)

$$F_{2,12}=\frac{\partial q_2(t+\Delta t)}{\partial \delta {\psi }_{bias}}=\frac{q_1}{2};$$

(32)

$$F_{3,0}=\frac{\partial q_3(t+\Delta t)}{\partial q_0}=\delta \psi /2-\delta {\psi }_{bias}/2;$$

(33)

$$F_{3,1}=\frac{\partial q_3(t+\Delta t)}{\partial q_1}=\delta \theta /2-\delta {\theta }_{bias}/2;$$

(34)

$$F_{3,2}=\frac{\partial q_3(t+\Delta t)}{\partial q_2}=\delta \varphi {}_{bias}/2-\delta \varphi /2;$$

(35)

$$F_{3,10}=\frac{\partial q_3(t+\Delta t)}{\partial \delta \varphi {}_{bias}}=\frac{q_2}{2};$$

(36)

$$F_{3,11}=\frac{\partial q_3(t+\Delta t)}{\partial \delta {\theta }_{bias}}=-\frac{q_1}{2};$$

(37)

$$F_{3,12}=\frac{\partial q_3(t+\Delta t)}{\partial \delta {\psi }_{bias}}=-\frac{q_0}{2};$$

(38)

$$F_{4,0}=\frac{\partial v_x(t+\Delta t)}{\partial q_0}=2\ast \mathrm{q}0\ast (\delta V_x-\delta V_{xbias})-2\ast \mathrm{q}3\ast (\delta V_y-\delta V_{ybias})+2\ast \mathrm{q}2\ast (\delta V_z-\delta V_{zbias});$$

(39)

$$F_{4,1}=\frac{\partial v_x(t+\Delta t)}{\partial q_1}=2\ast \mathrm{q}1\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}2\ast (\delta V_y-\delta V_{ybias})+2\ast \mathrm{q}3\ast (\delta V_z-\delta V_{zbias});$$

(40)

$$F_{4,2}=\frac{\partial v_x(t+\Delta t)}{\partial q_2}=2\ast \mathrm{q}1\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}2\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}0\ast (\delta V_z-\delta V_{zbias});$$

(41)

$$F_{4,3}=\frac{\partial v_x(t+\Delta t)}{\partial q_3}=2\ast \mathrm{q}1\ast (\delta V_z-\delta V_{zbias})-2\ast \mathrm{q}0\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}3\ast (\delta V_x-\delta V_{xbias});$$

(42)

$$F_{4,13}=\frac{\partial v_x(t+\Delta t)}{\partial \delta V_{xbias}}=-q{0}^2-q{1}^2+q{2}^2+q{3}^2$$

(43)

$$F_{4,14}=\frac{\partial v_x(t+\Delta t)}{\partial \delta V_{ybias}}=2\ast \mathrm{q}0\ast \mathrm{q}3-2\ast \mathrm{q}1\ast \mathrm{q}2$$

(44)

$$F_{4,15}=\frac{\partial v_x(t+\Delta t)}{\partial \delta V_{zbias}}=-2(q_0\ast q_2+q_1\ast q_3);$$

(45)

$$F_{5,0}=\frac{\partial v_y(t+\Delta t)}{\partial q_0}=2\ast \mathrm{q}3\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}0\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}1\ast (\delta V_z-\delta V_{zbias});$$

(46)

$$F_{5,1}=\frac{\partial v_y(t+\Delta t)}{\partial q_1}=2\ast \mathrm{q}2\ast (\delta V_x-\delta V_{xbias})-2\ast \mathrm{q}1\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}0\ast (\delta V_z-\delta V_{zbias});$$

(47)

$$F_{5,2}=\frac{\partial v_y(t+\Delta t)}{\partial q_2}=2\ast \mathrm{q}1\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}2\ast (\delta V_y-\delta V_{ybias})+2\ast \mathrm{q}3\ast (\delta V_z-\delta V_{zbias});$$

(48)

$$F_{5,3}=\frac{\partial v_y(t+\Delta t)}{\partial q_3}=2\ast \mathrm{q}0\ast (\delta V_x-\delta V_{xbias})-2\ast \mathrm{q}3\ast (\delta V_y-\delta V_{ybias})+2\ast \mathrm{q}2\ast (\delta V_z-\delta V_{zbias});$$

(49)

$$F_{5,13}=\frac{\partial v_y(t+\Delta t)}{\partial \delta V_{xbias}}=-2\ast \mathrm{q}0\ast \mathrm{q}3-2\ast \mathrm{q}1\ast \mathrm{q}2$$

(50)

$$F_{5,14}=\frac{\partial v_y(t+\Delta t)}{\partial \delta V_{ybias}}=-q{0}^2+q{1}^2-q{2}^2+q{3}^2$$

(51)

$$F_{5,15}=\frac{\partial v_y(t+\Delta t)}{\partial \delta V_{zbias}}=2(q_0\ast q_1-q_2\ast q_3);$$

(52)

$$F_{6,0}=\frac{\partial v_z(t+\Delta t)}{\partial q_0}=2\ast \mathrm{q}1\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}2\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}0\ast (\delta V_z-\delta V_{zbias});$$

(53)

$$F_{6,1}=\frac{\partial v_z(t+\Delta t)}{\partial q_1}=2\ast \mathrm{q}3\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}0\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}1\ast (\delta V_z-\delta V_{zbias});$$

(54)

$$F_{6,2}=\frac{\partial v_z(t+\Delta t)}{\partial q_2}=2\ast \mathrm{q}3\ast (\delta V_y-\delta V_{ybias})-2\ast \mathrm{q}0\ast (\delta V_x-\delta V_{xbias})-2\ast \mathrm{q}2\ast (\delta V_z-\delta V_{zbias});$$

(55)

$$F_{6,3}=\frac{\partial v_z(t+\Delta t)}{\partial q_3}=2\ast \mathrm{q}1\ast (\delta V_x-\delta V_{xbias})+2\ast \mathrm{q}2\ast (\delta V_y-\delta V_{ybias})+2\ast \mathrm{q}3\ast (\delta V_z-\delta V_{zbias});$$

(56)

$$F_{6,13}=\frac{\partial v_z(t+\Delta t)}{\partial \delta V_{xbias}}=2\ast \mathrm{q}0\ast \mathrm{q}2-2\ast \mathrm{q}1\ast \mathrm{q}3$$

(57)

$$F_{6,14}=\frac{\partial v_z(t+\Delta t)}{\partial \delta V_{ybias}}=-2\ast \mathrm{q}0\ast \mathrm{q}1-2\ast \mathrm{q}2\ast \mathrm{q}3$$

(58)

$$F_{6,15}=\frac{\partial v_z(t+\Delta t)}{\partial \delta V_{zbias}}=-q{0}^2+q{1}^2+q{2}^2-q{3}^2$$

(59)

The control input Jacobian matrix is

$$G_k=\left[\begin{array}{cccccc}-\frac{q1}{2}& -\frac{q2}{2}& -\frac{q3}{2}& 0& 0& 0\\ \mathrm{SG}\left(1\right)& -\frac{q3}{2}& \frac{q2}{2}& 0& 0& 0\\ \frac{q3}{2}& \mathrm{SG}\left(1\right)& -\frac{q1}{2}& 0& 0& 0\\ -\frac{q2}{2}& \frac{q1}{2}& \mathrm{SG}\left(1\right)& 0& 0& 0\\ 0& 0& 0& \mathrm{SG}\left(4\right)-\mathrm{SG}\left(3\right)-\mathrm{SG}\left(2\right)+\mathrm{SG}\left(5\right)& \mathrm{SG}\left(8\right)-2\ast \mathrm{q}0\ast \mathrm{q}3& \mathrm{SG}\left(7\right)+2\ast \mathrm{q}0\ast \mathrm{q}2\\ 0& 0& 0& \mathrm{SG}\left(8\right)+2\ast \mathrm{q}0\ast \mathrm{q}3& \mathrm{SG}\left(3\right)-\mathrm{SG}\left(2\right)-\mathrm{SG}\left(4\right)+\mathrm{SG}\left(5\right)& \mathrm{SG}\left(6\right)-2\ast \mathrm{q}0\ast \mathrm{q}1\\ 0& 0& 0& \mathrm{SG}\left(7\right)-2\ast \mathrm{q}0\ast \mathrm{q}2& \mathrm{SG}\left(6\right)+2\ast \mathrm{q}0\ast \mathrm{q}1& \mathrm{SG}\left(2\right)-\mathrm{SG}\left(3\right)-\mathrm{SG}\left(4\right)+\mathrm{SG}\left(5\right)\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\end{array}\right]$$

(60)

where

$$SG=\left[\begin{array}{c}\frac{q_0}{2}\\ q_3{}^2\\ q_2{}^2\\ q_1{}^2\\ q_0{}^2\\ {2\ast \mathrm{q}}_2{\ast \mathrm{q}}_3\\ {2\ast \mathrm{q}}_1{\ast \mathrm{q}}_3\\ {2\ast \mathrm{q}}_1{\ast \mathrm{q}}_2\end{array}\right]$$

(61)

The process noise matrix is

$$Q_k=\left[\begin{array}{cccccccccccccccc}{q}_{0,0}& SQ(9)& SQ(8)& SQ(7)& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ SQ(9)& q_{1,1}& SQ(6)& SQ(5)& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ SQ(8)& SQ(6)& q_{2,2}& SQ(4)& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ SQ(7)& SQ(5)& SQ(4)& q_{3,3}& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& q_{4,4}& \mathrm{SQ}\left(3\right)& \mathrm{SQ}\left(2\right)& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathrm{SQ}\left(3\right)& q_{5,5}& \mathrm{SQ}\left(1\right)& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& \mathrm{SQ}\left(2\right)& \mathrm{SQ}\left(1\right)& q_{6,6}& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\end{array}\right]$$

(62)

where,

$$SQ=\left[\begin{array}{c}\mathrm{dvzCov}\ast \left(\mathrm{SG}\right(6)-2\ast \mathrm{q}0\ast \mathrm{q}1)\ast \left(\mathrm{SG}\right(2)-\mathrm{SG}(3)-\mathrm{SG}(4)+\mathrm{SG}(5\left)\right)-\mathrm{dvyCov}\ast \left(\mathrm{SG}\right(6)+2\ast \mathrm{q}0\ast \mathrm{q}1)\ast \left(\mathrm{SG}\right(2)-\mathrm{SG}(3)+\mathrm{SG}(4)-\mathrm{SG}(5\left)\right)+\mathrm{dvxCov}\ast \left(\mathrm{SG}\right(7)-2\ast \mathrm{q}0\ast \mathrm{q}2)\ast \left(\mathrm{SG}\right(8)+2\ast \mathrm{q}0\ast \mathrm{q}3)\\ \mathrm{dvzCov}\ast \left(\mathrm{SG}\right(7)+2\ast \mathrm{q}0\ast \mathrm{q}2)\ast \left(\mathrm{SG}\right(2)-\mathrm{SG}(3)-\mathrm{SG}(4)+\mathrm{SG}(5\left)\right)-\mathrm{dvxCov}\ast \left(\mathrm{SG}\right(7)-2\ast \mathrm{q}0\ast \mathrm{q}2)\ast \left(\mathrm{SG}\right(2)+\mathrm{SG}(3)-\mathrm{SG}(4)-\mathrm{SG}(5\left)\right)+\mathrm{dvyCov}\ast \left(\mathrm{SG}\right(6)+2\ast \mathrm{q}0\ast \mathrm{q}1)\ast \left(\mathrm{SG}\right(8)-2\ast \mathrm{q}0\ast \mathrm{q}3)\\ \mathrm{dvzCov}\ast \left(\mathrm{SG}\right(6)-2\ast \mathrm{q}0\ast \mathrm{q}1)\ast \left(\mathrm{SG}\right(7)+2\ast \mathrm{q}0\ast \mathrm{q}2)-\mathrm{dvyCov}\ast \left(\mathrm{SG}\right(8)-2\ast \mathrm{q}0\ast \mathrm{q}3)\ast \left(\mathrm{SG}\right(2)-\mathrm{SG}(3)+\mathrm{SG}(4)-\mathrm{SG}(5\left)\right)-\mathrm{dvxCov}\ast \left(\mathrm{SG}\right(8)+2\ast \mathrm{q}0\ast \mathrm{q}3)\ast \left(\mathrm{SG}\right(2)+\mathrm{SG}(3)-\mathrm{SG}(4)-\mathrm{SG}(5\left)\right)\\ (\mathrm{dayCov}\ast \mathrm{q}1\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dazCov}\ast \mathrm{q}1\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{daxCov}\ast \mathrm{q}2\ast \mathrm{q}3)/4\\ (\mathrm{dazCov}\ast \mathrm{q}2\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{daxCov}\ast \mathrm{q}2\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dayCov}\ast \mathrm{q}1\ast \mathrm{q}3)/4\\ (\mathrm{daxCov}\ast \mathrm{q}3\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dayCov}\ast \mathrm{q}3\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dazCov}\ast \mathrm{q}1\ast \mathrm{q}2)/4\\ (\mathrm{daxCov}\ast \mathrm{q}1\ast \mathrm{q}2)/4-(\mathrm{dazCov}\ast \mathrm{q}3\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dayCov}\ast \mathrm{q}1\ast \mathrm{q}2)/4\\ (\mathrm{dazCov}\ast \mathrm{q}1\ast \mathrm{q}3)/4-(\mathrm{daxCov}\ast \mathrm{q}1\ast \mathrm{q}3)/4-(\mathrm{dayCov}\ast \mathrm{q}2\ast \mathrm{SG}(1\left)\right)/2\\ (\mathrm{dayCov}\ast \mathrm{q}2\ast \mathrm{q}3)/4-(\mathrm{daxCov}\ast \mathrm{q}1\ast \mathrm{SG}(1\left)\right)/2-(\mathrm{dazCov}\ast \mathrm{q}2\ast \mathrm{q}3)/4\\ SG{(1)}^2\\ q{1}^2\end{array}\right]$$

(63)

$$\begin{array}{ll}{q}_{0,0}=& (dayCov\ast q{2}^2)/4+(dazCov\ast q{3}^2)/4+(daxCov\ast SQ(11))/4\\ q_{1,1}=& (dazCov\ast q{2}^2)/4+(dayCov\ast q{3}^2)/4+daxCov\ast SQ(10)\\ q_{2,2}=& (daxCov\ast q{3}^2)/4+dayCov\ast SQ(10)+(dazCov\ast SQ(11))/4\\ q_{3,3}=& (daxCov\ast q{2}^2)/4+(dayCov\ast SQ(11))/4+dazCov\ast SQ(10)\\ q_{4,4}=& dvyCov\ast {(SG(8)-2\ast q0\ast q3)}^2+dvzCov\ast {(SG(7)+2\ast q0\ast q2)}^2\\ & +dvxCov\ast {(SG(2)+SG(3)-SG(4)-SG(5))}^2\\ q_{5,5}=& dvxCov\ast {(SG(8)+2\ast q0\ast q3)}^2+dvzCov\ast {(SG(6)-2\ast q0\ast q1)}^2\\ & +dvyCov\ast {(SG(2)-SG(3)+SG(4)-SG(5))}^2\\ q_{6,6}=& dvxCov\ast {(SG(7)-2\ast q0\ast q2)}^2+dvyCov\ast {(SG(6)+2\ast q0\ast q1)}^2\\ & +dvzCov\ast {(SG(2)-SG(3)-SG(4)+SG(5))}^2\end{array}$$

(64)

where, $daxCov,dayCov,dazCov$ are the radius variance, and $dvxCov,dvyCov,dvzCov$ are the velocities variance.

The Kalman feedback matrix is

$$K_k=P_k^{-}{H}_k{\left[H_k{P}_k^{-}{H}_k^T+R_k\right]}^{-1}$$

(65)

where

$$H_k={\left(\frac{\partial z_p}{\partial x}\right)}_k$$

(66)

The covariance update equation is

$$P_k^{+}=\left[I-K_k{H}_k\right]P_k^{-}$$

(67)

Finally, we could get the state update equation

$$x_k^{+}=x_k^{-}+K_{k}v$$

(68)

The global EKF node is designed in the same method, the details are omitted here. The localization result in the practical environment can be found in Figure 10. The body coordinate represents the filtered pose of the robot now. The robot was running a circle in the practical environment. We can see that the localization performs well even sometimes the GNSS receiver is out of the lock and the result is ready for navigation. The localization and path planning could fail sometimes, even if so many sensors have been deployed. In this application, we designed a remote controller module. Users can control the robot remotely with real-time video streaming.

The Figure 11 shows a picture of an experiment environment for a surveillance job, this picture is captured from the camera on the robot. In this environment, the road is narrow and the GNSS signal is weak because of many trees. However, the proposed robot system works well. The practical results could be seen in Figure 12. The goal and the blue line are the planned global path. The green line is the local path planning result. We can see the robot locate itself well and find where to go correctly.

7. Environment Inspection

The dam crack identification and pedestrian detection are the key technologies related to the safety of the dam. We use the trained model of YOLO V3 [31] to recognize them. We use the dataset online to train the YOLO V3 algorithm and then recognize the dam crack. The YOLO V3 algorithm includes a darknet-53 module, eight DBL components, three convolutional layers, two upsampling layers, and two tensor concat layers. The darknet-53 includes a DBL component and five residual learning units res1, res2, res8, res8, and res4.

The DBL contains a convolutional layer, a BN layer and a leaky ReLU. The convolutional parameters of the convolutional layer are the kernel size 3 × 3, stride 1, same padding and output channels 32. The res1, res2, res8, and res4 contain 1, 2, 8, and 4 basic units of the residual learning, respectively, and each basic unit of the residual learning contains two DBL and an identity map. The convolutional parameters of the first DBL are the kernel size 1 × 1, stride 1, and the output channels are equal to the number of basic units of the residual learning. The convolutional parameters of the second DBL are the kernel size 3 × 3, stride 2, same padding, and output channels are equal to 2 times basic units of the residual learning.

The three convolutional layers are CConv1, CConv2, and CConv3. The filter sizes of the CConv1, CConv2, and CConv3 are 512, 256, and 128 respectively. The convolutional layers CConv1, CConv2, and CConv3 respectively contain six convolutional operations. The convolutional parameters of the first convolutional operation are the convolutional kernel size 1 × 1, and the number of output channels is equal to the filter size. The convolutional parameters of the second convolutional operation are the convolutional kernel size 3 × 3, and the number of output channels is equal to 2 times the filter size. The convolutional parameters of the third convolutional operation are the convolutional kernel size 1 × 1, and the number of output channels is equal to the filter size. The convolutional parameters of the fourth convolutional operation are the convolutional kernel size 3 × 3, and the number of output channels is equal to 2 times the filter size. The convolutional parameters of the fifth convolutional operation are the convolutional kernel size 1 × 1, and the number of output channels is equal to the filter size. The convolutional parameters of the sixth convolutional operation are the convolutional kernel size 3 × 3, and the number of output channels is equal to 2 times the filter size. The upsampling layers Upsample1 and Upsample2 sample the input feature maps and the input feature maps of the concat layer into the same size.

The network structure is shown in Figure 13. This method can do the detection job at 10 Hz with less than 20% CPU occupying. Some results are shown in Figure 14 and Figure 15.

8. Conclusions

In this paper, a general robot system is proposed for dam surveillance. The robot is connected to cloud servers and terminal users by mobile internet and IoT network. Like the robot itself, this paper introduces mechanics layout, sensor selection, and the navigation method. A simple controller and a wheel odometry calculation are proposed and achieve good performance. A two-node EKF structure localization framework is proposed to solve localization problem, which fuses LiDAR SLAM, wheel odometry, IMU, and GNSS signals. For unexpected circumstances, a remote controller based on real-time video streaming is deployed as an emergency supplement. To make the whole system able to work all-time, a control state machine is also introduced. A YOLO v3 network is trained and deployed to detect dam crack and people around. From the practical experiments, this system can work well and is capable of the surveillance job for dams. Afterward, we will pay more attention to specific dams’ surveillance jobs, such as intrusion detection and dam deformation detection. We believe robots will greatly improve the work efficiency for water conservancy in the future.