# A Novel Visual Sensor Stabilization Platform for Robotic Sharks Based on Improved LADRC and Digital Image Algorithm

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

- An improved window function-based LADRC algorithm was proposed for mechanical stabilization, which performs better than the general LADRC algorithm in 3-D motion.
- The IMU measurement data was applied as an estimation of motion to reduce the dependence of digital stabilization on image features, which guarantees real-time performance and is more suitable for underwater application.
- A stabilization scheme combining mechanical stabilization and digital stabilization employed for small robotic fish visual sensors was proposed. Compared to existing systems, the platform possesses a more compact structure and a better stabilization effect. Experiments verified that it can significantly improve the accuracy of target detection tasks. Remarkably, it offers valuable insight into the construction of underwater visual sensing platforms and lays the foundation for an underwater visual application such as target tracking of untethered robotic fishes with a stabilization system.

## 2. Problem Formulation

## 3. Control System Design

#### 3.1. LADRC

#### 3.2. Switching System

#### 3.3. Improved LADRC

#### 4. 1-DOF Visual Sensor Stabilization System Design

#### 4.1. Mechanical Stabilization

#### 4.2. Stability Analysis

**Theorem**

**1.**

#### 4.2.1. Convergence of LESO

**Assumption**

**1.**

#### 4.2.2. Stability of the Closed-Loop System

**Assumption**

**2.**

#### 4.3. Digital Stabilization

## 5. Experimental Results

#### 5.1. Mechanical Stabilization Experiment

_{p}and k

_{d}are the proportional and differential coefficient, respectively. LADRC, the switching system, and improved LADRC were also verified to choose the best algorithm. Since the mechanical stabilization targeted the yaw direction, the experimental results were concerned with the yaw angle. The experiments were progressed through a frequency range of 0.5 Hz to 1.5 Hz and we found that the higher the frequency, the greater the differences. Thus, 1.5 Hz was chosen as an example for detailed analysis, and the results are depicted in Figure 6.

_{pmax}and y

_{pmin}are the maximum and minimum angles over a period, y

_{max}and y

_{min}are the maximum and minimum angles of the whole process, y

_{i}is the yaw angle of the camera, y

_{axis}is the angle of the target axis, n is the number of the sampling points, and N is the number of the periods.

- The PD method does not show any drift phenomenon, which demonstrates its outstanding dynamic performance. The other algorithms have a significant drift when turning, which indirectly minimizes the range. Although the tracking performance is improved with less drift phenomenon, the excessive APR reduces this advantage.
- All of the indicators for LADRC are almost the biggest compared to the other methods. This is mainly due to the slow response to the axis changes, which is clearly shown in Figure 6. The RMSE is 9.5887°, which indicates that there are many obvious abnormal points. However, it is worth noting that it has the smallest APR, which shows its better control error performance. This phenomenon suggests that the LADRC is the best model when there is only linear motion.
- Although the switching system is slightly improved compared with LADRC, there is a significant gap compared to the improved LADRC, and parameter tuning is more difficult. The stability of the switching system is closely related to tuning of the parameters, which was previously discussed in [25].
- The range, MAE and RMSE of the improved LADRC are much enhanced, compared to the traditional LADRC. Moreover, the APR is almost the same. Thus, the effectiveness of the rational use of disturbance is verified. In particular, the drift phenomenon is not apparent in this method, thus excessive changes in range are avoided.

#### 5.2. Digital Stabilization Experiment

#### 5.3. Target Recognition

## 6. Conclusions and Future Work

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Triantafyllou, M.S.; Triantafyllou, G.S. An efficient swimming machine. Sci. Am.
**1995**, 272, 64–70. [Google Scholar] [CrossRef] - Yu, J.Z.; Wang, L.; Zhao, W.; Tan, M. Optimal design and motion control of biomimetic robotic fish. Sci. China Inf. Sci.
**2008**, 51, 535–549. [Google Scholar] [CrossRef] - Zhang, S.W.; Qian, Y.; Liao, P.; Qin, F.H.; Yang, J.M. Design and control of an agile robotic fish with integrative biomimetic mechanisms. IEEE/ASME Trans. Mechatron.
**2016**, 21, 1846–1857. [Google Scholar] [CrossRef] - Yu, J.Z.; Wang, M.; Dong, H.F.; Zhang, Y.L.; Wu, Z.X. Motion control and motion coordination of bionic robotic fish: A review. J. Bionic Eng.
**2018**, 15, 579–598. [Google Scholar] [CrossRef] - Yu, J.Z.; Wu, Z.X.; Su, Z.S.; Wang, T.Z.; Qi, S.W. Motion control strategies for a repetitive leaping robotic dolphin. IEEE/ASME Trans. Mechatron.
**2019**, 24, 913–923. [Google Scholar] [CrossRef] - Yu, J.Z.; Li, X.B.; Pang, L.; Wu, Z.X. Design and attitude control of a novel robotic jellyfish capable of 3D motion. Sci. China Inf. Sci.
**2019**, 62, 194–201. [Google Scholar] [CrossRef] [Green Version] - Katzschmann, R.K.; DelPreto, J.; MacCurdy, R.; Rus, D. Exploration of underwater life with an acoustically controlled soft robotic fish. Sci. Robot.
**2018**, 3, eaar3449. [Google Scholar] [CrossRef] [Green Version] - Raj, A.; Thakur, A. Fish-inspired robots: Design, sensing, actuation, and autonomy—A review of research. Bioinspir. Biomim.
**2016**, 11, 031001. [Google Scholar] [CrossRef] - Yu, J.Z.; Wang, K.; Tan, M.; Zhang, J.W. Design and control of an embedded vision guided robotic fish with multiple control surfaces. Sci. World J.
**2014**, 2014, 631296. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Stefanini, C.; Orofino, S.; Manfredi, L.; Mintchev, S.; Marrazza, S.; Capantini, L.; Dario, P.; Assaf, T.; Sinibaldi, E.; Grillner, S.; et al. A novel autonomous, bioinspired swimming robot developed by neuroscientists and bioengineers. Bioinspir. Biomim.
**2012**, 7, 025001. [Google Scholar] [CrossRef] [PubMed] - Ricard, P.A.; Abril, L.; Ernesto, O.D.; Alejandro, A. Vision-based autonomous underwater vehicle navigation in poor visibility conditions using a model-free robust control. J. Sens.
**2016**, 2016, 8594096. [Google Scholar] - Zheng, X.W.; Wang, W.; Xie, G.M. Autonomous vision-based underwater robot competition. ILUR Trans. Sci. Eng.
**2016**, 2016, 20160401. [Google Scholar] - Lim, A.; Ramesh, B.; Yang, Y.; Xiang, C.; Gao, Z.; Lin, F. Real-time optical flow-based video stabilization for unmanned aerial vehicles. J. Real-Time Image Proc.
**2019**, 16, 1975–1985. [Google Scholar] [CrossRef] [Green Version] - Kim, S.H.; Choi, H.L. Convolutional neural network-based multi-target detection and recognition method for unmanned airborne surveillance systems. J. Aeronaut. Space Sci.
**2019**, 20, 1038–1046. [Google Scholar] [CrossRef] - Bazeille, S.; Ortiz, J.; Rovida, F.; Camurri, M.; Meguenani, A.; Caldwell, D.; Semini, C. Active camera stabilization to enhance the vision of agile legged robots. Robotica
**2017**, 35, 942–960. [Google Scholar] [CrossRef] [Green Version] - Yang, X.; Wu, Z.X.; Yu, J.Z. Design and implementation of a robotic shark with a novel embedded vision system. In Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics, Qingdao, China, 3–7 December 2016; pp. 841–846. [Google Scholar]
- Yang, X.; Wu, Z.X.; Yu, J.Z. A novel active tracking system for robotic fish based on cascade control structure. In Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics, Qingdao, China, 3–7 December 2016; pp. 749–754. [Google Scholar]
- Zhang, P.F.; Wu, Z.X.; Wang, J.; Tan, M.; Yu, J.Z. 2-DOF camera stabilization platform for robotic fish based on active disturbance rejection control. In Proceedings of the 2019 IEEE 9th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems, Suzhou, China, 29 July–2 August 2019; pp. 283–288. [Google Scholar]
- Lou, B.; Ni, Y.; Mao, M.; Wang, P.; Cong, Y. Optimization of the Kinematic Model for Biomimetic Robotic Fish with Rigid Headshaking Mitigation. Robotics
**2017**, 6, 30. [Google Scholar] [CrossRef] [Green Version] - Windau, J.; Itti, L. Multilayer real-time video image stabilization. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots & Systems, San Francisco, CA, USA, 25–30 September 2011; pp. 2397–2402. [Google Scholar]
- Han, J.Q. From PID to active disturbance rejection control. IEEE Trans. Ind. Electron.
**2009**, 56, 900–906. [Google Scholar] [CrossRef] - Sun, J.L.; Pu, Z.Q.; Yi, J.Q. Conditional disturbance negation based active disturbance rejection control for hypersonic vehicles. Control Eng. Pract.
**2019**, 84, 159–171. [Google Scholar] [CrossRef] - Chen, S.; Bai, W.; Hu, Y.; Huang, Y.; Gao, Z. On the conceptualization of total disturbance and its profound implications. Sci. China Inf. Sci.
**2020**, 63, 129201. [Google Scholar] [CrossRef] [Green Version] - Guo, Y.Y.; Jiang, B.; Zhang, Y.M. A novel robust attitude control for quadrotor aircraft subject to actuator faults and wind gusts. IEEE/CAA J. Autom. Sin.
**2018**, 5, 292–300. [Google Scholar] [CrossRef] - Li, J.; Qi, X.H.; Xia, Y.Q.; Gao, Z.Q. On linear/nonlinear active disturbance rejection switching control. Acta Autom. Sin.
**2016**, 42, 202–212. [Google Scholar] - Rodríguez, J.A.M. Online self-camera orientation based on laser metrology and computer algorithms. Opt. Commun.
**2011**, 284, 5601–5612. [Google Scholar] [CrossRef] - Gao, Z.Q. Scaling and bandwidth-parameterization based controller tuning. In Proceedings of the American Control Conference, Denver, CO, USA, 4–6 June 2003; pp. 4989–4996. [Google Scholar]
- Fareh, R.; Al-Shabi, M.; Bettayeb, M.; Ghommam, J. Robust active disturbance rejection control for flexible link manipulator. Robotica
**2019**, 38, 118–135. [Google Scholar] [CrossRef] - Guo, B.Z.; Zhao, Z.L. On the convergence of an extended state observer for nonlinear systems with uncertainty. Syst. Control Lett.
**2011**, 60, 420–430. [Google Scholar] [CrossRef] - Zhang, D.Y.; Wu, Q.H.; Yao, X.L. Bandwidth based stability analysis of active disturbance rejection control for nonlinear uncertain systems. J. Syst. Sci. Complex
**2018**, 31, 1449–1468. [Google Scholar] [CrossRef] - Zheng, Q.; Gao, L.Q.; Gao, Z.Q. On stability analysis of active disturbance rejection control for nonlinear time-varying plants with unknown dynamics. In Proceedings of the IEEE Conference on Decision & Control, Los Angeles, CA, USA, 12–14 December 2007; pp. 3501–3506. [Google Scholar]
- Kim, N.J.; Lee, H.J.; Lee, J.B. Probabilistic global motion estimation based on Laplacian two-bit plane matching for fast digital image stabilization. EURASIP J. Adv. Signal Process.
**2008**, 2008, 1–10. [Google Scholar] [CrossRef] [Green Version] - Dong, S.P.; Chi, G.A.; Tae, K.K.; Myo, T.L. Advanced digital image stabilization using similarity-constrained optimization. Multimed. Tools Appl.
**2019**, 78, 16489–16506. [Google Scholar] - Li, X.; Li, S.; Qin, H.; Hao, A. Spatiotemporal consistency-based adaptive hand-held video stabilization. Sci. China Inf. Sci.
**2020**, 63, 114101. [Google Scholar] [CrossRef] - Ertürk, S. Real-time digital image stabilization using Kalman filters. Real-Time Imaging
**2002**, 8, 317–328. [Google Scholar] [CrossRef] [Green Version] - Dong, J.; Liu, H.B. Video stablization for strict real-time applications. IEEE Trans. Circuits Syst. Video Technol.
**2017**, 27, 716–724. [Google Scholar] [CrossRef] - Ahmed, M. Digital video stabilization-review with a perspective of real time implementation. In Proceedings of the 2017 International Conference on Recent Innovations in Signal processing and Embedded Systems, Bhopal, India, 18–20 December 2017; pp. 184–189. [Google Scholar]
- Karpenko, A.; Jacobs, D.; Back, J.; Levoy, M. Digital video stabilization and rolling shutter correction using gyroscopes. Stanf. Tech. Rep.
**2011**, 3, 1–7. [Google Scholar] - Redmon, J.; Farhadi, A. YOLOv3: An incremental improvement. arXiv
**2018**, arXiv:1804.02767. [Google Scholar] - Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollär, P.; Zitnick, C. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 740–755. [Google Scholar]

**Figure 1.**The shark-like robotic fish prototype [16].

**Figure 2.**Schematic diagram of the one degree-of-freedom (1-DOF) visual sensor stabilization platform in the shark-like robotic fish.

**Figure 3.**Stabilization block diagram of the improved linear active disturbance rejection control (LADRC) control, where θ and y are measured by inertial measurement units (IMUs), respectively. One of them is fixed on the camera and the other is fixed on the head of the shark-like robotic fish.

**Figure 4.**Schematic diagram of the 1-DOF visual sensor stabilization platform in the shark-like robotic fish.

**Figure 6.**Experimental results of proportion differential (PD) control, LADRC, switching system, improved LADRC, yaw disturbance, and axis angle under the yaw sine wave disturbance with frequency of 1.5 Hz and amplitude of 30°, the roll sine wave disturbance with a frequency of 1.5 Hz an amplitude of 10°, and a turning speed of 25°/s.

**Figure 7.**Comparison of the different visual sensor stabilization platform experiments. (

**a**) The system without mechanical stabilization and digital stabilization; (

**b**) The pure 1-DOF mechanical stabilization system; and (

**c**) The system combining mechanical stabilization with digital stabilization.

**Figure 8.**An example of the target recognition experiment, in which the quadrangle is the recognition frame and 98% is the confidence.

**Figure 9.**Confidence contrast scatter diagram, where the control group is the system without any stabilization methods, the test group is the visual sensor stabilization system, and the first period is stationary.

Term | Description |
---|---|

${v}_{1}$, ${v}_{2}$ | Target Euler angle and angular velocity |

y | Euler angle measured by IMUs |

h | Sampling period |

${\beta}_{01}$, ${\beta}_{02}$, ${\beta}_{03}$ | Observer gains |

b | Input gain |

${z}_{1}$, ${z}_{2}$ | Observer angle and angular velocity |

${z}_{3}$ | Total disturbance |

${\tilde{e}}_{1}$ | Tracking deviation |

${w}_{0}$ | Bandwidth of the observer |

Method | APR (°) | Range (°) | MAE (°) | RMSE (°) |
---|---|---|---|---|

PD | 20.0357 | 23.2881 | 7.0878 | 7.5673 |

LADRC | 14.7707 | 38.5620 | 7.0657 | 9.5887 |

Switching system | 16.7520 | 36.2005 | 6.8003 | 8.7368 |

Improved LADRC | 15.6724 | 23.4746 | 5.6715 | 4.8038 |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Pan, J.; Zhang, P.; Liu, J.; Yu, J.
A Novel Visual Sensor Stabilization Platform for Robotic Sharks Based on Improved LADRC and Digital Image Algorithm. *Sensors* **2020**, *20*, 4060.
https://doi.org/10.3390/s20144060

**AMA Style**

Pan J, Zhang P, Liu J, Yu J.
A Novel Visual Sensor Stabilization Platform for Robotic Sharks Based on Improved LADRC and Digital Image Algorithm. *Sensors*. 2020; 20(14):4060.
https://doi.org/10.3390/s20144060

**Chicago/Turabian Style**

Pan, Jie, Pengfei Zhang, Jincun Liu, and Junzhi Yu.
2020. "A Novel Visual Sensor Stabilization Platform for Robotic Sharks Based on Improved LADRC and Digital Image Algorithm" *Sensors* 20, no. 14: 4060.
https://doi.org/10.3390/s20144060