AUV-Based Side-Scan Sonar Real-Time Method for Underwater-Target Detection
Abstract
:1. Introduction
- An AUV-based side-scan sonar real-time detection method for underwater targets is proposed, which consists of real-time side-scan sonar data processing, deep-learning-based underwater-target detection model constructing, and a real-time target detection method based on navigation strip images.
- To address the conflict between the requirement of high-quality imaging and the unavailability of post-event processing in the real-time AUV-based detection, we proposed a real-time SSS data processing method, including real-time decoding and data cleaning, echo intensity data conversion, automatic seabed line detection, slant range correction, radiometric distortion correction, real-time noise cancelation correction, and geocoding.
- To satisfy the requirements of high levels of accuracy and efficiency in the underwater-target detection, we proposed a DETR-YOLO for quick detection and a BHP-UNet for high-precision segmentation. In addition, a data augmentation method is proposed that uses the SSS imaging mechanism, an underwater environment, and 3D printing to obtain a sufficient number of strongly representative samples for the model training.
- Considering the large size of the real-time SSS images, we proposed a real-time method for underwater-target detection that uses navigation strip images based on sliding detection and weighted fusion of bounding boxes.
2. Materials and Methods
2.1. Real-Time AUV-Based SSS Underwater-Target Detection System and Process
2.2. Key Techniques of the Proposed Detection Method
2.2.1. Real-Time Processing Method for SSS Data
- (1)
- Real-time Decoding and Data Cleaning
- (2)
- Echo Intensity Data Conversion
- (3)
- Automatic Seabed Line Detection
- (4)
- Slant Range Correction
- (5)
- Radiometric Distortion Correction
- Take a sliding window with a width of d and a length of l, where l is twice the image scan breadth.
- Calculate the mean value of the intensities of each column echo along the track direction within the window.
- Calculate the mean value of the echo intensities within the window and use it as the basic value for normalization.
- Calculate the correction coefficient for each column as follows:
- (6)
- Real-Time Noise Cancelation Correction
- (7)
- Geocoding
2.2.2. Method for Constructing the Deep-Learning-Based Detection Model
- Implementation of the Deep-Learning-Based Detection Algorithm
- Data Augmentation Method
- (1)
- Sample Model Fabrication Based on 3D Printing
- (2)
- Sample Augmentation Based on Geometry, Shadow Transformation, and Target Embedding
- (3)
- Style Transfer Sample Augmentation Considering Noise and Texture
2.2.3. Real-Time Method for Underwater-Target Detection with Navigation Strip Images
3. Results
3.1. Detection Model Construction and Analysis
3.2. Maritime Experiments and Analysis
3.2.1. Shipwreck Target Experiment
3.2.2. Mine Target Experiment
4. Discussion
4.1. Significance of the Proposed Method
4.1.1. Data Augmentation Method
4.1.2. Deep-Learning-Based Detection Algorithm
4.1.3. Comparison with other AUV-Based Detection Methods
4.2. Limitations of the Proposed Method
4.2.1. Real-Time Processing Method for SSS Data
4.2.2. Data Augmentation Method
4.2.3. Real-Time Method for Underwater-Target Detection
5. Conclusions
- A real-time AUV-based SSS underwater-target detection method was proposed that includes the system composition and implementation process.
- A real-time processing method for SSS data, a method for constructing a deep-learning-based underwater-target detection model, and a real-time underwater-target detection method based on navigation strip images were proposed. These methods solve the three key technical problems of real-time data processing, deep-learning-based detection model construction, and real-time target detection using SSS based on an AUV.
- Through two actual maritime experiments, the real-time intelligent detection of underwater targets using an SSS device on an AUV platform was realized, proving the feasibility of the proposed method and the effectiveness of the key techniques, providing a new solution for AUV-based SSS underwater-target real-time detection.
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Cao, H. Development status of deep-sea exploration technology and equipment in China. Ship Supplies Mark. 2005, 2, 19–22. [Google Scholar]
- Wang, J.; Cao, J.; Lu, B.; He, B. Underwater Target Detection Project Equipment Application and Development Trend. China Water Transp. 2016, 11, 43–44. [Google Scholar]
- Henriksen, L. Real-time underwater object detection based on an electrically scanned high-resolution sonar. In Proceedings of the 1994 Symposium on Autonomous Underwater Vehicle Technology, Cambridge, MA, USA, 19–20 July 1994; pp. 99–104. [Google Scholar]
- Arshad, M.R. Recent advancement in sensor technology for underwater applications. Indian Mar. Sci. 2009, 38, 267–273. [Google Scholar]
- Greene, A.; Rahman, A.F.; Kline, R.; Rahman, M.S. Side scan sonar: A cost-efficient alternative method for measuring seagrass cover in shallow environments. Estuar. Coast. Shelf Sci. 2018, 207, 250–258. [Google Scholar] [CrossRef]
- Natalia, H.; Narcis, P.; Arnau, C.; Marc, C. Autonomous detection, following and mapping of an underwater chain using sonar. Ocean Eng. 2017, 130, 336–350. [Google Scholar]
- Flowers, H.J.; Hightower, J.E. A Novel Approach to Surveying Sturgeon Using Side-Scan Sonar and Occupancy Modeling. Mar. Coast. Fish. 2013, 5, 211–223. [Google Scholar] [CrossRef]
- Johnson, S.G.; Deaett, M.A. The application of automated recognition techniques to side-scan sonar imagery. IEEE J. Ocean. Eng. 1994, 19, 138–144. [Google Scholar] [CrossRef]
- Avilash, S.; Santosha, K.; Dwivedya, P. Advancements in the field of autonomous underwater vehicle. Ocean Eng. 2019, 181, 145–160. [Google Scholar]
- Yu, S.-C. Development of real-time acoustic image recognition system using by autonomous marine vehicle. Ocean Eng. 2008, 35, 90–105. [Google Scholar] [CrossRef]
- Wu, Y.; Zhao, Y.; Lang, S. Development of Autonomous Underwater Vehicles Technology. Strateg. Study CAE 2020, 22, 26–31. [Google Scholar] [CrossRef]
- Burguera, A.; Bonin-Font, F. On-Line Multi-Class Segmentation of Side-Scan Sonar Imagery Using an Autonomous Underwater Vehicle. J. Mar. Sci. Eng. 2020, 8, 557. [Google Scholar] [CrossRef]
- Rutledge, J.; Yuan, W.; Wu, J. Intelligent Shipwreck Search Using Autonomous Underwater Vehicles. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia, 21–25 May 2018; p. 15623. [Google Scholar]
- Chen, E.; Guo, J. Real time map generation using sidescan sonar scanlines for unmanned underwater vehicles. Ocean Eng. 2014, 91, 252–262. [Google Scholar] [CrossRef]
- Kondo, H.; Ura, T. Navigation of an AUV for investigation of underwater structures. Control Eng. Pract. 2004, 12, 1551–1559. [Google Scholar] [CrossRef]
- Feezor, M.D.; Sorrell, F.Y.; Blankinship, P.R.; Bellingham, J.G. Autonomous underwater vehicle homing/docking via electromagnetic guidance. IEEE J. Ocean. Eng. 2001, 26, 515–521. [Google Scholar] [CrossRef]
- Zheng, T.; Chao, S.; Zong-wei, L.; Di, M. Multi-sensor data fusion for underwater target recognition under uncertainty. In Proceedings of the IEEE 2010 2nd International Conference on Information Science and Engineering: ICISE 2010, Hangzhou, China, 4–6 December 2010; pp. 1315–1318. [Google Scholar]
- Blondel, P. The Handbook of Sidescan Sonar; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2010. [Google Scholar]
- Wang, A.X. Research on 3D Seafloor Terrian Recovery from the Side Scan Sonar Image. Ph.D. Thesis, Wuhan University, Wuhan, China, 2014; pp. 45–48. [Google Scholar]
- Chavez, P.S., Jr.; Isbrecht, J.; Galanis, P. Processing, mosaicking and management of the Monterey Bay digital sidescan-sonar images. Mar. Geol. 2002, 181, 305–315. [Google Scholar] [CrossRef]
- Chang, Y.C.; Hsu, S.K.; Tsai, C.H. Side-scan sonar image processing: Correcting Brightness Variation and Patching Gaps. Mar. Sci. Technol.-Taiw. 2010, 18, 785–789. [Google Scholar]
- Wilken, D.; Feldens, P.; Wunderlich, T.; Heinrich, C. Application of 2D Fourier filtering for elimination of stripe noise in side-scan sonar mosaics. Geo-Mar. Lett. 2012, 32, 337–347. [Google Scholar] [CrossRef]
- Burguera, A.; Oliver, G. High-resolution underwater mapping using side-scan sonar. PLoS ONE 2016, 11, 0146396. [Google Scholar] [CrossRef] [Green Version]
- Yan, J.; Meng, J.; Zhao, J. Real-time bottom tracking using side scan sonar data through one-dimensional convolutional neural networks. Remote Sens. 2019, 12, 37. [Google Scholar] [CrossRef] [Green Version]
- Cho, H.; Yu, S.-C. Real-time sonar image enhancement for AUV-based acoustic vision. Ocean Eng. 2015, 104, 568–579. [Google Scholar] [CrossRef]
- Vasamsetti, S.; Mittal, N.; Neelapu, B.C.; Sardana, H.K. Wavelet based perspective on variational enhancement technique for underwater imagery. Ocean Eng. 2017, 141, 88–100. [Google Scholar] [CrossRef]
- Corchs, S.; Schettini, R. Underwater image processing: State of the art of restoration and image enhancement methods. EURASIP Adv. J. Signal Process. 2010, 2018, 746052. [Google Scholar]
- Nayak, N.; Nara, M.; Gambin, T.; Wood, Z.; Clark, C.M. Machine learning techniques for AUV side-scan sonar data feature extraction as applied to intelligent search for underwater archaeological sites. Field Serv. Robot. 2021, 16, 219–233. [Google Scholar]
- Yang, F.; Du, Z.; Wu, Z. Object Recognizing on Sonar Image Based on Histogram and Geometric Feature. Mar. Sci. Bull. 2006, 25, 64–69. [Google Scholar]
- Langner, F.; Knauer, C.; Jans, W.; Ebert, A. Side Scan Sonar Image Resolution and Automatic Object Detection, Classification and Identification. In Proceedings of the OCEANS 2009—Europe Conference, Bremen, Germany, 11–14 May 2009. [Google Scholar]
- Xiao, W.; Zhao, J.; Zhu, B.; Jiang, T.; Qin, T. A Side Scan Sonar Image Target Detection Algorithm Based on a Neutrosophic Set and Diffusion Maps. Remote Sens. 2018, 10, 295. [Google Scholar]
- Fakiris, E.; Papatheodorou, G.; Geraga, M.; Ferentinos, G. An Automatic Target Detection Algorithm for Swath Sonar Backscatter Imagery, Using Image Texture and Independent Component Analysis. Remote Sens. 2016, 8, 373. [Google Scholar] [CrossRef] [Green Version]
- Dura, E.; Zhang, Y.; Liao, X.J.; Dobeck, G.J.; Carin, L. Active learning for detection of mine-like objects in side-scan sonar imagery. IEEE J. Ocean. Eng. 2005, 30, 360–371. [Google Scholar] [CrossRef] [Green Version]
- Song, Y.; He, B.; Liu, P. Real-Time Object Detection for AUVs Using Self-Cascaded Convolutional Neural Networks. IEEE J. Ocean. Eng. 2021, 46, 56–67. [Google Scholar] [CrossRef]
- Topple, J.M.; Fawcett, J.A. MiNet: Efficient Deep Learning Automatic Target Recognition for Small Autonomous Vehicles. IEEE Geosci. Remote Sens. Lett. 2020, 16, 186–197. [Google Scholar] [CrossRef]
- Huo, G.Y.; Yang, S.X.; Li, Q.W.; Zhou, Y. A Robust and Fast Method for Sidescan Sonar Image Segmentation Using Nonlocal Despeckling and Active Contour Model. IEEE Trans. Cybern. 2017, 47, 855–872. [Google Scholar] [CrossRef]
- Feldens, P.; Darr, A.; Feldens, A.; Tauber, F. Detection of Boulders in Side Scan Sonar Mosaics by a Neural Network. Geosciences 2019, 4, 159. [Google Scholar] [CrossRef] [Green Version]
- Zheng, L.; Tian, K. Detection of small objects in sidescan sonar images based on POHMT and Tsallis entropy. Signal Process. 2018, 6, 142168–142177. [Google Scholar] [CrossRef]
- Li, C.L.; Ye, X.F.; Cao, D.X.; Hou, J.; Yang, H.B. Zero shot objects classification method of side scan sonar image based on synthesis of pseudo samples. Appl. Acoust. 2021, 173, 107691. [Google Scholar] [CrossRef]
- Bore, N.; Folkesson, J. Modeling and Simulation of Sidescan Using Conditional Generative Adversarial Network. IEEE J. Ocean. Eng. 2021, 46, 195–205. [Google Scholar] [CrossRef]
- Coiras, E.; Mignotte, P.Y.; Petillot, Y.; Bell, J.; Lebart, K. Supervised target detection and classification by training on augmented reality data. IET Radar Sonar Navig. 2007, 1, 83–90. [Google Scholar] [CrossRef] [Green Version]
- Zheng, G.; Zhang, H.; Li, Y.; Zhao, J. A Universal Automatic Bottom Tracking Method of Side Scan Sonar Data Based on Semantic Segmentation. Remote Sens. 2021, 13, 1945. [Google Scholar] [CrossRef]
- Xu, Y.; Wang, X.; Wang, K.; Shi, J.; Sun, W. Underwater sonar image classification using generative adversarial network and convolutional neural network. IET Image Process. 2020, 14, 2819–2825. [Google Scholar] [CrossRef]
- Kapetanović, N.; Mišković, N.; Tahirović, A. Saliency and Anomaly: Transition of Concepts from Natural Images to Side-Scan Sonar Images. In Proceedings of the 21st IFAC World Congress 2020, Berlin, Germany, 12–17 July 2020. [Google Scholar]
- Steiniger, Y.; Kraus, D.; Meisen, T. Survey on deep learning based computer vision for sonar imagery. Eng. Appl. Artif. Intell. 2022, 114, 105157. [Google Scholar] [CrossRef]
- Tang, Y.; Jin, S.; Bian, G.; Zhang, Y. Shipwreck Target Recognition in Side-scan Sonar Images by Improved YOLOv3 Model Based on Transfer Learning. IEEE Access 2019, 8, 173450–173460. [Google Scholar]
- Tang, Y.; Jin, S.; Bian, G.; Zhang, Y.; Li, F. The Transfer Learning with Convolutional Neural Network Method of Side-scan Sonar to Identify Wreck Images. Acta Geod. Cartogr. Sin. 2021, 50, 260–269. [Google Scholar]
- Tang, Y.; Li, H.; Zhang, W.; Bian, S.; Zhai, G.; Liu, M.; Zhang, X. Lightweight DETR-YOLO method for detecting shipwreck target in side-scan sonar. Syst. Eng. Electron. 2022, 44, 2427–2436. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 834–848. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar]
- Wang, H.; Gao, N.; Xiao, Y.; Tang, Y. Image feature extraction based on improved FCN for UUV side-scan sonar. Mar. Geophys. Res. 2020, 41, 18. [Google Scholar] [CrossRef]
- Huang, C.; Zhao, J.; Yu, Y.; Zhang, H. Comprehensive Sample Augmentation by Fully Considering SSS Imaging Mechanism and Environment for Shipwreck Detection Under Zero Real Samples. IEEE Trans. Geosci. Remote Sens. 2021, 60, 1–14. [Google Scholar] [CrossRef]
- Chen, G.Y.; Bui, T.D. Image denoising with neighbour dependency and customized wavelet and threshold. Pattern Recognit. 2005, 38, 115–124. [Google Scholar] [CrossRef]
Category | Number of Original Training Data Images | Number of Sample Augmented Data Images | Ratio of Training to Test Data |
---|---|---|---|
Shipwreck | 612 | 1200 | 4:1 |
Mine | 8 | 100 | 4:1 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB |
DETR-YOLO | 76.5 | 56.9 | 427 | 18.2 | 84.5 | 57.7 | 431 | 20.1 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | Dice/% | IOU/% | FPS | Weights/MB | Dice/% | IOU/% | FPS | Weights/MB |
BHP-UNet | 76.33 | 78.52 | 132 | 64.2 | 78.31 | 77.71 | 121 | 66.2 |
Group | Real SSS Image | Generated by Proposed Method | Generated by [52] Method |
---|---|---|---|
1 1 | 500 | 0 | 0 |
2 2 | 0 | 500 | 0 |
3 3 | 0 | 0 | 500 |
Model | AP0.5/% | AP0.5–0.95/% | Dice/% | IOU/% |
---|---|---|---|---|
DETR-YOLO-1 | 79.9 | 50.5 | - | - |
DETR-YOLO-2 | 83.1 | 54.9 | - | - |
DETR-YOLO-3 | 80.7 | 50.2 | - | - |
BHP-Unet-1 | - | - | 72.6 | 72.2 |
BHP-Unet-2 | - | - | 73.1 | 75.6 |
BHP-Unet-3 | - | - | 71.5 | 69.7 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB |
YOLOv5a | 74.2 | 47.4 | 426 | 16.1 | 81.8 | 51.6 | 430 | 18.6 |
DETR-YOLO | 76.5 | 56.9 | 427 | 18.2 | 84.5 | 57.7 | 431 | 20.1 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | Dice/% | IOU/% | FPS | Weights/MB | Dice/% | IOU/% | FPS | Weights/MB |
U-Net | 67.95 | 70.61 | 135 | 62.1 | 69.26 | 71.47 | 126 | 65.9 |
BHP-UNet | 76.33 | 78.52 | 132 | 64.2 | 78.31 | 77.71 | 121 | 66.2 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB | AP0.5/% | AP0.5–0.95/% | FPS | Weights/MB |
Transformer | 68.5 | 41.9 | 439 | 13.6 | 77.3 | 43.9 | 442 | 18.3 |
YOLOv5 | 74.2 | 47.4 | 426 | 16.1 | 81.8 | 51.6 | 430 | 18.6 |
Faster R-CNN | 74.3 | 44.2 | 408 | 30.7 | 80.31 | 49.9 | 411 | 34.6 |
DETR-YOLO | 76.5 | 56.9 | 427 | 18.2 | 84.5 | 57.7 | 431 | 20.1 |
Mine Target | Shipwreck Target | |||||||
---|---|---|---|---|---|---|---|---|
Model | Dice/% | IOU/% | FPS | Weights/MB | Dice/% | IOU/% | FPS | Weights/MB |
U-Net | 67.95 | 70.61 | 135 | 62.1 | 69.26 | 71.47 | 126 | 65.9 |
DeepLabv3+ | 73.01 | 72.58 | 95 | 100.1 | 76.80 | 75.20 | 88 | 103.6 |
BHP-UNet | 76.33 | 78.52 | 132 | 64.2 | 78.31 | 77.71 | 121 | 66.2 |
Num | Method | Algorithm | Task | Object | Accuracy | Efficiency |
---|---|---|---|---|---|---|
1 | Ours | DETR-YOLO BHP-UNet | Detection and Segmentation | Mine Target Shipwreck | 80.5% AP 78.1% IOU | 429 FPS (detection) 127 FPS (segmentation) |
2 | Son-Cheol Yu (2008) [10] | Binarization | Detection | Cubic Cylindrical | - | - |
3 | Jeffrey Rutledge (2018) [13] | Resnet | Detection | Target | 93.75% recall and 20.71% precision (data set A) 18.18%% recall and 10.43% precision (data set B) | - |
4 | Burguera et al. (2020) [12] | Fully Convolutional Neural Network | Segmentation | Rock Sand Other | 87.8% (F1 score, AUV) | 4.6 milliseconds per 100 pixels (AUV) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Tang, Y.; Wang, L.; Jin, S.; Zhao, J.; Huang, C.; Yu, Y. AUV-Based Side-Scan Sonar Real-Time Method for Underwater-Target Detection. J. Mar. Sci. Eng. 2023, 11, 690. https://doi.org/10.3390/jmse11040690
Tang Y, Wang L, Jin S, Zhao J, Huang C, Yu Y. AUV-Based Side-Scan Sonar Real-Time Method for Underwater-Target Detection. Journal of Marine Science and Engineering. 2023; 11(4):690. https://doi.org/10.3390/jmse11040690
Chicago/Turabian StyleTang, Yulin, Liming Wang, Shaohua Jin, Jianhu Zhao, Chao Huang, and Yongcan Yu. 2023. "AUV-Based Side-Scan Sonar Real-Time Method for Underwater-Target Detection" Journal of Marine Science and Engineering 11, no. 4: 690. https://doi.org/10.3390/jmse11040690
APA StyleTang, Y., Wang, L., Jin, S., Zhao, J., Huang, C., & Yu, Y. (2023). AUV-Based Side-Scan Sonar Real-Time Method for Underwater-Target Detection. Journal of Marine Science and Engineering, 11(4), 690. https://doi.org/10.3390/jmse11040690