A Pipeline Defect Instance Segmentation System Based on SparseInst
Abstract
:1. Introduction
1.1. Related Work
1.1.1. Traditional Computer Vision and Image Processing Techniques
1.1.2. Deep Learning Techniques
1.2. Contributions
- To address the issue of insufficient pipeline defect data, a clear image generation network for drainage pipeline defects, called Pipe-Gan-Net, is established based on StyleGAN3. This network was used to increase the number of new defect images;
- To improve the accuracy and speed of pipeline defect segmentation, a pipeline segmentation model called Pipe-Sparse-Net is proposed based on SparseInst. This model accurately predicts the regions of drainage pipeline defects;
- To further enhance the detection speed, an acceleration module called TensorRT is applied to the segmentation model.
2. Methodology
2.1. Drainage Pipeline Defect Image Generation Network Pipe-Gan-Net
2.2. Drainage Pipeline Defect Image Segmentation Network Pipe-Sparse-Net
3. Experiments and Analysis
3.1. Experimental Settings
3.2. Pipeline Dataset
3.3. Model Training and Validation
3.3.1. Parameter Settings
3.3.2. Comparison of Five Cases
3.4. Experimental Results
3.5. Comparative Experiment
4. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- 2020 Statistical Yearbook of Urban and Rural Construction, Ministry of Housing and Urban-Rural Development of the People’s Republic of China; Ministry of Housing and Urban-Rural Development: Beijing, China, 2021.
- Sun, J. Classification and Design of Trenchless Technology for Pipe Rehabilitation. China Water Wastewater 2005, 21, 3. [Google Scholar]
- Kumar, S.S.; Abraham, D.M.; Jahanshahi, M.R.; Iseley, T.; Starr, J. Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks. Autom. Constr. 2018, 91, 273–283. [Google Scholar] [CrossRef]
- Zhu, X.; Gao, J. Defect detection and repair of urban underground drainage pipeline. J. Jiangsu Vocat. Inst. Archit. Technol. 2017, 17, 3. [Google Scholar]
- Barreira, E.; de Freitas, V.P. Evaluation of building materials using infrared thermography. Constr. Build. Mater. 2007, 21, 218–224. [Google Scholar] [CrossRef]
- Sinha, K.; Fieguth, P.W. Applications, Morphological segmentation and classification of underground pipe images. Mach. Vis. Appl. 2006, 17, 21–31. [Google Scholar] [CrossRef]
- Sinha, S.K.; Fieguth, P.W. Segmentation of buried concrete pipe images. Autom. Constr. 2006, 15, 47–57. [Google Scholar] [CrossRef]
- Iyer, S.; Sinha, S.K. Segmentation of Pipe Images for Crack Detection in Buried Sewers. Comput.-Aided Civ. Infrastruct. Eng. 2006, 21, 395–410. [Google Scholar] [CrossRef]
- Li, H.; Wen, X.; Zhao, D. Pipes information recognition and 3D model reconstruction based on DWGDirect. Appl. Mech. Mater. 2013, 427–429, 2454–2457. [Google Scholar] [CrossRef]
- Huynh, P.; Ross, R.; Martchenko, A.; Devlin, J. 3D anomaly inspection system for sewer pipes using stereo vision and novel image processing. In Proceedings of the 2016 IEEE 11th Conference on Industrial Electronics and Applications (ICIEA), Hefei, China, 5–7 June 2016. [Google Scholar]
- Chen, Z.Q.; Hutchinson, T.C. Image-Based Framework for Concrete Surface Crack Monitoring and Quantification. Adv. Civ. Eng. 2010, 2010, 9–26. [Google Scholar] [CrossRef]
- Hyun-Woo, C.; Hyuk-Jin, Y.; Jae-Chan, Y. Analysis of Crack Image Recognition Characteristics in Concrete Structures Depending on the Illumination and Image Acquisition Distance through Outdoor Experiments. Sensors 2016, 16, 1646. [Google Scholar]
- Hao, X.; Zhang, G.; Ma, S. Deep Learning. Int. J. Semant. Comput. 2016, 10, 417–439. [Google Scholar] [CrossRef]
- MA, S.; Wu, Q.; Li, X.P. Deep learning with big data: State of the art and development. CAAI Trans. Intell. Syst. 2016, 11, 15. [Google Scholar]
- Xu, J.; Chen, J.; You, S.; Xiao, Z.; Lu, J. Robustness of deep learning models on graphs: A survey. AI Open 2021, 2, 69–78. [Google Scholar] [CrossRef]
- Wang, M.; Cheng, J. A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput.-Aided Civ. Infrastruct. Eng. 2020, 35, 162–177. [Google Scholar] [CrossRef]
- Pan, G.; Zheng, Y.; Guo, S.; Lv, Y. Automatic sewer pipe defect semantic segmentation based on improved U-Net. Autom. Constr. 2020, 119, 103383. [Google Scholar] [CrossRef]
- Zhou, Q.; Situ, Z.; Teng, S.; Liu, H.; Chen, W.; Chen, G. Automatic sewer defect detection and severity quantification based on pixel-level semantic segmentation. Tunn. Undergr. Space Technol. 2022, 123, 104403. [Google Scholar] [CrossRef]
- Ma, D.; Fang, H.; Wang, N.; Zhang, C.; Dong, J.; Hu, H. Automatic detection and counting system for pavement cracks based on PCGAN and YOLO-MF. IEEE Trans. Intell. Transp. Syst. 2022, 23, 22166–22178. [Google Scholar] [CrossRef]
- He, M.; Zhao, Q.; Gao, H.; Zhang, X.; Zhao, Q. Image segmentation of a sewer based on deep learning. Sustainability 2022, 14, 6634. [Google Scholar] [CrossRef]
- Li, Y.; Wang, H.; Dang, L.M.; Piran, M.J.; Moon, H. A robust instance segmentation framework for underground sewer defect detection. Measurement 2022, 190, 110727. [Google Scholar] [CrossRef]
- Myrans, J.; Everson, R.; Kapelan, Z. Automated detection of faults in sewers using CCTV image sequences. Autom. Constr. 2018, 95, 64–71. [Google Scholar] [CrossRef]
- Karras, T.; Aittala, M.; Laine, S.; Hrknen, E.; Hellsten, J.; Lehtinen, J.; Aila, T. Alias-Free Generative Adversarial Networks. Adv. Neural Inf. Process. Syst. 2021, 34, 852–863. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Adv. Neural Inf. Process. Syst. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [PubMed]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-CNN. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017. [Google Scholar]
- Bolya, D.; Zhou, C.; Xiao, F.; Lee, Y.J. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 2 November 2019; pp. 9157–9166. [Google Scholar]
- Zhang, R.; Tian, Z.; Shen, C.; You, M.; Yan, Y. Mask Encoding for Single Shot Instance Segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020. [Google Scholar]
- Tian, Z.; Shen, C.; Chen, H. Conditional convolutions for instance segmentation. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020; Proceedings, Part I 16. Springer: Berlin/Heidelberg, Germany, 2020; pp. 282–298. [Google Scholar]
- Tian, Z.; Shen, C.; Chen, H.; He, T. FCOS: Fully Convolutional One-Stage Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 2 November 2019. [Google Scholar]
- Wang, X.; Kong, T.; Shen, C.; Jiang, Y.; Li, L. SOLO: Segmenting Objects by Locations. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020. [Google Scholar]
- Wang, X.; Zhang, R.; Kong, T.; Li, L.; Shen, C. SOLOv2: Dynamic and Fast Instance Segmentation. Adv. Neural Inf. Process. Syst. 2020, 33, 17721–17732. [Google Scholar]
- Cheng, T.; Wang, X.; Chen, S.; Zhang, W.; Zhang, Q.; Huang, C.; Zhang, Z.; Liu, W. Sparse instance activation for real-time instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 14–18 June 2022; pp. 4433–4442. [Google Scholar]
- Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; Zagoruyko, S. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 213–229. [Google Scholar]
- Stewart, R.; Andriluka, M.; Ng, A.Y. End-to-end people detection in crowded scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal Loss for Dense Object Detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017. [Google Scholar]
- Milletari, F.; Navab, N.; Ahmadi, S.A. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In Proceedings of the 2016 Fourth International Conference on 3D Vision, Stanford, CA, USA, 25–28 October 2016. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: Common objects in context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, USA, 23–28 June 2014; pp. 740–755. [Google Scholar]
Category | Total | Training Set | Validation Set | Test Set |
---|---|---|---|---|
Misalignment | 675 | 415 | 131 | 129 |
Obstacle | 515 | 316 | 110 | 89 |
Leakage | 395 | 247 | 57 | 91 |
Total | 1585 | 978 | 298 | 309 |
Case | Learn Rate | Momentum | Weight Decay | mAP |
---|---|---|---|---|
1 | 1 × 10−3 | 0.9 | 5 × 10−4 | 0.906 |
2 | 5 × 10−4 | 0.8 | 1 × 10−4 | 0.907 |
3 | 5 × 10−3 | 0.85 | 5 × 10−4 | 0.905 |
4 | 1 × 10−3 | 0.8 | 1 × 10−4 | 0.911 |
5 | 1 × 10−3 | 0.9 | 1 × 10−4 | 0.901 |
6 | 5 × 10−4 | 0.9 | 5 × 10−4 | 0.914 |
7 | 5 × 10−3 | 0.9 | 5 × 10−4 | 0.910 |
8 | 5 × 10−3 | 0.8 | 1 × 10−4 | 0.904 |
9 | 5 × 10−4 | 0.85 | 5 × 10−4 | 0.896 |
10 | 1 × 10−3 | 0.85 | 1 × 10−4 | 0.909 |
Iterations | mAP | |||
---|---|---|---|---|
All | Misalignment | Obstacle | Leakage | |
5000 | 0.776 | 0.838 | 0.744 | 0.744 |
10,000 | 0.837 | 0.862 | 0.873 | 0.774 |
15,000 | 0.886 | 0.868 | 0.919 | 0.886 |
20,000 | 0.897 | 0.871 | 0.905 | 0.871 |
22,000 | 0.914 | 0.918 | 0.938 | 0.885 |
25,000 | 0.910 | 0.915 | 0.939 | 0.877 |
30,000 | 0.910 | 0.915 | 0.941 | 0.872 |
Case | mAP | |||
---|---|---|---|---|
All | Misalignment | Obstacle | Leakage | |
1 | 0.914 | 0.918 | 0.938 | 0.885 |
2 | 0.857 | 0.861 | 0.873 | 0.844 |
3 | 0.893 | 0.896 | 0.910 | 0.852 |
4 | 0.890 | 0.892 | 0.904 | 0.841 |
5 | 0.903 | 0.904 | 0.917 | 0.869 |
Type | Normal Situation | Motion Blur | ||
---|---|---|---|---|
Number of Category | mAP | Number of Category | mAP | |
All | 213 | 0.916 | 96 | 0.909 |
Misalignment | 81 | 0.921 | 47 | 0.915 |
Obstacle | 57 | 0.944 | 32 | 0.931 |
Leakage | 75 | 0.888 | 17 | 0.882 |
With/Without Tensor RT | mAP | Speed | |
---|---|---|---|
Our model | Yes | 0.914 | 56.7 fps |
Our model | No | 0.909 | 39.2 fps |
Yolact | No | 0.870 | 35.9 fps |
Condinst | No | 0.880 | 24.7 fps |
Mask R-CNN | No | 0.844 | 15.4 fps |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, N.; Zhang, J.; Song, X. A Pipeline Defect Instance Segmentation System Based on SparseInst. Sensors 2023, 23, 9019. https://doi.org/10.3390/s23229019
Wang N, Zhang J, Song X. A Pipeline Defect Instance Segmentation System Based on SparseInst. Sensors. 2023; 23(22):9019. https://doi.org/10.3390/s23229019
Chicago/Turabian StyleWang, Niannian, Jingzheng Zhang, and Xiaotian Song. 2023. "A Pipeline Defect Instance Segmentation System Based on SparseInst" Sensors 23, no. 22: 9019. https://doi.org/10.3390/s23229019
APA StyleWang, N., Zhang, J., & Song, X. (2023). A Pipeline Defect Instance Segmentation System Based on SparseInst. Sensors, 23(22), 9019. https://doi.org/10.3390/s23229019