Automated Multiple Concrete Damage Detection Using Instance Segmentation Deep Learning Model
Abstract
:1. Introduction
2. Mask and Region-Based Convolutional Neural Network (Mask R-CNN)
2.1. Feature Extraction by Convolutional Neural Network and Feature Pyramid Network
2.2. Region Proposal Network
2.3. Object Classification and Bounding Box Refinement
2.4. Mask Branch
2.5. Loss Functions
2.6. Model Modification for Optimal Training of Mask R-CNN
3. Training Mask R-CNN for Multiple Concrete Damage Detection
4. Experimental Evaluation
4.1. Damage Detection Results
4.2. Evaluation Using Performance Measures
4.3. Two Possible Methods for Improved Accuracy
5. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- ASCE’s 2017 Infrastructure Report Card | GPA: D+. Available online: https://www.infrastructurereportcard.org/tag/2020/ (accessed on 12 November 2020).
- Bridge Inspector Dies after Cherry Picker Accident—CBS Pittsburgh. Available online: https://pittsburgh.cbslocal.com/2013/06/19/bridge-inspector-in-critical-condition-after-cherry-picker-accident/ (accessed on 12 November 2020).
- Worker Killed in Bridge Inspection Accident—NBC Connecticut. Available online: https://www.nbcconnecticut.com/on-air/as-seen-on/worker-killed-in-accident-with-snooper-truck_hartford/51812/ (accessed on 12 November 2020).
- Abdel-Qader, I.; Abudayyeh, O.; Kelly, M.E. Analysis of Edge-Detection Techniques for Crack Identification in Bridges. J. Comput. Civ. Eng. 2003, 17, 255–263. [Google Scholar] [CrossRef]
- Lecompte, D.; Vantomme, J.; Sol, H. Crack Detection in a Concrete Beam using Two Different Camera Techniques. Struct. Health Monit. Int. J. 2006, 5, 59–68. [Google Scholar] [CrossRef]
- Yamaguchi, T.; Nakamura, S.; Saegusa, R.; Hashimoto, S. Image-based crack detection for real concrete surfaces. IEEJ Trans. Electr. Electron. Eng. 2008, 3, 128–135. [Google Scholar] [CrossRef]
- Rabah, M.; Elhattab, A.; Fayad, A. Automatic concrete cracks detection and mapping of terrestrial laser scan data. NRIAG J. Astron. Geophys. 2013, 2, 250–255. [Google Scholar] [CrossRef] [Green Version]
- Torok, M.M.; Golparvar-Fard, M.; Kochersberger, K.B. Image-Based Automated 3D Crack Detection for Post-disaster Building Assessment. J. Comput. Civ. Eng. 2014, 28, A4014004. [Google Scholar] [CrossRef]
- Prasanna, P.; Dana, K.J.; Gucunski, N.; Basily, B.B.; La, H.M.; Lim, R.S.; Parvardeh, H. Automated Crack Detection on Concrete Bridges. IEEE Trans. Autom. Sci. Eng. 2016, 13, 591–599. [Google Scholar] [CrossRef]
- Kim, H.; Ahn, E.; Cho, S.; Shin, M.; Sim, S.H. Comparative analysis of image binarization methods for crack identification in concrete structures. Cem. Concr. Res. 2017, 99, 53–61. [Google Scholar] [CrossRef]
- Cha, Y.J.; Choi, W.; Suh, G.; Mahmoudkhani, S.; Büyüköztürk, O. Autonomous Structural Visual Inspection Using Region-Based Deep Learning for Detecting Multiple Damage Types. Comput.-Aided Civ. Infrastruct. Eng. 2018, 33, 731–747. [Google Scholar] [CrossRef]
- Wang, N.; Zhao, Q.; Li, S.; Zhao, X.; Zhao, P. Damage Classification for Masonry Historic Structures Using Convolutional Neural Networks Based on Still Images. Comput.-Aided Civ. Infrastruct. Eng. 2018, 33, 1073–1089. [Google Scholar] [CrossRef]
- Wu, W.; Qurishee, M.A.; Owino, J.; Fomunung, I.; Onyango, M.; Atolagbe, B. Coupling Deep Learning and UAV for Infrastructure Condition Assessment Automation. In Proceedings of the 2018 IEEE International Smart Cities Conference (ISC2), Kansas City, MO, USA, 16–19 September 2018; pp. 1–7. [Google Scholar] [CrossRef]
- Xue, Y.; Li, Y. A Fast Detection Method via Region-Based Fully Convolutional Neural Networks for Shield Tunnel Lining Defects. Comput.-Aided Civ. Infrastruct. Eng. 2018, 33, 638–654. [Google Scholar] [CrossRef]
- Li, S.; Zhao, X.; Zhou, G. Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network. Comput.-Aided Civ. Infrastruct. Eng. 2019, 34, 616–634. [Google Scholar] [CrossRef]
- Liu, Y.F.; Cho, S.; Fan, J.S. Concrete Crack Assessment Using Digital Image Processing and 3D Scene Reconstruction. J. Comput. Civ. Eng. 2016, 30, 1–19. [Google Scholar] [CrossRef]
- Kim, B.; Cho, S. Image-based Concrete Crack Assessment using Mask and Region-based Convolutional Neural Network. Struct. Control Health Monit. 2019, 26, e2381. [Google Scholar] [CrossRef]
- Ni, F.; Zhang, J.; Chen, Z. Zernike-moment measurement of thin-crack width in images enabled by dual-scale deep learning. Comput.-Aided Civ. Infrastruct. Eng. 2019, 34, 367–384. [Google Scholar] [CrossRef]
- Kim, B.; Cho, S. Efflorescence assessment using hyperspectral imaging for concrete structures. Smart Struct. Syst. 2018, 22, 209. [Google Scholar] [CrossRef]
- Kim, B.; Kim, D.; Cho, S. A Study on Concrete Efflorescence Assessment using Hyperspectral Camera. J. Korean Soc. Saf. 2017, 32, 98–103. [Google Scholar] [CrossRef]
- Kramar, S.; Urosevic, M.; Pristacz, H.; Mirtič, B. Assessment of limestone deterioration due to salt formation by micro-Raman spectroscopy: Application to architectural heritage. J. Raman Spectrosc. 2010, 41, 1441–1448. [Google Scholar] [CrossRef]
- González-Jorge, H.; Puente, I.; Riveiro, B.; Martínez-Sánchez, J.; Arias, P. Automatic segmentation of road overpasses and detection of mortar efflorescence using mobile LiDAR data. Opt. Laser Technol. 2013, 54, 353–361. [Google Scholar] [CrossRef]
- Dawood, T.; Zhu, Z.; Zayed, T. Machine vision-based model for spalling detection and quantification in subway networks. Autom. Constr. 2017, 81, 149–160. [Google Scholar] [CrossRef]
- German, S.; Brilakis, I.; DesRoches, R. Rapid entropy-based detection and properties measurement of concrete spalling with machine vision for post-earthquake safety assessments. Adv. Eng. Inform. 2012, 26, 846–858. [Google Scholar] [CrossRef]
- Tanaka, H.; Tottori, S.; Nihei, T. Detection of Concrete Spalling Using Active Infrared Thermography. Q. Rep. RTRI 2006, 47, 138–144. [Google Scholar] [CrossRef]
- Wang, H.W.; Chen, C.H.; Cheng, D.Y.; Lin, C.H.; Lo, C.C. A Real-Time Pothole Detection Approach for Intelligent Transportation System. Math. Probl. Eng. 2015, 2015, 1–7. [Google Scholar] [CrossRef] [Green Version]
- Mednis, A.; Strazdins, G.; Zviedris, R.; Kanonirs, G.; Selavo, L. Real time pothole detection using Android smartphones with accelerometers. In Proceedings of the 2011 International Conference on Distributed Computing in Sensor Systems and Workshops (DCOSS), Barcelona, Spain, 27–29 June 2011; pp. 1–6. [Google Scholar] [CrossRef]
- Lee, Y.I.; Kim, B.; Cho, S. Image-based spalling detection of concrete structures using deep learning. J. Korea Concr. Inst. 2018, 30, 91–99. [Google Scholar] [CrossRef]
- Talab, A.M.A.; Huang, Z.; Xi, F.; HaiMing, L. Detection crack in image using Otsu method and multiple filtering in image processing techniques. Optik 2016, 127, 1030–1033. [Google Scholar] [CrossRef]
- Yokoyama, S.; Matsumoto, T. Development of an Automatic Detector of Cracks in Concrete Using Machine Learning. Procedia Eng. 2017, 171, 1250–1255. [Google Scholar] [CrossRef]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [Green Version]
- Long, J.; Shelhamer, E.; Darrell, T. Fully Convolutional Networks for Semantic Segmentation. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar] [CrossRef] [Green Version]
- He, K.; Gkioxari, G.; Dollar, P.; Girshick, R. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar] [CrossRef]
- Lin, T.Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; Zitnick, C.L. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, Zurich, Switzerland, 6–12 September 2014; pp. 740–755. [Google Scholar] [CrossRef] [Green Version]
- Xie, S.; Girshick, R.; Dollár, P.; Tu, Z.; He, K. Aggregated residual transformations for deep neural networks. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017; pp. 5987–5995. [Google Scholar] [CrossRef] [Green Version]
- Lin, T.Y.; Dollár, P.; Girshick, R.; He, K.; Hariharan, B.; Belongie, S. Feature pyramid networks for object detection. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017; pp. 936–944. [Google Scholar] [CrossRef] [Green Version]
- Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar] [CrossRef]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. [Google Scholar]
- GitHub-waspinator/pycococreator: Helper functions to create COCO datasets. Available online: https://github.com/waspinator/pycococreator (accessed on 12 November 2020).
- Griffiths, D.; Boehm, J. Rapid Object Detection Systems, Utilising Deep Learning and Unmanned Aerial Systems (UAS) for Civil Engineering Applications. Int. Soc. Photogramm. Remote Sens. (ISPRS) 2018, 42, 391–398. [Google Scholar] [CrossRef] [Green Version]
- Oksuz, K.; Can Cam, B.; Akbas, E.; Kalkan, S. Localization Recall Precision (LRP): A New Performance Metric for Object Detection. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 504–519. [Google Scholar]
- Cai, Z.; Vasconcelos, N. Cascade R-CNN: High Quality Object Detection and Instance Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 2019. [Google Scholar] [CrossRef] [Green Version]
Image No. | Localization Purpose | Segmentation Purpose | Remarks | ||
---|---|---|---|---|---|
Precision | Recall | Precision | Recall | ||
Crack-1 | 93.75 | 100 | 89.97 | 92.37 | Figure 5a |
Crack-2 | 100 | 100 | 91.34 | 87.88 | |
Crack-3 | 97.82 | 79.16 | 86.28 | 81.81 | Figure 5b |
Crack-4 | 100 | 100 | 87.97 | 90.94 | |
Crack-5 | 100 | 100 | 89.02 | 75.15 | |
Crack: average | 98.31 | 95.83 | 88.92 | 85.63 | |
Efflorescence-1 | 100 | 100 | 95.20 | 93.65 | |
Efflorescence-2 | 100 | 100 | 93.91 | 88.73 | |
Efflorescence-3 | 50 | 100 | 78.28 | 94.8 | Figure 6b |
Efflorescence-4 | 50 | 100 | 89.33 | 98.83 | |
Efflorescence-5 | 100 | 100 | 99.02 | 99.89 | Figure 6a |
Efflorescence: average | 80 | 100 | 91.15 | 95.18 | |
Spalling/Rebar-1 | 100 | 100 | 96.16 | 99.44 | Figure 7a |
Spalling/Rebar-2 | 95.23 | 77.78 | 87.51 | 89.36 | |
Spalling/Rebar-3 | 100 | 100 | 94.66 | 91.11 | |
Spalling/Rebar-4 | 100 | 100 | 96.38 | 82.82 | |
Spalling/Rebar-5 | 66.67 | 100 | 65.56 | 83.77 | Figure 7b |
Spalling/Rebar: average | 92.37 | 95.55 | 88.05 | 89.30 | |
Collocated damages-1 | 100 | 83.33 | 87.59 | 86.01 | |
Collocated damages-2 | 100 | 66.67 | 88.44 | 68.92 | Figure 8d |
Collocated damages-3 | 87.5 | 80 | 88.65 | 84.80 | |
Collocated damages-4 | 85.71 | 100 | 67.67 | 88.93 | |
Collocated damages-5 | 100 | 100 | 90.86 | 91.72 | Figure 8a |
Collocated damages-6 | 75 | 85.71 | 74.84 | 79.48 | |
Collocated damages-7 | 79.31 | 59.25 | 83.14 | 78.85 | Figure 8c |
Collocated damages-8 | 100 | 83.33 | 92.80 | 77.70 | |
Collocated damages-9 | 83.33 | 75 | 85.58 | 84.51 | |
Collocated damages-10 | 96 | 80 | 89.47 | 89.74 | Figure 8b |
Collocated damages: average | 86.73 | 76.66 | 85.16 | 82.06 | |
Overall average | 90.41 | 90.81 | 87.24 | 87.58 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Kim, B.; Cho, S. Automated Multiple Concrete Damage Detection Using Instance Segmentation Deep Learning Model. Appl. Sci. 2020, 10, 8008. https://doi.org/10.3390/app10228008
Kim B, Cho S. Automated Multiple Concrete Damage Detection Using Instance Segmentation Deep Learning Model. Applied Sciences. 2020; 10(22):8008. https://doi.org/10.3390/app10228008
Chicago/Turabian StyleKim, Byunghyun, and Soojin Cho. 2020. "Automated Multiple Concrete Damage Detection Using Instance Segmentation Deep Learning Model" Applied Sciences 10, no. 22: 8008. https://doi.org/10.3390/app10228008