Improved Pixel-Level Pavement-Defect Segmentation Using a Deep Autoencoder
Abstract
:1. Introduction
2. Deep Neural Network Model
2.1. Residual Blocks
2.2. Atrous (Dilated) Convolution Blocks
2.3. Attention Blocks (Attention Gates)
3. Data
3.1. CrackForest
3.2. Crack500
3.3. GAPs384
3.4. Data Preparation
4. Experiments and Evaluation
- U-Net (Baseline);
- U-Net with residual connections (ResU-Net);
- U-Net with residual connections and ASPP module (ResU-Net + ASPP);
- U-Net with residual connections and ASPP module when connected in “Waterfall” order (ResU-Net + ASPP_WF);
- U-Net with residual connections ASPP and AG modules (ResU-Net + ASPP + AG); and
- U-Net with residual connections ASPP (connected in “Waterfall” order) and AG modules (ResU-Net + ASPP_WF + AG).
5. Results
6. Discussion
Author Contributions
Funding
Conflicts of Interest
References
- Gopalakrishnan, K. Deep learning in data-driven pavement image analysis and automated distress detection: A review. Data 2018, 3, 28. [Google Scholar] [CrossRef] [Green Version]
- ImageNet Large Scale Visual Recognition Challenge (ILSVRC). Available online: http://image-net.org/challenges/LSVRC/ (accessed on 22 December 2019).
- Hwang, D.; Kim, D.E. Special features on intelligent imaging and analysis. Appl. Sci. 2019, 9, 4804. [Google Scholar] [CrossRef] [Green Version]
- Cao, W.; Liu, Q.; He, Z. Review of pavement defect detection methods. IEEE Access 2020, 8, 14531–14544. [Google Scholar] [CrossRef]
- Sy, N.T.; Avila, M.; Begot, S.; Bardet, J.C. Detection of defects in road surface by a vision system. In Proceedings of the MELECON 2008—The 14th IEEE Mediterranean Electrotechnical Conference, Ajaccio, France, 5–7 May 2008; Volume 2, pp. 847–851. [Google Scholar]
- Koch, C.; Brilakis, I. Pothole detection in asphalt pavement images. Adv. Eng. Inform. 2011, 25, 507–515. [Google Scholar] [CrossRef]
- Salari, E.; Bao, G. Automated pavement distress inspection based on 2D and 3D information. In Proceedings of the 2011 IEEE International Conference on Electro/Information Technology, Mankato, MN, USA, 15–17 May 2011 2011; pp. 2–5. [Google Scholar]
- Nisanth, A.; Mathew, A. Automated Visual Inspection of Pavement Crack Detection and Characterization. Int. J. Technol. Eng. Syst. 2014, 6, 14–20. [Google Scholar]
- Otsu, N. A Threshold Selection Method from Gray-Level Histograms. IEEE Trans. Syst. Man Cybern. 1979, 9, 62–66. [Google Scholar] [CrossRef] [Green Version]
- Danielsson, P.-E.; Seger, O. Generalized and Separable Sobel Operators. In Machine Vision for Three-Dimensional Scenes; Academic Press: San Diego, CA, USA, 1990; pp. 347–3870. [Google Scholar]
- Cord, A.; Chambon, S. Automatic Road Defect Detection by Textural Pattern Recognition Based on AdaBoost. Comput. Civ. Infrastruct. Eng. 2012, 27, 244–259. [Google Scholar] [CrossRef]
- Schapire, R.E. A Brief Introduction to Boosting. In Proceedings of the 16th International Joint Conference on Artificial Intelligence—Volume 2; Morgan Kaufmann Publishers Inc.: San Francisco, CA, USA, 1999; pp. 1401–1406. [Google Scholar]
- Zhang, L.; Yang, F.; Zhang, Y.; Zhu, Y.J. Road crack detection using deep convolutional neural network. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 3708–3712. [Google Scholar]
- Jenkins, M.D.; Carr, T.A.; Iglesias, M.I.; Buggy, T.; Morison, G. A Deep Convolutional Neural Network for Semantic Pixel-Wise Segmentation of Road and Pavement Surface Cracks. In Proceedings of the 2018 26th European Signal Processing Conference (EUSIPCO), Rome, Italy, 3–7 September 2018; pp. 2134–2138. [Google Scholar]
- Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 5–9 October 2015; pp. 234–241. [Google Scholar]
- Shi, Y.; Cui, L.; Qi, Z.; Meng, F.; Chen, Z. Automatic road crack detection using random structured forests. IEEE Trans. Intell. Transp. Syst. 2016, 17, 3434–3445. [Google Scholar] [CrossRef]
- Cui, L.; Qi, Z.; Chen, Z.; Meng, F.; Shi, Y. Pavement Distress Detection Using Random Decision Forests; Springer: Sydney, Australia, 2015; pp. 95–102. [Google Scholar]
- Yang, F.; Zhang, L.; Yu, S.; Prokhorov, D.; Mei, X.; Ling, H. Feature Pyramid and Hierarchical Boosting Network for Pavement Crack Detection. IEEE Trans. Intell. Transp. Syst. 2020, 21, 1525–1535. [Google Scholar] [CrossRef] [Green Version]
- Fan, R.; Bocus, M.J.; Zhu, Y.; Jiao, J.; Wang, L.; Ma, F.; Cheng, S.; Liu, M. Road crack detection using deep convolutional neural network and adaptive thresholding. In Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France, 9–12 June 2019; pp. 474–479. [Google Scholar]
- Eisenbach, M.; Stricker, R.; Seichter, D.; Amende, K.; Debes, K.; Sesselmann, M.; Ebersbach, D.; Stoeckert, U.; Gross, H. How to get pavement distress detection ready for deep learning? A systematic approach. In Proceedings of the 2017 International Joint Conference on Neural Networks IJCNN, Anchorage, AK, USA, 14–19 May 2017; pp. 2039–2047. [Google Scholar]
- Stricker, R.; Eisenbach, M.; Sesselmann, M.; Debes, K.; Gross, H. Improving Visual Road Condition Assessment by Extensive Experiments on the Extended GAPs Dataset. In Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019; pp. 1–8. [Google Scholar]
- Wu, S.; Zhong, S.; Liu, Y. Deep residual learning for image steganalysis. Multimed. Tools Appl. 2018, 77, 10437–10453. [Google Scholar] [CrossRef]
- Li, B.; Wang, K.C.P.; Zhang, A.; Fei, Y.; Sollazzo, G. Automatic Segmentation and Enhancement of Pavement Cracks Based on 3D Pavement Images. J. Adv. Transp. 2019, 2019, 1813763. [Google Scholar] [CrossRef]
- Loupos, K.; Doulamis, A.D.; Stentoumis, C.; Protopapadakis, E.; Makantasis, K.; Doulamis, N.D.; Amditis, A.; Chrobocinski, P.; Victores, J.; Montero, R.; et al. Autonomous robotic system for tunnel structural inspection and assessment. Int. J. Intell. Robot. Appl. 2018, 2, 43–66. [Google Scholar] [CrossRef]
- Protopapadakis, E.; Voulodimos, A.; Doulamis, A.; Doulamis, N.; Stathaki, T. Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing. Appl. Intell. 2019, 49, 2793–2806. [Google Scholar] [CrossRef]
- Augustaukas, R.; Lipnickas, A. Pixel-wise Road Pavement Defects Detection Using U-Net Deep Neural Network. In Proceedings of the 2019 10th IEEE International Conference on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), Metz, France, 18–21 September 2019; IEEE: Metz, France, 2019; pp. 468–472. [Google Scholar]
- Augustaukas, R.; Lipnickas, A. Road Pavement Segmentation Project Code. Available online: https://github.com/rytisss/RoadPavementSegmentation (accessed on 22 April 2020).
- He, K.; Zhang, X.; Ren, S.; Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7–13 December 2015; pp. 1026–1034. [Google Scholar]
- Ioffe, S.; Szegedy, C. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv 2015, arXiv:1502.03167. [Google Scholar]
- Van Laarhoven, T. L2 Regularization versus Batch and Weight Normalization. arXiv 2017, arXiv:1706.05350. [Google Scholar]
- Chu, Z.; Tian, T.; Feng, R.; Wang, L. Sea-Land Segmentation With Res-UNet And Fully Connected CRF. In Proceedings of the IGARSS 2019—2019 IEEE International Geoscience and Remote Sensing Symposium, Yokohama, Japan, 28 July–2 August 2019; pp. 3840–3843. [Google Scholar]
- Xiao, X.; Lian, S.; Luo, Z.; Li, S. Weighted Res-U-Net for High-Quality Retina Vessel Segmentation. In Proceedings of the 2018 9th International Conference on Information Technology in Medicine and Education (ITME), Hangzhou, China, 19–21 October 2018; pp. 327–331. [Google Scholar]
- Riid, A.; Lõuk, R.; Pihlak, R.; Tepljakov, A.; Vassiljeva, K. Pavement distress detection with deep learning using the orthoframes acquired by a mobile mapping system. Appl. Sci. 2019, 9, 4829. [Google Scholar] [CrossRef] [Green Version]
- Xu, W.; Liu, H.; Wang, X.; Qian, Y. Liver segmentation in CT based on ResU-Net with 3D Probabilistic and Geometric Post Process. In Proceedings of the 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), Wuxi, China, 19–21 July 2019; pp. 685–689. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, 26 June–1 July 2016; pp. 770–778. [Google Scholar]
- Yu, F.; Koltun, V. Multi-Scale Context Aggregation by Dilated Convolutions. arXiv 2015, arXiv:1511.07122. [Google Scholar]
- Chen, L.C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; Yuille, A.L. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 834–848. [Google Scholar] [CrossRef]
- Chen, L. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv 2017, arXiv:1706.05587. [Google Scholar]
- Liu, W.; Rabinovich, A.; Berg, A.C. ParseNet: Looking Wider to See Better. arXiv 2015, arXiv:1506.04579. [Google Scholar]
- Wang, Y.; Liang, B.; Ding, M.; Li, J. Dense semantic labeling with atrous spatial pyramid pooling and decoder for high-resolution remote sensing imagery. Remote Sens. 2019, 11, 20. [Google Scholar] [CrossRef] [Green Version]
- Chen, G.; Li, C.; Wei, W.; Jing, W.; Woźniak, M.; Blažauskas, T.; Damaševičius, R. Fully convolutional neural network with augmented atrous spatial pyramid pool and fully connected fusion path for high resolution remote sensing image segmentation. Appl. Sci. 2019, 9, 1816. [Google Scholar] [CrossRef] [Green Version]
- Zhang, P.; Ke, Y.; Zhang, Z.; Wang, M.; Li, P.; Zhang, S. Urban land use and land cover classification using novel deep learning models based on high spatial resolution satellite imagery. Sensors 2018, 18, 3717. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bo Guo, Y.; Matuszewski, B.J. Giana polyp segmentation with fully convolutional dilation neural networks. In Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Prague, Czech Republic, 25–27 February 2019; Volume 4, pp. 632–641. [Google Scholar]
- Chen, L.-C.; Zhu, Y.; Papandreou, G.; Schroff, F.; Adam, H. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation. arXiv 2018, arXiv:1802.02611. [Google Scholar]
- Artacho, B.; Savakis, A. Waterfall atrous spatial pooling architecture for efficient semantic segmentation. Sensors 2019, 19, 5361. [Google Scholar] [CrossRef] [Green Version]
- Jetley, S.; Lord, N.A.; Lee, N.; Torr, P.H.S. Learn To Pay Attention. In Proceedings of the ICLR 2018, Vancouver, Canada, 30 April–3 May 2018. [Google Scholar]
- Schlemper, J.; Oktay, O.; Schaap, M.; Heinrich, M.; Kainz, B.; Glocker, B.; Rueckert, D. Attention gated networks: Learning to leverage salient regions in medical images. Med. Image Anal. 2019, 53, 197–207. [Google Scholar] [CrossRef]
- Oktay, O.; Schlemper, J.; Folgoc, L.L.; Lee, M.; Heinrich, M.; Misawa, K.; Mori, K.; McDonagh, S.; Hammerla, N.Y.; Kainz, B.; et al. Attention U-Net: Learning Where to Look for the Pancreas. arXiv 2018, arXiv:1804.03999. [Google Scholar]
- Cheng, H.D.; Chen, J.-R.; Glazier, C.; Hu, Y.G. Novel Approach to Pavement Cracking Detection Based on Fuzzy Set Theory. J. Comput. Civ. Eng. 1999, 13, 270–280. [Google Scholar] [CrossRef]
- Saar, T.; Talvik, O. Automatic Asphalt pavement crack detection and classification using Neural Networks. In Proceedings of the 2010 12th Biennial Baltic Electronics Conference, Tallinn, Estonia, 4–6 October 2010; pp. 345–348. [Google Scholar]
- Stutz, D.; Hermans, A.; Leibe, B. Superpixels: An evaluation of the state-of-the-art. Comput. Vis. Image Underst. 2018, 166, 1–27. [Google Scholar] [CrossRef] [Green Version]
- Velinsky, S.A.; Kirschke, K.R. Design Considerations for Automated Pavement Crack Sealing Machinery. In Proceedings of the Second International Conference on Applications of Advanced Technologies in Transportation Engineering, Minneapolis, Minnesota, 18-21 August 1991; pp. 77–80. [Google Scholar]
- Liu, W.; Huang, Y.; Li, Y.; Chen, Q. FPCNet: Fast Pavement Crack Detection Network Based on Encoder-Decoder Architecture. arXiv 2019, arXiv:1907.022481. [Google Scholar]
- Wu, S.; Fang, J.; Zheng, X.; Li, X. Sample and Structure-Guided Network for Road Crack Detection. IEEE Access 2019, 7, 130032–130043. [Google Scholar] [CrossRef]
- Lau, S.L.H.; Wang, X.; Xu, Y.; Chong, E.K.P. Automated Pavement Crack Segmentation Using Fully Convolutional U-Net with a Pretrained ResNet-34 Encoder. arXiv 2020, arXiv:2001.01912. [Google Scholar]
- Fan, Z.; Wu, Y.; Lu, J.; Li, W. Automatic Pavement Crack Detection Based on Structured Prediction with the Convolutional Neural Network. arXiv 2018, arXiv:1802.02208. [Google Scholar]
- Escalona, U.; Arce, F.; Zamora, E.; Sossa, H. Fully convolutional networks for automatic pavement crack segmentation. Comput. Sist. 2019, 23, 451–460. [Google Scholar] [CrossRef]
- Keras. Available online: https://keras.io/ (accessed on 22 December 2019).
- Tensorflow. Available online: https://www.tensorflow.org/ (accessed on 22 December 2019).
- Dice, L.R. Measures of the Amount of Ecologic Association Between Species. Ecology 1945, 26, 297–302. [Google Scholar] [CrossRef]
- Iglovikov, V. TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation. arXiv 2018, arXiv:1801.05746. [Google Scholar]
- Popescu, D.; Ichim, L.; Stoican, F. Flooded Area Segmentation from UAV Images Based on Generative Adversarial Networks. In Proceedings of the 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV), Singapore, 18–21 November 2018; pp. 1361–1366. [Google Scholar]
- Liu, F.; Xia, Y.; Yang, D.; Yuille, A.; Xu, D. An Alarm System For Segmentation Algorithm Based On Shape Model. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019. [Google Scholar]
Authors | Tolerance in Pixels | Precision | Recall | Dice |
---|---|---|---|---|
Wu et al. [54] | 0 | 0.4330 | 0.7623 | 0.4809 |
Liu et al. [53] | 2 | 0.9748 | 0.9639 | 0.9693 |
Lau et al. [55] | 2 | 0.9702 | 0.9432 | 0.9555 |
Fan et al. [56] | 2 | 0.9119 | 0.9481 | 0.9244 |
Escalona et al. [57] | 5 | 0.9731 | 0.9428 | 0.9575 |
CrackForest | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0.9898 | 0.7465 | 0.6803 | 0.5489 | 0.7015 |
ResU-Net | 0.9901 | 0.7391 | 0.6928 | 0.5546 | 0.7058 |
ResU-Net+ASPP | 0.9902 | 0.7474 | 0.692 | 0.5603 | 0.7121 |
ResU-Net + ASPP + AG | 0.9899 | 0.7271 | 0.6906 | 0.5442 | 0.6969 |
ResU-Net + ASPP_WF | 0.9900 | 0.7494 | 0.6896 | 0.5595 | 0.7114 |
ResU-Net + ASPP_WF + AG | 0.9896 | 0.7695 | 0.6715 | 0.5575 | 0.7106 |
Crack500 | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0.9845 | 0.7033 | 0.6996 | 0.5282 | 0.6803 |
ResU-Net | 0.9846 | 0.7002 | 0.7083 | 0.5306 | 0.6819 |
ResU-Net + ASPP | 0.9848 | 0.6944 | 0.7152 | 0.5311 | 0.6820 |
ResU-Net + ASPP + AG | 0.9841 | 0.7386 | 0.6808 | 0.5389 | 0.6893 |
ResU-Net + ASPP_WF | 0.9843 | 0.7524 | 0.6789 | 0.5430 | 0.6931 |
ResU-Net + ASPP_WF + AG | 0.9832 | 0.7829 | 0.6447 | 0.5373 | 0.6882 |
GAPs384 | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0.9953 | 0.4798 | 0.7231 | 0.3925 | 0.5448 |
ResU-Net | 0.9954 | 0.4957 | 0.7134 | 0.4038 | 0.557 |
ResU-Net + ASPP | 0.9948 | 0.5754 | 0.6285 | 0.4224 | 0.5786 |
ResU-Net + ASPP + AG | 0.9951 | 0.5526 | 0.6675 | 0.4264 | 0.5822 |
ResU-Net + ASPP_WF | 0.9955 | 0.5459 | 0.7232 | 0.4179 | 0.5696 |
ResU-Net + ASPP_WF + AG | 0.9955 | 0.5251 | 0.7143 | 0.4162 | 0.5693 |
CrackForest | Tolerance, px | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0 | 0.9898 | 0.7465 | 0.6803 | 0.5489 | 0.7015 |
U-Net (Baseline) | 2 | 0.9983 | 0.9797 | 0.9194 | 0.9486 | |
U-Net (Baseline) | 5 | 0.9990 | 0.9994 | 0.9411 | 0.9694 | |
ResU-Net + ASPP | 0 | 0.9900 | 0.7494 | 0.6896 | 0.5595 | 0.7114 |
ResU-Net + ASPP | 2 | 0.9986 | 0.9879 | 0.9280 | - | 0.9570 |
ResU-Net + ASPP | 5 | 0.9991 | 1.0000 | 0.9472 | - | 0.9729 |
Crack500 | Tolerance, px | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0 | 0.9845 | 0.7033 | 0.6996 | 0.5282 | 0.6803 |
U-Net (Baseline) | 2 | 0.9957 | 0.9403 | 0.8759 | - | 0.9070 |
U-Net (Baseline) | 5 | 0.9982 | 0.9949 | 0.9323 | - | 0.9626 |
ResU-Net + ASPP_WF | 0 | 0.9841 | 0.7386 | 0.6808 | 0.5389 | 0.6893 |
ResU-Net + ASPP_WF | 2 | 0.9960 | 0.9309 | 0.9017 | - | 0.9161 |
ResU-Net + ASPP_WF | 5 | 0.9986 | 0.9932 | 0.9481 | - | 0.9702 |
GAPs384 | Tolerance, px | Accuracy | Recall | Precision | IoU | Dice |
U-Net (Baseline) | 0 | 0.9953 | 0.4798 | 0.7231 | 0.3925 | 0.5448 |
U-Net (Baseline) | 2 | 0.9979 | 0.9742 | 0.6799 | - | 0.8009 |
U-Net (Baseline) | 5 | 0.9986 | 1.0000 | 0.7772 | - | 0.8746 |
ResU-Net + ASPP + AG | 0 | 0.9951 | 0.5526 | 0.6675 | 0.4264 | 0.5822 |
ResU-Net + ASPP + AG | 2 | 0.9981 | 0.9438 | 0.7280 | - | 0.8219 |
ResU-Net + ASPP + AG | 5 | 0.9988 | 0.9997 | 0.8127 | - | 0.8966 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Augustauskas, R.; Lipnickas, A. Improved Pixel-Level Pavement-Defect Segmentation Using a Deep Autoencoder. Sensors 2020, 20, 2557. https://doi.org/10.3390/s20092557
Augustauskas R, Lipnickas A. Improved Pixel-Level Pavement-Defect Segmentation Using a Deep Autoencoder. Sensors. 2020; 20(9):2557. https://doi.org/10.3390/s20092557
Chicago/Turabian StyleAugustauskas, Rytis, and Arūnas Lipnickas. 2020. "Improved Pixel-Level Pavement-Defect Segmentation Using a Deep Autoencoder" Sensors 20, no. 9: 2557. https://doi.org/10.3390/s20092557
APA StyleAugustauskas, R., & Lipnickas, A. (2020). Improved Pixel-Level Pavement-Defect Segmentation Using a Deep Autoencoder. Sensors, 20(9), 2557. https://doi.org/10.3390/s20092557