# FPGA-Based Optical Surface Inspection of Wind Turbine Rotor Blades Using Quantized Neural Networks

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Binarized Convolutional Neural Networks

#### 2.2. Dataset

#### 2.3. Neural Network Configuration

#### 2.4. Training

#### 2.5. Field-Programmable Gate Arrays (FPGA)

## 3. Results

## 4. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- AG Energiebilanzen. Erneuerbare Energien. 2018. Available online: https://www.bmwi.de/Redaktion/DE/Dossier/erneuerbare-energien.html (accessed on 30 October 2020).
- Elkmann, N.; Felsch, T.; Förster, T. Robot for rotor blade inspection. In Proceedings of the 2010 1st International Conference on Applied Robotics for the Power Industry, Montreal, QC, Canada, 5–7 October 2010; pp. 1–5. [Google Scholar]
- Zhang, D.; Burnham, K.; Mcdonald, L.; Macleod, C.; Dobie, G.; Summan, R.; Pierce, G. Remote inspection of wind turbine blades using UAV with photogrammetry payload. In Proceedings of the 56th Annual British Conference of Non-Destructive Testing-NDT, Telford, UK, 5–7 September 2017. [Google Scholar]
- Stokkeland, M.; Klausen, K.; Johansen, T.A. Autonomous visual navigation of unmanned aerial vehicle for wind turbine inspection. In Proceedings of the 2015 International Conference on Unmanned Aircraft Systems (ICUAS), Denver, CO, USA, 9–12 June 2015; pp. 998–1007. [Google Scholar]
- Morgenthal, G.; Hallermann, N. Quality assessment of unmanned aerial vehicle (UAV) based visual inspection of structures. Adv. Struct. Eng.
**2014**, 17, 289–302. [Google Scholar] [CrossRef] - Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv
**2014**, arXiv:1409.1556. [Google Scholar] - Staar, B.; Lütjen, M.; Freitag, M. Anomaly detection with convolutional neural networks for industrial surface inspection. Procedia CIRP
**2019**, 79, 484–489. [Google Scholar] [CrossRef] - Agour, M.; Falldorf, C.; Staar, B.; von Freyberg, A.; Fischer, A.; Lütjen, M.; Bergmann, R.B. Fast Quality Inspection of Micro Cold Formed Parts using Telecentric Digital Holographic Microscopy. In MATEC Web of Conferences; EDP Sciences: Les Ulis, France, 2018; Volume 190, p. 15008. [Google Scholar]
- Masci, J.; Meier, U.; Ciresan, D.; Schmidhuber, J.; Fricout, G. Steel defect classification with max-pooling convolutional neural networks. In Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN), Brisbane, Australia, 10–15 June 2012; pp. 1–6. [Google Scholar]
- Gupta, S.; Agrawal, A.; Gopalakrishnan, K.; Narayanan, P. Deep learning with limited numerical precision. In Proceedings of the International Conference on Machine Learning, Lille, France, 6–11 July 2015; pp. 1737–1746. [Google Scholar]
- Rastegari, M.; Ordonez, V.; Redmon, J.; Farhadi, A. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision; Springer: Cham, Switzerland, 2016; pp. 525–542. [Google Scholar]
- Courbariaux, M.; Hubara, I.; Soudry, D.; El-Yaniv, R.; Bengio, Y. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or −1. arXiv
**2016**, arXiv:1602.02830. [Google Scholar] - Angelini, C. Nvidia GeForce GTX 1080 Ti 11GB Review. 2017. Available online: https://www.tomshardware.com/reviews/nvidia-geforce-gtx-1080-ti,4972-6.html (accessed on 30 October 2020).
- Wang, L.; Zhang, Z. Automatic detection of wind turbine blade surface cracks based on UAV-taken images. IEEE Trans. Ind. Electron.
**2017**, 64, 7293–7303. [Google Scholar] [CrossRef] - Giefer, L.A.; Lütjen, M.; Rohde, A.K.; Freitag, M. Determination of the Optimal State of Dough Fermentation in Bread Production by Using Optical Sensors and Deep Learning. Appl. Sci.
**2019**, 9, 4266. [Google Scholar] [CrossRef][Green Version] - Giefer, L.A.; Arango Castellanos, J.D.; Babr, M.M.; Freitag, M. Deep Learning-Based Pose Estimation of Apples for Inspection in Logistic Centers Using Single-Perspective Imaging. Processes
**2019**, 7, 424. [Google Scholar] [CrossRef][Green Version] - Giefer, L.A.; Arango, J.D.; Faghihabdolahi, M.; Freitag, M. Orientation detection of fruits by means of convolutional neural networks and laser line projection for the automation of fruit packing systems. Procedia CIRP
**2020**, 88, 533–538. [Google Scholar] [CrossRef] - Bullock, J.; Cuesta-Lázaro, C.; Quera-Bofarull, A. XNet: A convolutional neural network (CNN) implementation for medical X-ray image segmentation suitable for small datasets. In Medical Imaging 2019: Biomedical Applications in Molecular, Structural, and Functional Imaging; International Society for Optics and Photonics: Bellingham, WA, USA, 2019; Volume 10953, p. 109531Z. [Google Scholar]
- Cai, Z.; Vasconcelos, N. Cascade R-CNN: High Quality Object Detection and Instance Segmentation. arXiv
**2019**, arXiv:1906.09756. [Google Scholar] [CrossRef][Green Version] - Zhao, B.; Feng, J.; Wu, X.; Yan, S. A survey on deep learning-based fine-grained object classification and semantic segmentation. Int. J. Autom. Comput.
**2017**, 14, 119–135. [Google Scholar] [CrossRef] - Denhof, D.; Staar, B.; Lütjen, M.; Freitag, M. Automatic Optical Surface Inspection of Wind Turbine Rotor Blades using Convolutional Neural Networks. Procedia CIRP
**2019**, 81, 1166–1170. [Google Scholar] [CrossRef] - Shihavuddin, A.S.M.; Chen, X.; Fedorov, V.; Nymark Christensen, A.; Andre Brogaard Riis, N.; Branner, K.; Reinhold Paulsen, R. Wind Turbine Surface Damage Detection by Deep Learning Aided Drone Inspection Analysis. Energies
**2019**, 12, 676. [Google Scholar] [CrossRef][Green Version] - Umuroglu, Y.; Fraser, N.J.; Gambardella, G.; Blott, M.; Leong, P.; Jahre, M.; Vissers, K. Finn: A framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, CA, USA, 22–24 February 2017; pp. 65–74. [Google Scholar]
- Zhou, S.; Wu, Y.; Ni, Z.; Zhou, X.; Wen, H.; Zou, Y. Dorefa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients. arXiv
**2016**, arXiv:1606.06160. [Google Scholar] - Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; Ng, A. Reading Digits in Natural Images with Unsupervised Feature Learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, Granada, Spain, 12–17 December 2011. [Google Scholar]
- Sanaullah, A.; Yang, C.; Alexeev, Y.; Yoshii, K.; Herbordt, M.C. Real-time data analysis for medical diagnosis using FPGA-accelerated neural networks. BMC Bioinform.
**2018**, 19, 490. [Google Scholar] [CrossRef] [PubMed][Green Version] - Li, H.; Acceleration of Deep Learning on FPGA. Electronic Theses and Dissertations. 2017. Available online: https://scholar.uwindsor.ca/etd/5947 (accessed on 30 October 2020).
- Jain, M. Paper Explanation: Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1. 2018. Available online: https://mohitjain.me/2018/07/14/bnn/ (accessed on 30 October 2020).
- Kim, M.; Smaragdis, P. Bitwise Neural Networks. arXiv
**2016**, arXiv:1601.06071. [Google Scholar] - Bengio, Y. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade; Springer: Berlin/Heidelberg, Germany, 2012; pp. 437–478. [Google Scholar]
- Kingma, D.P.; Ba, J. Adam: A Method for Stochastic Optimization. arXiv
**2014**, arXiv:1412.6980. [Google Scholar] - Smith, L.N. Cyclical learning rates for training neural networks. In Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA, 24–31 March 2017; pp. 464–472. [Google Scholar]
- Refaeilzadeh, P.; Tang, L.; Liu, H. Cross-Validation. Encycl. Database Syst.
**2009**, 5, 532–538. [Google Scholar] - Nane, R.; Sima, V.M.; Pilato, C.; Choi, J.; Fort, B.; Canis, A.; Anderson, J. A survey and evaluation of FPGA high-level synthesis tools. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst.
**2015**, 35, 1591–1604. [Google Scholar] [CrossRef][Green Version] - Abdelouahab, K.; Pelcat, M.; Serot, J.; Berry, F. Accelerating CNN Inference on Fpgas: A Survey. arXiv
**2018**, arXiv:1806.01683. [Google Scholar] - Kaara, K. Fpgasystems. SpooNN. 2018. Available online: https://github.com/fpgasystems/spooNN (accessed on 30 October 2020).
- Kilts, S. Advanced FPGA Design: Architecture, Implementation, and Optimization; John Wiley & Sons: Hoboken, NJ, USA, 2007. [Google Scholar]
- Youden, W.J. Index for rating diagnostic tests. Cancer
**1950**, 3, 32–35. [Google Scholar] [CrossRef] - Zhang, C.; Ma, Y. (Eds.) Ensemble Machine Learning: Methods and Applications; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012. [Google Scholar]

**Figure 1.**Difference between real-valued (

**top**) and binarized neural networks (

**bottom**): Own representation based on [28].

**Figure 3.**Example of rotor blades without defects: (

**a**) bird droppings; (

**b**) sticker leftovers; (

**c**) dirt; (

**d**) material change; (

**e**) color change; and (

**f**) shadow.

**Figure 7.**Classification accuracies of $BCN{N}_{quant0}$ (blue), $BCN{N}_{quant13}$ (green) and $BCN{N}_{quant11}$ (red) based on 10-fold cross-validation.

**Figure 8.**Boxplots of true positive (TPR) and true negative rates (TNR) based on 10-fold cross-validation.

Parameter Name | Parameter Value |
---|---|

Horizontal flip | True (probability 0.5) |

Vertical flip | True (probability 0.5) |

Transpose | True (probability 0.5) |

Shift | Horizontal and vertical fraction 0.2 |

Resource Type | Used | Available | Utilization (%) |
---|---|---|---|

Slices | 11,172 | 13,300 | 84.00 |

Look-up tables | 27,132 | 53,200 | 51.00 |

Block RAM | 117.5 | 140 | 83.93 |

DSPs | 84 | 220 | 38.18 |

MUXes | 73 | 26,600 | 0.27 |

Network Model | Inference Time (ms) | Accuracy (%) |
---|---|---|

$BCN{N}_{quant13}$ (FPGA) | 4.6 | 96.4 |

$BCN{N}_{quant13}$ (Jetson) | 6.9 | 96.4 |

VGG16 (GPU) | 26 | 97.0 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Giefer, L.A.; Staar, B.; Freitag, M.
FPGA-Based Optical Surface Inspection of Wind Turbine Rotor Blades Using Quantized Neural Networks. *Electronics* **2020**, *9*, 1824.
https://doi.org/10.3390/electronics9111824

**AMA Style**

Giefer LA, Staar B, Freitag M.
FPGA-Based Optical Surface Inspection of Wind Turbine Rotor Blades Using Quantized Neural Networks. *Electronics*. 2020; 9(11):1824.
https://doi.org/10.3390/electronics9111824

**Chicago/Turabian Style**

Giefer, Lino Antoni, Benjamin Staar, and Michael Freitag.
2020. "FPGA-Based Optical Surface Inspection of Wind Turbine Rotor Blades Using Quantized Neural Networks" *Electronics* 9, no. 11: 1824.
https://doi.org/10.3390/electronics9111824