Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity
Abstract
:1. Introduction
2. General Structures of Data Collection and Processing for Fault Diagnosis
3. Internet of Things Protocols for Batch and Stream Processing of Fault Diagnosis
4. General Structure of Fault Diagnosis and Perspective Maintenance
5. Feature Extraction Methods
6. Demonstration of Fault Diagnosis with MQTT Communication
7. Recognition of a Time–Frequency RGB Image of Vibration
8. Discussion
9. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Kasper, W.C. Power from Trash. Environ. Sci. Policy Sustain. Dev. 1974, 16, 34–38. [Google Scholar] [CrossRef]
- Sullivan, J. Trash or Treasure: Global Trade and the Accumulation of E-Waste in Lagos, Nigeria. Afr. Today 2014, 61, 89–112. [Google Scholar] [CrossRef]
- Kan, Y.; Liu, H.; Yang, Y.; Wei, Y.; Yu, Y.; Qiu, R.; Ouyang, Y. Two Birds with One Stone: The Route from Waste Printed Circuit Board Electronic Trash to Multifunctional Biomimetic Slippery Liquid-Infused Coating. J. Ind. Eng. Chem. 2022, 114, 233–241. [Google Scholar] [CrossRef]
- Ji, X.; Yang, M.; Wan, A.; Yu, S.; Yao, Z. Bioleaching of Typical Electronic Waste—Printed Circuit Boards (WPCBs): A Short Review. Int. J. Environ. Res. Public Health 2022, 19, 7508. [Google Scholar] [CrossRef] [PubMed]
- Rene, E.R.; Sethurajan, M.; Kumar Ponnusamy, V.; Kumar, G.; Bao Dung, T.N.; Brindhadevi, K.; Pugazhendhi, A. Electronic Waste Generation, Recycling and Resource Recovery: Technological Perspectives and Trends. J. Hazard. Mater. 2021, 416, 125664. [Google Scholar] [CrossRef] [PubMed]
- Pan, X.; Wong, C.W.Y.; Li, C. Circular Economy Practices in the Waste Electrical and Electronic Equipment (WEEE) Industry: A Systematic Review and Future Research Agendas. J. Clean. Prod. 2022, 365, 132671. [Google Scholar] [CrossRef]
- Marinello, S.; Gamberini, R. Multi-Criteria Decision Making Approaches Applied to Waste Electrical and Electronic Equipment (WEEE): A Comprehensive Literature Review. Toxics 2021, 9, 13. [Google Scholar] [CrossRef]
- Breque, M.; De Nul, L.; Petridis, A. Directorate-General for Research and Innovation (European Commission). In Industry 5.0: Towards a Sustainable, Human Centric and Resilient European Industry; Publications Office of the European Union: Luxembourg, 2021; ISBN 978-92-76-25308-2. [Google Scholar]
- Smit, J.; Kreutzer, S.; Moeller, C.; Carlberg, C. Industry 4.0, European Parliament, Policy Department A: Economic and Scientific Policy; European Parliament: Strasbourg, France, 2016. [Google Scholar] [CrossRef]
- Teixeira, J.E.; Tavares-Lehmann, A.T.C.P. Industry 4.0 in the European Union: Policies and National Strategies. Technol. Forecast. Soc. Chang. 2022, 180, 121664. [Google Scholar] [CrossRef]
- Piechocki, M.; Kraft, M.; Pajchrowski, T.; Aszkowski, P.; Pieczynski, D. Efficient People Counting in Thermal Images: The Benchmark of Resource-Constrained Hardware. IEEE Access 2022, 10, 124835–124847. [Google Scholar] [CrossRef]
- Kraft, M.; Aszkowski, P.; Pieczyński, D.; Fularz, M. Low-Cost Thermal Camera-Based Counting Occupancy Meter Facilitating Energy Saving in Smart Buildings. Energies 2021, 14, 4542. [Google Scholar] [CrossRef]
- Ptak, B.; Pieczyński, D.; Piechocki, M.; Kraft, M. On-Board Crowd Counting and Density Estimation Using Low Altitude Unmanned Aerial Vehicles—Looking beyond Beating the Benchmark. Remote Sens. 2022, 14, 2288. [Google Scholar] [CrossRef]
- Yu, Y.; Samali, B.; Rashidi, M.; Mohammadi, M.; Nguyen, T.N.; Zhang, G. Vision-Based Concrete Crack Detection Using a Hybrid Framework Considering Noise Effect. J. Build. Eng. 2022, 61, 105246. [Google Scholar] [CrossRef]
- Chen, H.-Y.; Lee, C.-H. Vibration Signals Analysis by Explainable Artificial Intelligence (XAI) Approach: Application on Bearing Faults Diagnosis. IEEE Access 2020, 8, 134246–134256. [Google Scholar] [CrossRef]
- Kim, M.S.; Yun, J.P.; Park, P. Deep Learning-Based Explainable Fault Diagnosis Model With an Individually Grouped 1-D Convolution for Three-Axis Vibration Signals. IEEE Trans. Ind. Inform. 2022, 18, 8807–8817. [Google Scholar] [CrossRef]
- Zhang, X.; Zhao, Z.; Wang, Z.; Wang, X. Fault Detection and Identification Method for Quadcopter Based on Airframe Vibration Signals. Sensors 2021, 21, 581. [Google Scholar] [CrossRef]
- Wang, Y.; Yang, M.; Li, Y.; Xu, Z.; Wang, J.; Fang, X. A Multi-Input and Multi-Task Convolutional Neural Network for Fault Diagnosis Based on Bearing Vibration Signal. IEEE Sens. J. 2021, 21, 10946–10956. [Google Scholar] [CrossRef]
- IEC 61784-1:2019 Industrial Communication Networks—Profiles Part 1: Fieldbus Profiles. Available online: https://webstore.iec.ch/publication/59887#additionalinfo (accessed on 28 October 2022).
- RFC791; Internet Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1981.
- RFC2460; Internet Protocol, Version 6 (IPv6) Specification. Internet Engineering Task Force: Fremont, CA, USA, 1998.
- RFC768; User Datagram Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1980.
- RFC793; Transmission Control Protocol. Internet Engineering Task Force: Fremont, CA, USA, 1981.
- IEEE Std 802.3-2022 (Revision of IEEE Std 802.3-2018); IEEE Standard for Ethernet. IEEE: Piscataway, NJ, USA, 2022. [CrossRef]
- RFC8446; The Transport Layer Security (TLS) Protocol Version 1.3. Internet Engineering Task Force: Fremont, CA, USA, 2018.
- RFC2616; Hypertext Transfer Protocol—HTTP/1.1. Internet Engineering Task Force: Fremont, CA, USA, 1999.
- RFC2068; Hypertext Transfer Protocol—HTTP/1.1. Internet Engineering Task Force: Fremont, CA, USA, 1997.
- RFC6455; The WebSocket Protocol. Internet Engineering Task Force: Fremont, CA, USA, 2011.
- RFC7540; Hypertext Transfer Protocol Version 2 (HTTP/2). Internet Engineering Task Force: Fremont, CA, USA, 2015.
- RFC9114; HTTP/3. Internet Engineering Task Force: Fremont, CA, USA, 2022.
- RFC9000; QUIC: A UDP-Based Multiplexed and Secure Transport. Internet Engineering Task Force: Fremont, CA, USA, 2021.
- RFC826; An Ethernet Address Resolution Protocol: Or Converting Network Protocol Addresses to 48.Bit Ethernet Address for Transmission on Ethernet Hardware. Internet Engineering Task Force: Fremont, CA, USA, 1982.
- MQTT Version 3.1.1. Available online: http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/errata01/os/mqtt-v3.1.1-errata01-os-complete.html (accessed on 31 October 2022).
- MQTT Version 5.0. Available online: http://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html (accessed on 31 October 2022).
- RFC7252; The Constrained Application Protocol (CoAP). Internet Engineering Task Force: Fremont, CA, USA, 2014.
- RFC8323; CoAP (Constrained Application Protocol) over TCP, TLS, and WebSockets. Internet Engineering Task Force: Fremont, CA, USA, 2018.
- JSON. Available online: https://www.json.org/json-en.html (accessed on 23 November 2022).
- ECMA-404; The JSON Data Interchange Syntax; Ecma International, 2nd edition, Geneva, Switzerland, December 2017. Available online: https://www.ecma-international.org/wp-content/uploads/ECMA-404_2nd_edition_december_2017.pdf (accessed on 30 February 2023).
- Extensible Markup Language (XML). Available online: https://www.w3.org/XML/ (accessed on 23 November 2022).
- RFC6838; Media Type Specifications and Registration Procedures. Internet Engineering Task Force: Fremont, CA, USA, 2013.
- Media Types. Available online: https://www.iana.org/assignments/media-types/media-types.xhtml (accessed on 23 November 2022).
- RFC7303; XML Media Types. Internet Engineering Task Force: Fremont, CA, USA, 2014.
- RFC8259; The JavaScript Object Notation (JSON) Data Interchange Format. Internet Engineering Task Force: Fremont, CA, USA, 2017.
- Mendes, M.J.G.C.; Neto, Í.M.M.S.; Calado, J.M.F. Fault Diagnosis System via Internet Applied to a Gantry Robot—A Proposal for Industry 4.0. In Proceedings of the 2018 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC), Torres Vedras, Portugal, 25–27 April 2018; pp. 160–166. [Google Scholar]
- Jagatheesaperumal, S.K.; Rahouti, M.; Ahmad, K.; Al-Fuqaha, A.; Guizani, M. The Duo of Artificial Intelligence and Big Data for Industry 4.0: Applications, Techniques, Challenges, and Future Research Directions. IEEE Internet Things J. 2022, 9, 12861–12885. [Google Scholar] [CrossRef]
- Natesha, B.V.; Guddeti, R.M.R. Fog-Based Intelligent Machine Malfunction Monitoring System for Industry 4.0. IEEE Trans. Ind. Inform. 2021, 17, 7923–7932. [Google Scholar] [CrossRef]
- Rastogi, V.; Srivastava, S.; Mishra, M.; Thukral, R. Predictive Maintenance for SME in Industry 4.0. In Proceedings of the 2020 Global Smart Industry Conference (GloSIC), Chelyabinsk, Russia, 17–19 November 2020; pp. 382–390. [Google Scholar]
- Anil Kumar, T.C.; Bhusal Sharma, N.; Mishra, A.K.; Patil, P.; Sarveswara Reddy, S.; Bhardwaj, R. Fault Identification Model Using IIoT for Industrial Application. Meas. Sens. 2022, 24, 100526. [Google Scholar] [CrossRef]
- Radonjić, M.; Vujnović, S.; Krstić, A.; Zečević, Ž. IoT System for Detecting the Condition of Rotating Machines Based on Acoustic Signals. Appl. Sci. 2022, 12, 4385. [Google Scholar] [CrossRef]
- Behera, S.; Misra, R. Generative Adversarial Networks Based Remaining Useful Life Estimation for IIoT. Comput. Electr. Eng. 2021, 92, 107195. [Google Scholar] [CrossRef]
- Casado-Vara, R.; Vale, Z.; Prieto, J.; Corchado, J.M. Fault-Tolerant Temperature Control Algorithm for IoT Networks in Smart Buildings. Energies 2018, 11, 3430. [Google Scholar] [CrossRef] [Green Version]
- Akay, A.; Lefley, P. Open-Circuit Fault-Tolerant Control of Multi-Phase PM Machines by Compensating the d-q Axes Currents. Energies 2021, 14, 192. [Google Scholar] [CrossRef]
- Patel, H.R.; Shah, V.A. Stable Fault Tolerant Controller Design for Takagi–Sugeno Fuzzy Model-Based Control Systems via Linear Matrix Inequalities: Three Conical Tank Case Study. Energies 2019, 12, 2221. [Google Scholar] [CrossRef] [Green Version]
- Nguyen, N.P.; Hong, S.K. Fault Diagnosis and Fault-Tolerant Control Scheme for Quadcopter UAVs with a Total Loss of Actuator. Energies 2019, 12, 1139. [Google Scholar] [CrossRef] [Green Version]
- Siembab, K.; Zawirski, K. Modified Space Vector Modulation for Fault Tolerant Control of PMSM Drive. In Proceedings of the 2016 IEEE International Power Electronics and Motion Control Conference (PEMC), Varna, Bulgaria, 25–28 September 2016; pp. 1064–1071. [Google Scholar]
- Łuczak, D.; Siembab, K. Comparison of Fault Tolerant Control Algorithm Using Space Vector Modulation of PMSM Drive. In Proceedings of the 16th International Conference on Mechatronics—Mechatronika 2014, Brno, Czech Republic, 3–5 December 2014; pp. 24–31. [Google Scholar]
- Moghadam, F.K.; Nejad, A.R. Online Condition Monitoring of Floating Wind Turbines Drivetrain by Means of Digital Twin. Mech. Syst. Signal Process. 2022, 162, 108087. [Google Scholar] [CrossRef]
- Leng, J.; Wang, D.; Shen, W.; Li, X.; Liu, Q.; Chen, X. Digital Twins-Based Smart Manufacturing System Design in Industry 4.0: A Review. J. Manuf. Syst. 2021, 60, 119–137. [Google Scholar] [CrossRef]
- Peng, F.; Zheng, L.; Peng, Y.; Fang, C.; Meng, X. Digital Twin for Rolling Bearings: A Review of Current Simulation and PHM Techniques. Measurement 2022, 201, 111728. [Google Scholar] [CrossRef]
- Wang, T.; Cheng, J.; Yang, Y.; Esposito, C.; Snoussi, H.; Tao, F. Adaptive Optimization Method in Digital Twin Conveyor Systems via Range-Inspection Control. IEEE Trans. Autom. Sci. Eng. 2022, 19, 1296–1304. [Google Scholar] [CrossRef]
- Nguyen, T.; Duong, Q.H.; Nguyen, T.V.; Zhu, Y.; Zhou, L. Knowledge Mapping of Digital Twin and Physical Internet in Supply Chain Management: A Systematic Literature Review. Int. J. Prod. Econ. 2022, 244, 108381. [Google Scholar] [CrossRef]
- Teng, S.Y.; Touš, M.; Leong, W.D.; How, B.S.; Lam, H.L.; Máša, V. Recent Advances on Industrial Data-Driven Energy Savings: Digital Twins and Infrastructures. Renew. Sustain. Energy Rev. 2021, 135, 110208. [Google Scholar] [CrossRef]
- Unal, P.; Albayrak, Ö.; Jomâa, M.; Berre, A.J. Data-Driven Artificial Intelligence and Predictive Analytics for the Maintenance of Industrial Machinery with Hybrid and Cognitive Digital Twins. In Technologies and Applications for Big Data Value; Curry, E., Auer, S., Berre, A.J., Metzger, A., Perez, M.S., Zillner, S., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 299–319. ISBN 978-3-030-78307-5. [Google Scholar]
- Cohen, Y.; Singer, G. A Smart Process Controller Framework for Industry 4.0 Settings. J. Intell. Manuf. 2021, 32, 1975–1995. [Google Scholar] [CrossRef]
- Zhu, G.; Li, Z.; Wu, N. Model-Based Fault Identification of Discrete Event Systems Using Partially Observed Petri Nets. Automatica 2018, 96, 201–212. [Google Scholar] [CrossRef]
- Xu, B.; Yin, X.; Yin, X.; Wang, Y.; Pang, S. Fault Diagnosis of Power Systems Based on Temporal Constrained Fuzzy Petri Nets. IEEE Access 2019, 7, 101895–101904. [Google Scholar] [CrossRef]
- Wang, H.; Wu, Y.; Min, G.; Miao, W. A Graph Neural Network-Based Digital Twin for Network Slicing Management. IEEE Trans. Ind. Inform. 2022, 18, 1367–1376. [Google Scholar] [CrossRef]
- Łuczak, D. Nonlinear Identification with Constraints in Frequency Domain of Electric Direct Drive with Multi-Resonant Mechanical Part. Energies 2021, 14, 7190. [Google Scholar] [CrossRef]
- Luczak, D. Mathematical Model of Multi-Mass Electric Drive System with Flexible Connection. In Proceedings of the 2014 19th International Conference on Methods and Models in Automation and Robotics (MMAR), Miedzyzdroje, Poland, 2–5 September 2014; pp. 590–595. [Google Scholar]
- Luczak, D.; Nowopolski, K. Identification of Multi-Mass Mechanical Systems in Electrical Drives. In Proceedings of the 2014 16th International Conference on Mechatronics—Mechatronika (ME), Brno, Czech Republic, 3–5 December 2014; pp. 275–282. [Google Scholar]
- Huang, W.; Du, J.; Hua, W.; Lu, W.; Bi, K.; Zhu, Y.; Fan, Q. Current-Based Open-Circuit Fault Diagnosis for PMSM Drives With Model Predictive Control. IEEE Trans. Power Electron. 2021, 36, 10695–10704. [Google Scholar] [CrossRef]
- Wu, C.; Guo, C.; Xie, Z.; Ni, F.; Liu, H. A Signal-Based Fault Detection and Tolerance Control Method of Current Sensor for PMSM Drive. IEEE Trans. Ind. Electron. 2018, 65, 9646–9657. [Google Scholar] [CrossRef]
- Jiang, L.; Deng, Z.; Tang, X.; Hu, L.; Lin, X.; Hu, X. Data-Driven Fault Diagnosis and Thermal Runaway Warning for Battery Packs Using Real-World Vehicle Data. Energy 2021, 234, 121266. [Google Scholar] [CrossRef]
- Chang, C.; Zhou, X.; Jiang, J.; Gao, Y.; Jiang, Y.; Wu, T. Electric Vehicle Battery Pack Micro-Short Circuit Fault Diagnosis Based on Charging Voltage Ranking Evolution. J. Power Sources 2022, 542, 231733. [Google Scholar] [CrossRef]
- Gao, S.; Xu, L.; Zhang, Y.; Pei, Z. Rolling Bearing Fault Diagnosis Based on SSA Optimized Self-Adaptive DBN. ISA Trans. 2022, 128, 485–502. [Google Scholar] [CrossRef]
- Wang, C.-S.; Kao, I.-H.; Perng, J.-W. Fault Diagnosis and Fault Frequency Determination of Permanent Magnet Synchronous Motor Based on Deep Learning. Sensors 2021, 21, 3608. [Google Scholar] [CrossRef]
- Feng, Z.; Gao, A.; Li, K.; Ma, H. Planetary Gearbox Fault Diagnosis via Rotary Encoder Signal Analysis. Mech. Syst. Signal Process. 2021, 149, 107325. [Google Scholar] [CrossRef]
- Ma, J.; Li, C.; Zhang, G. Rolling Bearing Fault Diagnosis Based on Deep Learning and Autoencoder Information Fusion. Symmetry 2022, 14, 13. [Google Scholar] [CrossRef]
- Abbas, S.H.; Jang, J.-K.; Kim, D.-H.; Lee, J.-R. Underwater Vibration Analysis Method for Rotating Propeller Blades Using Laser Doppler Vibrometer. Opt. Lasers Eng. 2020, 132, 106133. [Google Scholar] [CrossRef]
- Dutta, S.; Basu, B.; Talukdar, F.A. Classification of Motor Faults Based on Transmission Coefficient and Reflection Coefficient of Omni-Directional Antenna Using DCNN. Expert Syst. Appl. 2022, 198, 116832. [Google Scholar] [CrossRef]
- Zhang, X.; Niu, H.; Hou, C.; Di, F. An Edge-Filter FBG Interrogation Approach Based on Tunable Fabry-Perot Filter for Strain Measurement of Planetary Gearbox. Opt. Fiber Technol. 2020, 60, 102379. [Google Scholar] [CrossRef]
- Zhang, P.; Lu, D. A Survey of Condition Monitoring and Fault Diagnosis toward Integrated O&M for Wind Turbines. Energies 2019, 12, 2801. [Google Scholar] [CrossRef] [Green Version]
- Wu, J.; Yang, Y.; Wang, P.; Wang, J.; Cheng, J. A Novel Method for Gear Crack Fault Diagnosis Using Improved Analytical-FE and Strain Measurement. Measurement 2020, 163, 107936. [Google Scholar] [CrossRef]
- Fedorko, G.; Molnár, V.; Vasiľ, M.; Salai, R. Proposal of Digital Twin for Testing and Measuring of Transport Belts for Pipe Conveyors within the Concept Industry 4.0. Measurement 2021, 174, 108978. [Google Scholar] [CrossRef]
- Pu, H.; He, L.; Zhao, C.; Yau, D.K.Y.; Cheng, P.; Chen, J. Fingerprinting Movements of Industrial Robots for Replay Attack Detection. IEEE Trans. Mob. Comput. 2022, 21, 3629–3643. [Google Scholar] [CrossRef]
- Rafati, A.; Shaker, H.R.; Ghahghahzadeh, S. Fault Detection and Efficiency Assessment for HVAC Systems Using Non-Intrusive Load Monitoring: A Review. Energies 2022, 15, 341. [Google Scholar] [CrossRef]
- Sabry, A.H.; Nordin, F.H.; Sabry, A.H.; Abidin Ab Kadir, M.Z. Fault Detection and Diagnosis of Industrial Robot Based on Power Consumption Modeling. IEEE Trans. Ind. Electron. 2020, 67, 7929–7940. [Google Scholar] [CrossRef]
- Sánchez-Sutil, F.; Cano-Ortega, A.; Hernández, J.C. Design and Implementation of a Smart Energy Meter Using a LoRa Network in Real Time. Electronics 2021, 10, 3152. [Google Scholar] [CrossRef]
- Wang, Z.; Tian, B.; Qiao, W.; Qu, L. Real-Time Aging Monitoring for IGBT Modules Using Case Temperature. IEEE Trans. Ind. Electron. 2016, 63, 1168–1178. [Google Scholar] [CrossRef]
- Dhiman, H.S.; Deb, D.; Muyeen, S.M.; Kamwa, I. Wind Turbine Gearbox Anomaly Detection Based on Adaptive Threshold and Twin Support Vector Machines. IEEE Trans. Energy Convers. 2021, 36, 3462–3469. [Google Scholar] [CrossRef]
- Glowacz, A. Fault Diagnosis of Electric Impact Drills Using Thermal Imaging. Measurement 2021, 171, 108815. [Google Scholar] [CrossRef]
- Al-Musawi, A.K.; Anayi, F.; Packianather, M. Three-Phase Induction Motor Fault Detection Based on Thermal Image Segmentation. Infrared Phys. Technol. 2020, 104, 103140. [Google Scholar] [CrossRef]
- Li, Z.; Zhang, Y.; Abu-Siada, A.; Chen, X.; Li, Z.; Xu, Y.; Zhang, L.; Tong, Y. Fault Diagnosis of Transformer Windings Based on Decision Tree and Fully Connected Neural Network. Energies 2021, 14, 1531. [Google Scholar] [CrossRef]
- Rauber, T.W.; da Silva Loca, A.L.; de Assis Boldt, F.; Rodrigues, A.L.; Varejão, F.M. An Experimental Methodology to Evaluate Machine Learning Methods for Fault Diagnosis Based on Vibration Signals. Expert Syst. Appl. 2021, 167, 114022. [Google Scholar] [CrossRef]
- Meyer, A. Vibration Fault Diagnosis in Wind Turbines Based on Automated Feature Learning. Energies 2022, 15, 1514. [Google Scholar] [CrossRef]
- Lee, J.-H.; Pack, J.-H.; Lee, I.-S. Fault Diagnosis of Induction Motor Using Convolutional Neural Network. Appl. Sci. 2019, 9, 2950. [Google Scholar] [CrossRef] [Green Version]
- Cao, Y.; Sun, Y.; Xie, G.; Li, P. A Sound-Based Fault Diagnosis Method for Railway Point Machines Based on Two-Stage Feature Selection Strategy and Ensemble Classifier. IEEE Trans. Intell. Transp. Syst. 2022, 23, 12074–12083. [Google Scholar] [CrossRef]
- Shiri, H.; Wodecki, J.; Ziętek, B.; Zimroz, R. Inspection Robotic UGV Platform and the Procedure for an Acoustic Signal-Based Fault Detection in Belt Conveyor Idler. Energies 2021, 14, 7646. [Google Scholar] [CrossRef]
- Karabacak, Y.E.; Gürsel Özmen, N.; Gümüşel, L. Intelligent Worm Gearbox Fault Diagnosis under Various Working Conditions Using Vibration, Sound and Thermal Features. Appl. Acoust. 2022, 186, 108463. [Google Scholar] [CrossRef]
- Yao, Y.; Wang, H.; Li, S.; Liu, Z.; Gui, G.; Dan, Y.; Hu, J. End-To-End Convolutional Neural Network Model for Gear Fault Diagnosis Based on Sound Signals. Appl. Sci. 2018, 8, 1584. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Z.; Li, J.; Song, Y.; Sun, Y.; Zhang, X.; Hu, Y.; Guo, R.; Han, X. A Novel Ultrasound-Vibration Composite Sensor for Defects Detection of Electrical Equipment. IEEE Trans. Power Deliv. 2022, 37, 4477–4480. [Google Scholar] [CrossRef]
- Wang, W.; Xue, Y.; He, C.; Zhao, Y. Review of the Typical Damage and Damage-Detection Methods of Large Wind Turbine Blades. Energies 2022, 15, 5672. [Google Scholar] [CrossRef]
- Wang, X.; Mao, D.; Li, X. Bearing Fault Diagnosis Based on Vibro-Acoustic Data Fusion and 1D-CNN Network. Measurement 2021, 173, 108518. [Google Scholar] [CrossRef]
- Maruyama, T.; Maeda, M.; Nakano, K. Lubrication Condition Monitoring of Practical Ball Bearings by Electrical Impedance Method. Tribol. Online 2019, 14, 327–338. [Google Scholar] [CrossRef] [Green Version]
- Wakiru, J.M.; Pintelon, L.; Muchiri, P.N.; Chemweno, P.K. A Review on Lubricant Condition Monitoring Information Analysis for Maintenance Decision Support. Mech. Syst. Signal Process. 2019, 118, 108–132. [Google Scholar] [CrossRef]
- Rizk, P.; Younes, R.; Ilinca, A.; Khoder, J. Wind Turbine Ice Detection Using Hyperspectral Imaging. Remote Sens. Appl. Soc. Environ. 2022, 26, 100711. [Google Scholar] [CrossRef]
- Rizk, P.; Younes, R.; Ilinca, A.; Khoder, J. Wind Turbine Blade Defect Detection Using Hyperspectral Imaging. Remote Sens. Appl. Soc. Environ. 2021, 22, 100522. [Google Scholar] [CrossRef]
- Meribout, M. Gas Leak-Detection and Measurement Systems: Prospects and Future Trends. IEEE Trans. Instrum. Meas. 2021, 70. [Google Scholar] [CrossRef]
- Li, Y.; Yu, Q.; Xie, M.; Zhang, Z.; Ma, Z.; Cao, K. Identifying Oil Spill Types Based on Remotely Sensed Reflectance Spectra and Multiple Machine Learning Algorithms. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 9071–9078. [Google Scholar] [CrossRef]
- Zhou, Q.; Chen, R.; Huang, B.; Liu, C.; Yu, J.; Yu, X. An Automatic Surface Defect Inspection System for Automobiles Using Machine Vision Methods. Sensors 2019, 19, 644. [Google Scholar] [CrossRef] [Green Version]
- Yang, L.; Fan, J.; Liu, Y.; Li, E.; Peng, J.; Liang, Z. A Review on State-of-the-Art Power Line Inspection Techniques. IEEE Trans. Instrum. Meas. 2020, 69, 9350–9365. [Google Scholar] [CrossRef]
- Davari, N.; Akbarizadeh, G.; Mashhour, E. Intelligent Diagnosis of Incipient Fault in Power Distribution Lines Based on Corona Detection in UV-Visible Videos. IEEE Trans. Power Deliv. 2021, 36, 3640–3648. [Google Scholar] [CrossRef]
- Kim, S.; Kim, D.; Jeong, S.; Ham, J.-W.; Lee, J.-K.; Oh, K.-Y. Fault Diagnosis of Power Transmission Lines Using a UAV-Mounted Smart Inspection System. IEEE Access 2020, 8, 149999–150009. [Google Scholar] [CrossRef]
- Ullah, Z.; Lodhi, B.A.; Hur, J. Detection and Identification of Demagnetization and Bearing Faults in PMSM Using Transfer Learning-Based VGG. Energies 2020, 13, 3834. [Google Scholar] [CrossRef]
- Long, H.; Xu, S.; Gu, W. An Abnormal Wind Turbine Data Cleaning Algorithm Based on Color Space Conversion and Image Feature Detection. Appl. Energy 2022, 311, 118594. [Google Scholar] [CrossRef]
- Kreutz, M.; Alla, A.A.; Eisenstadt, A.; Freitag, M.; Thoben, K.-D. Ice Detection on Rotor Blades of Wind Turbines Using RGB Images and Convolutional Neural Networks. Procedia CIRP 2020, 93, 1292–1297. [Google Scholar] [CrossRef]
- Xie, T.; Huang, X.; Choi, S.-K. Intelligent Mechanical Fault Diagnosis Using Multisensor Fusion and Convolution Neural Network. IEEE Trans. Ind. Inform. 2022, 18, 3213–3223. [Google Scholar] [CrossRef]
- Łuczak, D.; Brock, S.; Siembab, K. Fault Detection and Localisation of a Three-Phase Inverter with Permanent Magnet Synchronous Motor Load Using a Convolutional Neural Network. Actuators 2023, 12, 125. [Google Scholar] [CrossRef]
- Liang, M.; Cao, P.; Tang, J. Rolling Bearing Fault Diagnosis Based on Feature Fusion with Parallel Convolutional Neural Network. Int. J. Adv. Manuf. Technol. 2021, 112, 819–831. [Google Scholar] [CrossRef]
- Toma, R.N.; Kim, J.-M. Induction Motor Bearing Fault Diagnosis Using Statistical Time Domain Features and Hypertuning of Classifiers. In Advances in Computer Science and Ubiquitous Computing; Park, J.J., Fong, S.J., Pan, Y., Sung, Y., Eds.; Springer: Singapore, 2021; pp. 259–265. [Google Scholar]
- Bi, X.; Cao, S.; Zhang, D. Diesel Engine Valve Clearance Fault Diagnosis Based on Improved Variational Mode Decomposition and Bispectrum. Energies 2019, 12, 661. [Google Scholar] [CrossRef] [Green Version]
- Wang, J.; Li, S.; Xin, Y.; An, Z. Gear Fault Intelligent Diagnosis Based on Frequency-Domain Feature Extraction. J. Vib. Eng. Technol. 2019, 7, 159–166. [Google Scholar] [CrossRef]
- Li, H.; Feng, G.; Zhen, D.; Gu, F.; Ball, A.D. A Normalized Frequency-Domain Energy Operator for Broken Rotor Bar Fault Diagnosis. IEEE Trans. Instrum. Meas. 2021, 70, 3009011. [Google Scholar] [CrossRef]
- Luczak, D. Frequency Analysis of Mechanical Resonance in Direct Drive. In Proceedings of the 2012 12th IEEE International Workshop on Advanced Motion Control (AMC), Sarajevo, Bosnia and Herzegovina, 25–27 March 2012; pp. 1–5. [Google Scholar]
- Ramteke, S.M.; Chelladurai, H.; Amarnath, M. Diagnosis and Classification of Diesel Engine Components Faults Using Time–Frequency and Machine Learning Approach. J. Vib. Eng. Technol. 2022, 10, 175–192. [Google Scholar] [CrossRef]
- Lim, H.; Kwon, G.-Y.; Shin, Y.-J. Fault Detection and Localization of Shielded Cable via Optimal Detection of Time–Frequency-Domain Reflectometry. IEEE Trans. Instrum. Meas. 2021, 70, 3092514. [Google Scholar] [CrossRef]
- Iglesias-Martínez, M.E.; Antonino-Daviu, J.A.; Fernández de Córdoba, P.; Conejero, J.A. Rotor Fault Detection in Induction Motors Based on Time-Frequency Analysis Using the Bispectrum and the Autocovariance of Stray Flux Signals. Energies 2019, 12, 597. [Google Scholar] [CrossRef] [Green Version]
- Luczak, D. Spectral Analysis of Digital Filter Tuned for Mechanical Resonant Frequency Reduction in Multi-Mass Mechanical Systems in Electrical Direct Drive. Int. J. Simul. Syst. Sci. Technol. 2020, 17, 11.1–11.8. [Google Scholar] [CrossRef]
- Strack, J.L.; Carugati, I.; Orallo, C.M.; Maestri, S.O.; Donato, P.G.; Funes, M.A. Three-Phase Voltage Events Classification Algorithm Based on an Adaptive Threshold. Electr. Power Syst. Res. 2019, 172, 167–176. [Google Scholar] [CrossRef]
- Yoon, Y.; Brahma, A. Air–Fuel Ratio Imbalance Diagnostic of Spark-Ignited Engines With Modulated Sliding Discrete Fourier Transform. J. Dyn. Syst. Meas. Control 2020, 142, 081003. [Google Scholar] [CrossRef]
- Peña-Alzola, R.; Sztykiel, M.; Jones, C.E.; Norman, P.J.; Moore, G.; Pou, J.; Burt, G.M. First-Fault Detection in DC Distribution With IT Grounding Based on Sliding Discrete Fourier-Transform. IEEE Trans. Power Electron. 2021, 36, 3649–3654. [Google Scholar] [CrossRef]
- Li, X.; Bi, F.; Zhang, L.; Yang, X.; Zhang, G. An Engine Fault Detection Method Based on the Deep Echo State Network and Improved Multi-Verse Optimizer. Energies 2022, 15, 1205. [Google Scholar] [CrossRef]
- Gu, J.; Peng, Y.; Lu, H.; Chang, X.; Chen, G. A Novel Fault Diagnosis Method of Rotating Machinery via VMD, CWT and Improved CNN. Measurement 2022, 200, 111635. [Google Scholar] [CrossRef]
- Łuczak, D. Mechanical Resonance Frequensies Identyfication of Direct Drive Using Wavelet Analysis. In Proceedings of the 2012 17th International Conference on Methods & Models in Automation & Robotics (MMAR), Miedzyzdroje, Poland, 27–30 August 2012; pp. 29–32. [Google Scholar]
- Łuczak, D. Mechanical Vibrations Analysis in Direct Drive Using CWT with Complex Morlet Wavelet. Power Electron. Drives 2023, 8, 65–73. [Google Scholar] [CrossRef]
- Singleton, R. A Method for Computing the Fast Fourier Transform with Auxiliary Memory and Limited High-Speed Storage. IEEE Trans. Audio Electroacoust. 1967, 15, 91–98. [Google Scholar] [CrossRef]
- Cooley, J.W.; Tukey, J.W. An Algorithm for the Machine Calculation of Complex Fourier Series. Math. Comput. 1965, 19, 297–301. [Google Scholar] [CrossRef]
- Corinthios, M.J.; Smith, K.C.; Yen, J.L. A Parallel Radix-4 Fast Fourier Transform Computer. IEEE Trans. Comput. 1975, C-24, 80–92. [Google Scholar] [CrossRef]
- Corinthios, M.J. A Fast Fourier Transform for High-Speed Signal Processing. IEEE Trans. Comput. 1971, C-20, 843–846. [Google Scholar] [CrossRef]
- Soares, M.N.; Mollet, Y.; Kinnaert, M.; Gyselinck, J.; Helsen, J. Multiphysical Time- and Frequency-Domain Fault Detection and Isolation Technique for Power-Electronic Converters in DFIG Wind Turbines. IEEE Trans. Power Electron. 2021, 36, 3793–3802. [Google Scholar] [CrossRef]
- Han, T.; Ding, L.; Qi, D.; Li, C.; Fu, Z.; Chen, W. Compound Faults Diagnosis Method for Wind Turbine Mainshaft Bearing with Teager and Second-Order Stochastic Resonance. Measurement 2022, 202, 111931. [Google Scholar] [CrossRef]
- Li, H.; Fan, B.; Jia, R.; Zhai, F.; Bai, L.; Luo, X. Research on Multi-Domain Fault Diagnosis of Gearbox of Wind Turbine Based on Adaptive Variational Mode Decomposition and Extreme Learning Machine Algorithms. Energies 2020, 13, 1375. [Google Scholar] [CrossRef] [Green Version]
- Liu, D.; Cheng, W.; Wen, W. Rolling Bearing Fault Diagnosis via STFT and Improved Instantaneous Frequency Estimation Method. Procedia Manuf. 2020, 49, 166–172. [Google Scholar] [CrossRef]
- Wang, Z.; Yang, J.; Li, H.; Zhen, D.; Xu, Y.; Gu, F. Fault Identification of Broken Rotor Bars in Induction Motors Using an Improved Cyclic Modulation Spectral Analysis. Energies 2019, 12, 3279. [Google Scholar] [CrossRef] [Green Version]
- Grcić, I.; Pandžić, H.; Novosel, D. Fault Detection in DC Microgrids Using Short-Time Fourier Transform. Energies 2021, 14, 277. [Google Scholar] [CrossRef]
- Fu, X.; Yang, K.; Liu, M.; Xing, T.; Wu, C. LightFD: Real-Time Fault Diagnosis with Edge Intelligence for Power Transformers. Sensors 2022, 22, 5296. [Google Scholar] [CrossRef]
- Liu, J.; Hao, R.; Zhang, T.; Wang, X. Vibration Fault Diagnosis Based on Stochastic Configuration Neural Networks. Neurocomputing 2021, 434, 98–125. [Google Scholar] [CrossRef]
- Miao, Y.; Zhang, B.; Li, C.; Lin, J.; Zhang, D. Feature Mode Decomposition: New Decomposition Theory for Rotating Machinery Fault Diagnosis. IEEE Trans. Ind. Electron. 2023, 70, 1949–1960. [Google Scholar] [CrossRef]
- Shi, H.; Li, Y.; Bai, X.; Zhang, K. Sound-Aided Fault Feature Extraction Method for Rolling Bearings Based on Stochastic Resonance and Time-Domain Index Fusion. Appl. Acoust. 2022, 189, 108611. [Google Scholar] [CrossRef]
- Douglas, S.C.; Soh, J.K. A Numerically-Stable Sliding-Window Estimator and Its Application to Adaptive Filters. In Proceedings of the Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136), Pacific Grove, CA, USA,, 2–5 November 1997; Volume 1, pp. 111–115. [Google Scholar]
- Jacobsen, E.; Lyons, R. The Sliding DFT. IEEE Signal Process. Mag. 2003, 20, 74–80. [Google Scholar] [CrossRef] [Green Version]
- Jacobsen, E.; Lyons, R. An Update to the Sliding DFT. IEEE Signal Process. Mag. 2004, 21, 110–111. [Google Scholar] [CrossRef]
- Duda, K. Accurate, Guaranteed Stable, Sliding Discrete Fourier Transform [DSP Tips & Tricks]. IEEE Signal Process. Mag. 2010, 27, 124–127. [Google Scholar] [CrossRef]
- Park, C.-S. Fast, Accurate, and Guaranteed Stable Sliding Discrete Fourier Transform [Sp Tips&Tricks]. IEEE Signal Process. Mag. 2015, 32, 145–156. [Google Scholar] [CrossRef]
- Gudovskiy, D.A.; Chu, L. An Accurate and Stable Sliding DFT Computed by a Modified CIC Filter [Tips & Tricks]. IEEE Signal Process. Mag. 2017, 34, 89–93. [Google Scholar] [CrossRef]
- Park, C.-S. Guaranteed-Stable Sliding DFT Algorithm With Minimal Computational Requirements. IEEE Trans. Signal Process. 2017, 65, 5281–5288. [Google Scholar] [CrossRef]
- Kollar, Z.; Plesznik, F.; Trumpf, S. Observer-Based Recursive Sliding Discrete Fourier Transform [Tips & Tricks]. IEEE Signal Process. Mag. 2018, 35, 100–106. [Google Scholar] [CrossRef]
- Eclipse Mosquitto. Available online: https://mosquitto.org/ (accessed on 22 November 2022).
- Create MQTT Client Connected to Broker—MATLAB. Available online: https://www.mathworks.com/help/icomm/ug/icomm.mqtt.client.html (accessed on 22 November 2022).
- Train Models to Classify Data Using Supervised Machine Learning—MATLAB. Available online: https://www.mathworks.com/help/stats/classificationlearner-app.html (accessed on 23 November 2022).
- Deep Learning Toolbox. Available online: https://www.mathworks.com/products/deep-learning.html (accessed on 7 February 2023).
IoT Connectivity | Types of Fault and Labels | Signal or Sensor | Features Extraction Method | Features | Classificator | Article |
---|---|---|---|---|---|---|
MQTT and HTTP | demonstration of fan blades’ imbalance (normal, fan off, fan with fault) | 3-axis accelerometer and 3-axis gyroscope | SDFT or STFT at six axis | RGB image made of six time–frequency domain data | CNN | Proposed |
MQTT and HTTP | demonstration of fan blades’ imbalance (normal, fan off, fan with fault) | preliminary selected one axis of 3-axis accelerometer and 3-axis gyroscope | SDFT at one axis enhanced to STFT | frequency domain data | classical classifier | Enhanced |
Not specified | bearing (normal, inner ring, outer ring, ball) | vibration one-axis | STFT | color spectrogram of one signal | CNN | [15] |
Not specified | bearing normal and four faulty states (ball, inner ring, outer ring, inner + outer) | 3-axis accelerometer | frequency transform with weight map | frequency domain for each axis | CNN | [16] |
Not specified | blades non-damaged and two fault (5% and 15% broken blades) | from one axis to 3-axis of angular velocity | WPT (wavelet packet transform)—wavelet name not specified | third level of WPT decomposition | LSTM (long and short-term memory) | [17] |
Not specified | bearing (normal, outer, ball, inner) | raw data is one-dimensional signal; sensor is not specified | CWT (continuous wavelet transform), STFT | CWT, time domain, and frequency domain features aggregation | MIMTNet (multiple—input, multiple—task CNN) | [18] |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Łuczak, D.; Brock, S.; Siembab, K. Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity. Sensors 2023, 23, 3755. https://doi.org/10.3390/s23073755
Łuczak D, Brock S, Siembab K. Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity. Sensors. 2023; 23(7):3755. https://doi.org/10.3390/s23073755
Chicago/Turabian StyleŁuczak, Dominik, Stefan Brock, and Krzysztof Siembab. 2023. "Cloud Based Fault Diagnosis by Convolutional Neural Network as Time–Frequency RGB Image Recognition of Industrial Machine Vibration with Internet of Things Connectivity" Sensors 23, no. 7: 3755. https://doi.org/10.3390/s23073755