Interaction with Industrial Digital Twin Using Neuro-Symbolic Reasoning
Abstract
:1. Introduction
2. Background
2.1. Digital Twins
2.2. Neuro-Symbolic AI
3. Digital Twin and Neuro-Symbolic Dataset
3.1. Overview
3.2. Neuro-Symbolic Dataset
3.2.1. Symbolic Vocabulary
- Attach-Performing installation of object A on object B;
- Detach-Performing disassembly of object A from object B;
- Rotate-Performing rotation operation on object A on specific angle B.
- FilterType-Traversing information nodes and filtering of certain types of nodes;
- Count-Counting the number of objects in a given list;
- 3D Filter Attr-Traversing 3D components and filtering of models based on attributes and values.
3.2.2. Neuro-Symbolic Dataset of Queries
- “query” represents the user requests;
- “symbolic programs” defines the corresponding symbolic programs;
- “context” provides information on the task, subtask, and instruction of the query;
- “reply” shows the expected answer from the execution of the request.
4. Neuro-Symbolic Reasoning
4.1. Overview
4.2. Understanding User Requests
4.3. Execution of Operations
- ExtractNumbers Query—Extract numbers from the query, where the query is a symbol that represents a user request in the system. The outputs of this step are the numbers that exist in the query.
- CreateActions Attach—Based on predicted parameters, queries in the form of actions are represented in case the query is a command-to-action type.
- SaveVal2Val—Saving computed values from the previous execution to be used later. Some queries are more complex than just sequential processing; therefore, the temporary values are stored.
- FilterType Tasks Root—Traversing manuals to find nodes with type tasks.
- FilterAttr task_id current_task_id—Traversing tasks to find the task of the context.
- FilterType Subtasks—Traversing the current task to find nodes with type subtasks.
- FilterAttr subtask_id current_subtask_id—Traversing subtasks to find the subtask of the context.
- FilterType Instructions—Traversing the current subtask to find nodes with type instructions.
- FilterAttr order current_instruction_order—Traversing instructions to find the instruction of the context.
- FilterType Actions—Traversing the current instruction to find nodes with type actions.
- CheckActionsValidity—Checking whether the requested actions from the user query, which are saved in the temporary variable in step 3, overlap with valid actions of the current context.
- Attach—Performing actual actions requested by the user in case they are valid.
4.4. Digital Twin Interaction Software
4.5. Choosing Neuro-Symbolic AI over Neural Networks
- Scalability. Digital twins usually represent complex structures that may consist of millions of components (for example, aircraft maintenance digital twins) that change regularly. Therefore, using neural networks with embedded context is not scalable since every time a new context appears, neural nets must be retrained to be updated. On the other hand, neuro-symbolic AI has a mechanism to interact with context (manuals and 3D models) instead of memorizing it. This makes the modifications of manuals and 3D models much easier, without the need to change the reasoning.
- Explainability. The reasoning mechanism of neural networks is a black box and the results obtained are not explainable. In contrast, in our neuro-symbolic AI, the execution of symbolic programs can be analyzed step by step, bringing clarity to the inference process, and understanding the intermediate outputs.
- Finer Modifications. Along with the explainability of neuro-symbolic AI, finer modifications to the reasoning mechanism are possible. Since the intermediate steps in the execution of symbolic programs can be analyzed, a certain symbol can be updated based on needs, reflecting changes in the system. Therefore, the neural part of neuro-symbolic AI can remain unchanged while modifying the symbolic AI only, which is not the case for neural networks since the only solution is to retrain them.
5. Evaluation
5.1. Data
- Training Data—8000 queries;
- Test Data—1000 queries.
5.2. Metrics and Strategy
- GRU with Attention—Multi-layered gated recurrent unit with decoder attention, which applies an attention mechanism by computing the weighted sums of the encoder’s hidden states [50].
- GRU with Padding—Multi-layered gated recurrent unit architecture that applies packed padded sequences and masking. Padded sequences force the model to skip padding tokens in the encoder, whereas masking allows the network to ignore attention over padded elements.
- Transformer Attention—Multi-layered transformer model with attention mechanisms [51]. The encoder and decoder are made of multiple layers, with each layer consisting of multi-head attention and position-wise feed-forward sublayers.
- Neural accuracy is the ratio of the number of correctly predicted symbolic programs (or translated user requests) to the total number of input pairs (requests to programs).
- Bilingual Evaluation Understudy (BLEU) [52] is a metric used to assess automatic translation by measuring the difference between reference translations made by humans and machines for the same source sentence. The metric is language-independent and correlates highly with human evaluation. Since a narrow domain is applied in this work, a high BLEU score is expected [53].
- BERTScore [54] is a metric for the evaluation of automatic text generation. The BERTScore applies the pre-trained embeddings from the BERT model [55] and computes the semantic similarity between the reference and predicted tokens. The metric computes the cosine similarity between each word from the reference and predictions.
- Metric for the Evaluation of Translation with Explicit Ordering (METEOR) [56] is a machine translation evaluation technique that matches unigrams between the reference and predicted translations. METEOR is based on the harmonic mean of unigram precision and recall, giving higher weights to recall. It produces a substantial correlation with human judgment at the sentence level [57].
- Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [58] is a measure that counts the number of overlapping units, such as n-grams, word sequences, and word pairs, between machine-generated predictions and human-produced references [58]. In our case, ROUGE-1 is applied, which compares the similarity of unigrams between the reference and predicted summaries.
- Neuro-symbolic accuracy is the ratio of the number of correctly predicted responses (or executed symbolic programs) based on a given query and context to the total number of examples. This metric represents the accuracy of the whole neuro-symbolic reasoning pipeline.
- Failure rate is the ratio of the number of unsuccessful executions of generated symbolic programs to the total number of symbolic programs.
5.3. Results and Discussion
6. Conclusions and Future Works
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Mohammadi, N.; Taylor, J.E. Smart city digital twins. In Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI), Honolulu, HI, USA, 27 November–1 December 2017; pp. 1–5. [Google Scholar] [CrossRef]
- Vrabič, R.; Erkoyuncu, J.A.; Butala, P.; Roy, R. Digital twins: Understanding the added value of integrated models for through-life engineering services. Procedia Manuf. 2018, 16, 139–146. [Google Scholar] [CrossRef]
- Kritzinger, W.; Karner, M.; Traar, G.; Henjes, J.; Sihn, W. Digital Twin in manufacturing: A categorical literature review and classification. IFAC-PapersOnLine 2018, 51, 1016–1022. [Google Scholar] [CrossRef]
- Stavropoulos, P.; Mourtzis, D. Design and Operation of Production Networks for Mass Personalization in the Era of Cloud Technology; Elsevier: Amsterdam, The Netherlands, 2022; pp. 277–316. [Google Scholar] [CrossRef]
- Dimitrov, H.; Petrova-Antonova, D. 3D city model as a first step towards digital twin of sofia city. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2021, XLIII-B4-2021, 23–30. [Google Scholar] [CrossRef]
- Bartos, M.; Kerkez, B. Pipedream: An interactive digital twin model for natural and urban drainage systems. Environ. Model. Softw. 2021, 144, 105120. [Google Scholar] [CrossRef]
- Dahmen, U.; Rossmann, J. Experimentable Digital Twins for a Modeling and Simulation-based Engineering Approach. In Proceedings of the 2018 IEEE International Systems Engineering Symposium (ISSE), Rome, Italy, 1–3 October 2018; pp. 1–8. [Google Scholar] [CrossRef]
- Xiong, M.; Wang, H. Digital twin applications in aviation industry: A review. Int. J. Adv. Manuf. Technol. 2022, 121, 5677–5692. [Google Scholar] [CrossRef]
- Li, L.; Aslam, S.; Wileman, A.; Perinpanayagam, S. Digital Twin in Aerospace Industry: A Gentle Introduction. IEEE Access 2021, 10, 9543–9562. [Google Scholar] [CrossRef]
- Montejo-Ráez, A.; Jiménez-Zafra, S.M. Current Approaches and Applications in Natural Language Processing. Appl. Sci. 2022, 12, 4859. [Google Scholar] [CrossRef]
- Massaro, A.; Maritati, V.; Galiano, A.M. Automated Self-learning Chatbot Initially Build as a FAQs Database Information Retrieval System: Multi-level and Intelligent Universal Virtual Front-office Implementing Neural Network. Inform. Slov. 2018, 42, 515–525. [Google Scholar] [CrossRef]
- Mah, P.M.; Skalna, I.; Muzam, J. Natural Language Processing and Artificial Intelligence for Enterprise Management in the Era of Industry 4.0. Appl. Sci. 2022, 12, 9207. [Google Scholar] [CrossRef]
- Massaro, A.; Meuli, G.; Savino, N.; Galiano, A.M. Voice analysis rehabilitation platform based on LSTM algorithm. Int. J. Telemed. Clin. Pract. 2022, 3, 327–340. [Google Scholar] [CrossRef]
- Alexakis, G.; Panagiotakis, S.; Fragkakis, A.; Markakis, E.; Vassilakis, K. Control of Smart Home Operations Using Natural Language Processing, Voice Recognition and IoT Technologies in a Multi-Tier Architecture. Designs 2019, 3, 32. [Google Scholar] [CrossRef]
- Susskind, Z.; Arden, B.; John, L.K.; Stockton, P.; John, E.B. Neuro-Symbolic AI: An Emerging Class of AI Workloads and their Characterization. arXiv 2021, arXiv:2109.06133. [Google Scholar]
- Haenlein, M.; Kaplan, A. A Brief History of Artificial Intelligence: On the Past, Present, and Future of Artificial Intelligence. Calif. Manag. Rev. 2019, 61, 000812561986492. [Google Scholar] [CrossRef]
- Garcez, A.d.; Lamb, L.C. Neurosymbolic AI: The 3rd Wave. arXiv 2012, arXiv:2012.05876. [Google Scholar]
- Sittón-Candanedo, I.; Alonso, R.; Hernández Nieves, E.; Rodríguez, S.; Rivas, A. Neuro-Symbolic Hybrid Systems for Industry 4.0: A Systematic Mapping Study; Springer: Berlin/Heidelberg, Germany, 2019; pp. 455–465. [Google Scholar] [CrossRef]
- Boeing: Next-Generation 737. Available online: https://www.boeing.com/commercial/737ng/ (accessed on 12 December 2022).
- Aheleroff, S.; Mostashiri, N.; Xu, X.; Zhong, R.Y. Mass Personalisation as a Service in Industry 4.0: A Resilient Response Case Study. Adv. Eng. Inform. 2021, 50, 101438. [Google Scholar] [CrossRef]
- Aheleroff, S.; Huang, H.; Xu, X.; Zhong, R. Toward sustainability and resilience with Industry 4.0 and Industry 5.0. Front. Manuf. Technol. 2022, 26, 1–20. [Google Scholar] [CrossRef]
- Fuller, A.; Fan, Z.; Day, C.; Barlow, C. Digital Twin: Enabling Technologies, Challenges and Open Research. IEEE Access 2020, 8, 108952–108971. [Google Scholar] [CrossRef]
- Liu, Z.; Meyendorf, N.; Mrad, N. The role of data fusion in predictive maintenance using digital twin. AIP Conf. Proc. 2018, 1949, 020023. [Google Scholar] [CrossRef]
- Tao, F.; Xiao, B.; Qi, Q.; Cheng, J.; Ji, P. Digital twin modeling. J. Manuf. Syst. 2022, 64, 372–389. [Google Scholar] [CrossRef]
- Grieves, M. Digital Twin: Manufacturing Excellence Through Virtual Factory Replication. White Paper 2015, 1, 1–7. [Google Scholar]
- Aheleroff, S.; Xu, X.; Zhong, R.Y.; Lu, Y. Digital Twin as a Service (DTaaS) in Industry 4.0: An Architecture Reference Model. Adv. Eng. Inform. 2021, 47, 101225. [Google Scholar] [CrossRef]
- Zheng, Y.; Yang, S.; Cheng, H. An application framework of digital twin and its case study. J. Ambient. Intell. Humaniz. Comput. 2019, 10, 1141–1153. [Google Scholar] [CrossRef]
- Lohtander, M.; Ahonen, N.; Lanz, M.; Ratava, J.; Kaakkunen, J. Micro Manufacturing Unit and the Corresponding 3D-Model for the Digital Twin. Procedia Manuf. 2018, 25, 55–61. [Google Scholar] [CrossRef]
- Chen, X.; Kang, E.; Shiraishi, S.; Preciado, V.; Zhihao, J. Digital Behavioral Twins for Safe Connected Cars. In Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, New York, NY, USA, 14–19 October 2018; pp. 144–153. [Google Scholar] [CrossRef]
- Angjeliu, G.; Coronelli, D.; Cardani, G. Development of the simulation model for Digital Twin applications in historical masonry buildings: The integration between numerical and experimental reality. Comput. Struct. 2020, 238, 106282. [Google Scholar] [CrossRef]
- Mabkhot, M.M.; Al-Ahmari, A.M.; Salah, B.; Alkhalefah, H. Requirements of the Smart Factory System: A Survey and Perspective. Machines 2018, 6, 23. [Google Scholar] [CrossRef]
- Errandonea, I.; Beltrán, S.; Arrizabalaga, S. Digital Twin for maintenance: A literature review. Comput. Ind. 2020, 123, 103316. [Google Scholar] [CrossRef]
- Glaessgen, E.; Stargel, D. The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles. In Proceedings of the 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Honolulu, HI, USA, 23–26 April 2012. [Google Scholar] [CrossRef]
- Yin, Z.H.; Wang, L. Application and Development Prospect of Digital Twin Technology in Aerospace. IFAC-PapersOnLine 2020, 53, 732–737. [Google Scholar] [CrossRef]
- Oyekan, J.; Farnsworth, M.; Hutabarat, W.; Miller, D.; Tiwari, A. Applying a 6 DoF Robotic Arm and Digital Twin to Automate Fan-Blade Reconditioning for Aerospace Maintenance, Repair, and Overhaul. Sensors 2020, 20, 4637. [Google Scholar] [CrossRef]
- Qiu, C.; Zhou, S.; Liu, Z.; Gao, Q.; Tan, J. Digital assembly technology based on augmented reality and digital twins: A review. Virtual Real. Intell. Hardw. 2019, 1, 597–610. [Google Scholar] [CrossRef]
- Garnelo, M.; Shanahan, M. Reconciling deep learning with symbolic artificial intelligence: Representing objects and relations. Curr. Opin. Behav. Sci. 2019, 29, 17–23. [Google Scholar] [CrossRef]
- Hitzler, P.; Eberhart, A.; Ebrahimi, M.; Sarker, M.K.; Zhou, L. Neuro-symbolic approaches in artificial intelligence. Natl. Sci. Rev. 2022, 9, nwac035. [Google Scholar] [CrossRef]
- Kramer, S. A Brief History of Learning Symbolic Higher-Level Representations from Data (And a Curious Look Forward). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20; Bessiere, C., Ed.; International Joint Conferences on Artificial Intelligence Organization: Yokohama, Japan, 2020; pp. 4868–4876, Survey track. [Google Scholar] [CrossRef]
- Hassan, M.; Guan, H.; Melliou, A.; Wang, Y.; Sun, Q.; Zeng, S.; Liang, W.; Zhang, Y.; Zhang, Z.; Hu, Q.; et al. Neuro-Symbolic Learning: Principles and Applications in Ophthalmology. arXiv 2022, arXiv:2208.00374. [Google Scholar]
- Drancé, M. Neuro-Symbolic XAI: Application to Drug Repurposing for Rare Diseases. In Proceedings of the Database Systems for Advanced Applications; Bhattacharya, A., Lee Mong Li, J., Agrawal, D., Reddy, P.K., Mohania, M., Mondal, A., Goyal, V., Uday Kiran, R., Eds.; Springer International Publishing: Cham, Switzerland, 2022; pp. 539–543. [Google Scholar]
- Bennetot, A.; Laurent, J.L.; Chatila, R.; Díaz-Rodríguez, N. Towards Explainable Neural-Symbolic Visual Reasoning. arXiv 2019, arXiv:1909.09065. [Google Scholar] [CrossRef]
- Morel, G. Neuro-symbolic A.I. for the smart city. J. Phys. Conf. Ser. 2021, 2042, 012018. [Google Scholar] [CrossRef]
- Hanson, D.; Imran, A.; Vellanki, A.; Kanagaraj, S. A Neuro-Symbolic Humanlike Arm Controller for Sophia the Robot. arXiv 2020, arXiv:2010.13983. [Google Scholar]
- Unity Real-Time Development Platform|3D, 2D VR & AR Engine. Available online: https://unity.com/ (accessed on 12 August 2022).
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef]
- Sutskever, I.; Vinyals, O.; Le, Q.V. Sequence to Sequence Learning with Neural Networks. arXiv 2014, arXiv:arXiv:1409.3215. [Google Scholar]
- Chung, J.; Gülçehre, Ç.; Cho, K.; Bengio, Y. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv 2014, arXiv:1412.3555. [Google Scholar]
- Cho, K.; van Merrienboer, B.; Gülçehre, Ç.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv 2014, arXiv:1406.1078. [Google Scholar]
- Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv 2014, arXiv:1409.0473. [Google Scholar]
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv 2017, arXiv:1706.03762. [Google Scholar]
- Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002; Association for Computational Linguistics: Philadelphia, PA, USA, 2002; pp. 311–318. [Google Scholar]
- What Is a BLEU Score?-Custom Translator-Azure Cognitive Services|Microsoft Learn. Available online: https://learn.microsoft.com/en-us/azure/cognitive-services/translator/custom-translator/concepts/bleu-score (accessed on 23 December 2022).
- Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K.Q.; Artzi, Y. BERTScore: Evaluating Text Generation with BERT. In Proceedings of the International Conference on Learning Representations, Addis Ababa, Ethiopia, 26–30 April 2020. [Google Scholar]
- Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv 2018, arXiv:1810.04805. [Google Scholar]
- Lavie, A.; Denkowski, M.J. The Meteor Metric for Automatic Evaluation of Machine Translation. Mach. Transl. 2009, 23, 105–115. [Google Scholar] [CrossRef]
- Agarwal, A.; Lavie, A. Meteor, M-BLEU and M-TER: Evaluation Metrics for High-Correlation with Human Rankings of Machine Translation Output. In Proceedings of the Third Workshop on Statistical Machine Translation; Association for Computational Linguistics: Columbus, OH, USA, 2008; pp. 115–118. [Google Scholar]
- Lin, C.Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Proceedings of the Text Summarization Branches Out; Association for Computational Linguistics: Barcelona, Spain, 2004; pp. 74–81. [Google Scholar]
Model | Accuracy | BLEU | BERTScore | METEOR | ROUGE |
---|---|---|---|---|---|
Multi-layered LSTM | 0.446 | 0.903 | 0.866 | 0.927 | 0.956 |
Multi-layered GRU | 0.962 | 0.989 | 0.867 | 0.944 | 0.994 |
GRU with Attention | 0.856 | 0.965 | 0.866 | 0.935 | 0.980 |
GRU with Padding | 0.907 | 0.974 | 0.866 | 0.938 | 0.986 |
Transformers Attention | 0.960 | 0.986 | 0.867 | 0.942 | 0.993 |
Model | Neural Accuracy | Neuro-Symbolic Accuracy | Fail Rate |
---|---|---|---|
Multi-layered LSTM | 0.446 | 0.449 | 0.010 |
Multi-layered GRU | 0.962 | 0.962 | 0.002 |
GRU with Attention | 0.856 | 0.856 | 0.010 |
GRU with Padding | 0.907 | 0.914 | 0.016 |
Transformers Attention | 0.960 | 0.960 | 0.006 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Siyaev, A.; Valiev, D.; Jo, G.-S. Interaction with Industrial Digital Twin Using Neuro-Symbolic Reasoning. Sensors 2023, 23, 1729. https://doi.org/10.3390/s23031729
Siyaev A, Valiev D, Jo G-S. Interaction with Industrial Digital Twin Using Neuro-Symbolic Reasoning. Sensors. 2023; 23(3):1729. https://doi.org/10.3390/s23031729
Chicago/Turabian StyleSiyaev, Aziz, Dilmurod Valiev, and Geun-Sik Jo. 2023. "Interaction with Industrial Digital Twin Using Neuro-Symbolic Reasoning" Sensors 23, no. 3: 1729. https://doi.org/10.3390/s23031729