Digital Twin to Control and Monitor an Industrial Cyber-Physical Environment Supported by Augmented Reality

Abstract

Increasing industrial development and digital transformations have given rise to a technology called Digital Twin (DT) that has the potential to break the barrier between physical and cyberspace. DT is a virtual and dynamic model enabled through a bidirectional data flow that creates high-reliability models with interconnection and fusion between the physical and digital systems for full integration. In smart manufacturing, this technology is increasingly used in research and industry. However, the studies conducted do not provide a definition or a single integrally connected model. To develop the Digital Twin shown in this research, the literature was reviewed to learn about the enabling technologies and architectures used at the industrial level. Then, a methodology was used to obtain the physical process information, create the digital environment, communicate the physical environment, apply simulation models in the digital environment, and parameterize the simulation environment with the physical process in real-time to obtain the digital twin supported with augmented reality. The system was implemented in the MPS-500 modular production station that has industrial sensors and actuators. The virtual environment was designed with Blender and Vuforia to create the augmented reality environment. In the proposed methodology, robust devices (field and control level) and low-cost embedded systems were used for the creation and communication of the virtual environment (monitoring and control); for the joint work of these technologies, they were carried out through the use of the following protocols: Open Platform Communications United Architecture (OPC UA), Ethernet, and machine to machine (M2M), with which a communication was achieved between the different levels of the automation pyramid. The results show that the proposed methodology for the implementation of the DT allows bidirectional communication between the physical and virtual environment and can also be visualized with the support of AR, thus providing its characteristics to the proposed DT. Digital Twin is an essential factor in creating virtual environments and improving applications between the real and digital world, establishing a bidirectional communication through the Ethernet protocol, with a communication time of approximately 100 ms. This technology interacts with the virtual environment and performs mappings, thus achieving timely and dynamic adjustment. This improves data management and production and incorporates process simulation and physical control in real-time, allowing to execute and trigger actions in the physical equipment simultaneously.

1. Introduction

Advances in manufacturing technologies and digital transformation are changing the industrial sector worldwide, implementing new concepts and cutting-edge technologies such as the Internet of Things (IoT), Big Data (BD), Cyber-Physical Systems (CPS), Cloud Computing (CC), and Artificial Intelligence (AI) [1]. These technologies have greatly stimulated the development of smart manufacturing, also called Industry 4.0, where there is greater data management that allows factories to save time, improve productivity, customize products, and respond flexibly to current requirements. Due to disruptions caused by the COVID-19 pandemic, most industries are considering a foray into Industry 4.0, taking advantage of its benefits to simulate working conditions in real-time, operate production systems remotely, and maintain sustainable operations [2].

Cyber-physical systems are a promising tool to transform many application fields positively. However, they are also a great challenge because there is much to be explored and discovered regarding cyber-physical integration for achieving smart manufacturing [3]. Currently, digital models used for process simulation are stored as cloud services. Due to updates or latency, these models may not reflect the actual state of the physical system, resulting in inaccurate simulation and control [4]. The industrial revolution has given rise to an emerging concept called DT that has the potential to break the barrier between physical and cyberspace in smart manufacturing [5,6]. It integrates CPS information and fuses heterogeneous data from multiple sources, facilitating the connection between virtual and actual data [7,8]. This technology can improve the integrity and feasibility of process planning, through training artificial intelligence and 3D computer-aided (CAD) models, and then build an evaluation and optimization twin for the transformation of theoretical processes into practical operations, improving the viability and effectiveness to optimize processes, and providing a friendly interface. DT is one of the main concepts associated with Industry 4.0. However, in the scientific literature, there is no single definition for this concept [9,10], no complete understanding, and no unifying model [11]. Studies show that DT still needs to form an integrally connected technology and is still in its early stage of development [12]. So far, very little research has been conducted in the industry because it is difficult to analyze and estimate the structural and environmental parameters in real-time considering dynamic changes, and also due to the high complexity of representing and modeling the physical processes involved in the production stage [13]. One of the main challenges is the need to monitor and interact with the physical entity to concentrate the functionalities of a highly reliable system that bridges and integrates the physical and virtual worlds [14]. In the industrial field, AR is a complementary technique that helps operators perform tasks such as assembly, assistance, visualization, and interaction. This technology also allows operators and technical personnel to manage industrial variables in a more dynamic and less risky way; it is also used for the maintenance and programming of industrial machinery. Augmented and virtual reality, also known as “extended reality” (XR), offer a new set of 3D applications that may revolutionize the factories of the future, enabling the development of novel virtual applications that included improved forms of collaboration, efficient training processes, and improved productivity [15]. AR allows the improvement of the visualization and interactivity effects, where the digital twin receives data from sensors and controllers. Its main features include real-time monitoring, reprogramming, offline simulation, virtual test bench for optimization, and operator training, improving traditional methods because more accurate data and timely responses to dynamic process events can be obtained. AR and DT are tools that facilitate the fusion of the virtual and the physical and have shown a growing interest in the area of research and industry under the wave of the human-centered concept such as Industry 5.0, being one of the enabling technologies that can positively influence the man–machine relationship [16].

Several authors have conducted research in these areas to show their potential and have presented various methodologies, such as in [17], which presents a dynamic system for operators of “virtual eyes and hands” in the physical system; each system produces and consumes services to have interoperability, that is, it has heterogeneous and remotely accessible web services, and also a middleware to allow interaction with existing legacy systems (SCADA, ERP). Reference [18] presents an implementation of an architecture that covers four production stages, namely: design, operation, optimization, and validation, and implementation where a model is created to simulate the behavior of the physical system, and later bidirectional communication between the real world (PLC) and the virtual model is established. In [19], they propose a semi-automatic methodology to generate a DT of an industrial process that consists of extracting information from the diagrams and converting the information to a graphical format to generate a simulation model, then configure and parameterize the simulation model according to process data to obtain a digital twin. The results of several works implemented in the industry have shown an improvement in the productivity of the processes and quality standards, in addition to having the capacity for simulation, adaptive automation, and evaluation of new technologies without disturbing the production of the plant using industrial protocols such as OPC-UA.

The DT concept, introduced with the Industry 4.0 revolution, will continue to change many areas of our lives [20]. Several enabling technologies contribute to creating the Digital Twin of a physical process by collecting different data sets to develop updated digital simulation models based on the generated knowledge, implement optimization strategies, and evaluate the behavior of the physical environment [21]. By 2025, the global market is expected to reach USD 26.07 billion, with an annual growth rate of 38.2% [22], which will bring significant changes in many areas of our lives. Several pioneering companies in developing products for industrial automation are working to create Digital Twins—for example, General Electric, Hirotec Corporation, Rockwell Automation, ABB, Siemens, etc. [23]. The development of DTs supports the growing and accelerating demand for industrial evolution, increasing functionality, reliability, self-prediction, and autonomy to react to unexpected events [24]. Although DTs are very important, a methodology has yet to be defined to create and use them in smart manufacturing [25]. It is evident that there are several works carried out; however, there are no established methodologies or protocols to implement DT in the industry, for which the present work seeks to contribute with new proposals combining industrial hardware and software with embedded systems and immersion technologies.

This research shows the development of a Digital Twin applied to an industrial process. A methodology was used to obtain the physical process information, create the digital environment, communicate the physical and real environment, apply simulation models in the digital environment, and parameterize the simulation environment with the physical process in real time to obtain the Digital Twin. The article has several sections; Section 2 shows the key concepts for the development of DT; Section 3 shows the design and implementation of the DT; Section 4 shows the results and discussion; and Section 5 shows the conclusions.

2. Related Work

During the last decade, researchers have been working on the development of DT due to its characteristics and functionalities shown; however, in the industrial area, there are many challenges still existing mainly in undefined methodologies and architectures. Below are several works carried out in the industry with DT at the implementation and simulation level.

Among the industry 4.0 applications, there are many related works, where authors have conducted studies using different enabling technologies to create the virtual model of the process. For example, refs. [26,27,28] shows a path for creating completely functional DTs where continuous production systems are used to reproduce their physical features for registration, inspection, and digital communication. The objective is to have a structure without automation and obtain the data to perform a virtual analysis of various parameters such as predictive maintenance, dynamics, multiservice platform, configuration, and adaptation with high-efficiency automatic systems that are compatible with the planning system and the ERP (Enterprise Resource Planning) to obtain clients’ requirements and gradually develop the automation, to achieve better planning and cost reduction. In addition, the scenarios of industry 4.0 DT applications will enable the creation of vertical and horizontal life cycle integration concepts. Different architectures with DTs have been proposed for the design and improvements of CPS. An open-source modular and flexible architecture is developed in [29] for process control, light protocols, and flexible tools for 3D modeling and visualization. The tools were chosen to model and represent machinery and production systems. The virtual environment converges with reality, producing reliable data to validate execution time and enable the implementation of actions on the real plant based on data obtained through simulation. A CPS is created in [30] for the design and control using three enabling technologies: fast mapping method of distributed controllers, extensible distributed communication framework, and multiscale modeling method. Experimental results show that the CPS may achieve a fast design and distributed control for customized and flexible design. A digital mapping model of a physical production plant is realized in [31]. The model has four layers: dimension modeling, object modeling, monitoring, and prediction using the Markov chain. Results showed that the DT is the key technology in the virtual physical merging of the CPS, and in turn, increases the capabilities of operators. A systematic method is proposed in [32] for customized production of furniture, which includes high-fidelity 3D modeling, modeling of mechanisms to simulate movement, and data synchronization in real-time. Results showed that product quality and efficiency may be improved and may achieve virtual monitoring of devices in real-time. An operational decision-making system is proposed in [33] for an industrial environment with a high variety of products and demand; for this purpose, the principles of Industry 4.0 were adapted with a DT constituted by simulation and artificial intelligence focused on the planning of operational resources and cyclic and continuous decision-making. Results showed a decrease in the number of operators and a reduction in the delivery time.

A way to integrate a digital shadow (unidirectional communication) system with an MES system is proposed in [34] to create a DT that will be used for decision-making about the error state, and another to unchain low-quality dismantling processes, using the M2M communication protocol to create a communication channel between two levels and with an intelligent layer that houses rules and knowledge. The DT is simulated and integrated with the MES of the Industry 4.0 Laboratory, where the proposed frameworks have been tested and validated.

Current simulation techniques exhibit a high technological development that enables the simulation of all phases of the life cycle of an industrial process with high precision and complexity, where the algorithms proposed are executed to analyze their performance in the presence of different events. A computer simulation of discrete events through a DT is carried out in [35]; such a simulation consists of three stages: definition of future scenarios, periodic executions, and decision-making based on the integration with data in a real environment of a logistic process in the aeronautical industry. Results showed the possibility of optimizing decision-making related to supply routes and the integration of the simulation model with the ERP. A system for identifying, modeling, monitoring, and optimizing dynamic changes of small objects is proposed in [5]. A hybrid neural network model with a learning algorithm was built with the data simulated, for synchronizing the physical and virtual systems. Results show the effectiveness of the proposed method, with a higher detection precision for a DT in intelligent manufacturing. A cloud-based reference model for a CPS with DT is presented in [36]. A hybrid neural network model with a learning algorithm was built with the data simulated, for synchronizing the physical and virtual systems. Results show the effectiveness of the proposed method, with a higher detection precision for a DT in intelligent manufacturing. A cloud-based reference model for a CPS with DT has been presented in [37]. The authors present a methodology to reduce the number of messages through the DT between the CPS components, and in this way maintain a uniform communication interface and carry out tasks securely through a classification of events for a uniform distribution and to guarantee architecture scalability. Specific aspects and behaviors of the system are modeled in [38] to obtain information about the state or indicators through the aggregation of black-box modules to the simulation model. Results showed that it may be used to observe various aspects of the process and for updating the real system with improvements that may be obtained in the digital model. An IIoT architecture with DT is proposed in [39,40] to acquire data and features using cluster-based deep learning for anomaly detection. Results showed an improvement in learning, convergence, and energy-saving compared with various last-generation anomaly detection algorithms.

Among the different proposals presented by the authors, it is evident that one of the greatest challenges is creating reliable digital models with acceptable computational costs, low latency, and deep analysis methods. On the other hand, the opportunities open the possibility of bidirectional information transfer, diagnosis, optimization, replication of digital processes, development of intelligent systems, and predictive analysis using intelligent control techniques.

3. Materials and Methods

The contribution of the article is: (1) A method of control and monitoring of industrial processes based on digital twins is proposed, where immersive technology and industrial communication protocols are integrated. (2) Integrate technologies, such as augmented reality, and the Internet of Things, to link and integrate physical and virtual environments. (3) Based on the digital twin model, process conditions are analyzed through virtual simulation and immersive environments for process control and monitoring (see Figure 1).

The hardware and software used in this research are the MPS-500 modular production station that has industrial sensors and actuators to test the proposed methodology in a real environment. For the communication between the field and control level, the OPC UA protocol was used because there are devices of different brands. The PLC S7-1200 CPU 1212C is used for the DT implementation due to its versatility and Ethernet TCP/IP communication protocol. An SM 1233 digital expansion module connects the sorting station’s inputs and outputs through the SysLink interface. An Arduino mega is used with an Ethernet Shield module and a switch to establish an Ethernet TCP/IP network for communication between the virtual and the physical model. For the design of the DT, the Blender software is used, and to export and visualize with AR, it is performed using Unity version 2017.3.0f3; the META 2 glasses use the SDK “SDK2 Beta 2.4.0”. Additionally, the minimum PC requirements to run the AR app are Windows 10 (64-bit), Intel Core i7 6700 or AMD FX 9590, 16 GB DDR4 RAM, NVIDIA GeForce GTX 970 or AMD Radeon R9 390X. 10 GB of hard drive space, HDMI 1.4 b video output port, and USB 3.0 or higher port.

3.1. Digital Twin

DT is a virtual and dynamic model enabled through a data flow that creates highly reliable models with interconnection and fusion between the physical system and the created digital representation. These models contain all the information of the physical system and are fully integrated to exchange information in both directions [41,42]. The integration between IoT and data analytics simulates the physical counterpart’s characteristics, behavior, and performance by prediction, optimization, monitoring, control, and decision-making in real-time [43,44]. Interoperability, interchangeability, reusability, maintainability, flexibility, and autonomy throughout the life cycle are requirements for developing a DT [45].

i.

Digital model (DM): A digital representation of a physical object that does not use automatic data exchange between the digital model and the physical object.

ii.

Digital shadow (DS): This is a DM with unidirectional automatic flow between the state of the existing physical object and the digital counterpart.

iii.

Digital twin: There is a bidirectional flow between the state of the physical object and the digital object for total integration.

3.2. Cyber-Physical Systems (CPS)

CPSs are multidimensional interactive intelligent systems containing a set of physical devices, communication networks, and equipment interacting with virtual cyberspace [46], representing the virtualization of real systems from smart digital copies [47]. CPSs are the new backbone of digital systems; they use electronics, software, sensors, and wired and wireless network connectivity operating at virtual and physical levels [48,49], providing an essential element for designing interactive systems within an integrated environment and enabling connectivity and synchronization in Industry 4.0.

3.3. Industry 4.0

CPSs, Industrial Internet of Things (IIoT), and Cloud Computing contribute to establishing the fourth industrial revolution that seeks increasingly automated, integrated, and digitized processes, offering an opportunity to significantly improve operations’ quality and efficiency and enable customized production [50]. Industry 4.0 is driven by different technologies such as horizontal and vertical system integration, autonomous robots, simulation, augmented and virtual reality, IIoT, cybersecurity, additive manufacturing, big data, and analytics [51], providing autonomy, intelligence, and advanced connectivity to create an interactive and dynamic bridge between virtual systems and physical systems that are constantly changing in the industry [52]. Industry 4.0 is an emerging business paradigm that leverages the benefits of enabling technologies that drive smart systems and environments [53].

3.4. Industrial Internet of Things

IoT connects resources and collects data from the physical world. The IIoT is fundamental to Industry 4.0 as it enables the interconnection of smart heterogeneous objects (sensors, actuators, embedded systems, RFID, embedded computers, and mobile devices) through communication protocols and open interfaces [54]. This is one of the leading underlying technologies for Digital Twins [55] because it provides out-of-the-box IoT middleware solutions, allowing companies to choose open-source or licensed enterprise solutions, depending on their requirements [56].

3.5. META 2 Glasses

META 2 Glasses is a tool for interacting with Augmented Reality created in 2016. Its integrated sensors allow it to track movement in the environment. In addition, this tool facilitates the movement of virtual objects in real time with the help of fast reading of the user’s hands. It is worth noting that the META 2 Glasses need to be connected to the computer via cables to achieve a correct interaction with the AR application. This tool creates an environment giving depth to the image, making the user feel part of the environment and enjoy a realistic and intuitive experience [57].

4. Implementation and Results

Figure 2 shows the implementation of the Digital Twin. The different components were created individually and then assembled with Unity using Blender software. An S7-1200 PLC, an embedded system, and a switch were used to establish the bidirectional communication between the physical and the virtual part. The system operates manually or automatically and provides a digital interface containing all process components. Sensors and actuators are connected to a remote I/O interface for sending and receiving process data to the digital environment, following the OPC UA protocol for communication.

4.1. Communication

The communication protocol used is Modbus TCP/IP, and its configuration is shown in the class diagram in Figure 3. First, the Arduino encodes and decodes the data sent, for which the Ethernet and Modbus libraries are installed, and then the variables for sending and receiving data are created. Then the IP network, MAC, Gateway, and Subnet are configured, and the registers are started to send the data bi-directionally between the PLC and Unity.

Subsequently, the TIA Portal software is programmed with the configuration of the Modbus network parameters to send and receive data between PLC S71200 and Arduino. The programming in the TIA Portal software is performed in Lader language, while in Arduino, the reading and writing of the registers are performed through Modbus communication. The data blocks (DB) are configured for sending and receiving data in each direction, as shown in Figure 4a. Finally, serial communication between Unity and the embedded system is established. To achieve this, programming is performed in Visual Studio to develop serial communication, COM port assignment, and speed. The variables to transmit and store data are also created, and the digital and physical environment movements supported by augmented reality are interpreted. Figure 4b shows the class diagram of the configuration in the TIA Portal.

4.2. Digital Twin Implementation

The virtual space is the first part of the DT, which incorporates the 3D digital representation of the physical environment. This environment includes the attributes, properties, and operating rules of the process in the physical world. The DT was implemented in the sorting station of the MPS 500, which has industrial elements. Ethernet communication was used for sending and receiving data between the process and the digital environment. The hardware and software elements used are shown in Figure 5.

The system has a physical and virtual entity designed in a 3D model. These entities communicate bidirectionally and contain digital information, design parameters, and real-time sensor data. The proposed system must comply with several parameters described in

Table 1. Parameters for DT design.

Area	Requirements
R1. Communication	R1.1 The system provides communication services between the physical and virtual entities.
	R1.2 The system has bidirectional communication.
	R1.3 The system has real-time communication
R2. Operation	R2.1 The system receives the command from the operator.
	R2.2 The system executes the command sent by the operator.
	R2.3 The system tracks the status of the process.
	R2.4 The system detects objects and events of interest in the environment.
	R3.1 The system displays the status of the process.
	R3.2 The system displays the relevant sensor data.
R3. Information	R3.3 The system displays the simulation of the physical environment.
	R3.4 The system manages simulation data and analysis data

.

4.3. Results

As part of the results, a survey was carried out on 20 university professors with knowledge in the area of Industry 4.0 to determine the functionality and operability of the proposed DT. The survey consisted of five questions aimed at obtaining feedback from the research and knowing the point of view of the users regarding the functional, operational, and informative parts, as shown in

Table 2. Questions of operation and operability of the DT.

N°	Question
R1	Is the 3D digital environment created for the process simulation adequate?
R2	Would you like to use the DTs with AR as training before actual practice?
R3	Is the system capable of detecting events and objects of interest in the real environment?
R4	The system is capable of managing and processing the simulation data?
R5	Is there a perception in real time of the changes in the state of the process?

. For the selection of the people surveyed, their experience in the industrial field, and their professional training in the area of study, later the number of respondents was selected by simple random sampling to obtain their point of view and possible suggestions on the proposed methodology.

Figure 6 shows the results of the survey. A total of 87.5% of the respondents fully agreed with questions R1 and R2, which are informative and evaluate user satisfaction with the WP. Questions R3, R4, and R5 assess the operation and functionality of the proposed system, with 91.6% of acceptance.

Communication tests of the physical and virtual environment were performed in real-time to observe events and data management. The virtual environment and mapping interaction allows executing and triggering actions on the physical equipment simultaneously, as shown in Figure 7a,b. All 3D design features (dimensions, surfaces, materials, etc.) must be considered to create the digital model. The 3D model of a piece can be complemented with the product manufacturing information (PMI), according to the ISO 16792 standard [58].

The packet delivery time between the physical and virtual entities was analyzed. For this purpose, the WireShark tool was used in the Ethernet network, obtaining an average time of 100 ms. This time varies depending on the number of packets sent by the sensors, as shown in Figure 8.

5. Discussion

Due to their dynamic characteristics, Digital Twins are being valued in academia and industry. However, there is no defined architecture for their implementation. After reviewing different works and studies, we can say that the leading enabling technologies for developing DT are the Industrial Internet of Things, artificial intelligence, big data, cyber-physical systems, augmented reality, and virtual reality, which have had rapid development. These technologies are the right tools to support and improve manufacturing processes, offering an excellent opportunity to transform current manufacturing and implement smart manufacturing. These technologies also increase the ability of operators to understand, explore, and control production elements (See Figure 9).

The results show that the proposed methodology for the implementation of the DT allows bidirectional communication between the physical and virtual environment and can also be visualized with the support of AR, thus providing its characteristics to the proposed DT. When using an industrial environment with CPS that has robust devices and elements, communication with the DT cannot be carried out directly through the different communication protocols of each technology. However, thanks to the OPC UA, Ethernet, and M2M protocol, communication was achieved between the different levels of the automation pyramid and the components proposed for the DT. Among the advantages presented, it is shown that robust “PLC” devices that companies have with low-cost embedded systems can work together to take advantage of the advantages that each of these has.

One of the problems that arose in the investigation was the high consumption of computational resources that the META 2 glasses have, due to the high processing and graphics card that it requires for execution. Due to this, there was a small delay in the visualization of the changes and parameters of the DT with AR for the user; however, on the CPS screen it was observed in real time. CPS applications must be enhanced to achieve interconnection between the real world and its digital representation for manufacturing by creating virtual environments that are necessary and strategic to raise the quality and efficiency of several relevant factors in the industry, such as safety, optimization, monitoring, maintenance, installation, and prediction. DTs provide innovative solutions to improve features and fill gaps through simulation, improving manufacturing processes, and avoiding production problems. Figure 8 shows some of the opportunities and challenges of DTs. This technology has an excellent acceptance in industry and academia, which shows that it is an advance that is here to stay and has proposals at the simulation level that must be implemented to prove its functionality.

Several simulation-level investigations with promising results and architectures that can be tested in an actual implementation were considered to carry out this research. The results of this study were obtained by performance in a real process with a high-performance hardware and software infrastructure to simultaneously run the algorithms and trigger actions in the physical equipment. In addition, when using the META 2 glasses, an immersive environment of augmented reality was obtained, managing to increase the advantages presented by the Digital Twin.

6. Conclusions

DT is a promising technology that provides bidirectional communication between the real and digital worlds and improves manufacturing processes regarding safety, optimization, monitoring, maintenance, installation, prediction, etc. The leading technologies used to support and extend the capability of DT are the Industrial Internet of Things, artificial intelligence, big data, cyber-physical systems, augmented reality, and virtual reality.

The AR application to control industrial variables was developed with the appropriate versions and SDK for using META 2 glasses, creating an intuitive and easy-to-handle environment. Additionally, the Ethernet protocol established bidirectional communication between the physical and virtual environments for approximately 100 ms. PLC S7-1200 hardware, a switch, and an embedded system were used to communicate the systems for field and control level communication, and the OPC-UA protocol was used. Blender, Unity, Vuforia, and Visual Studio software were used to achieve the interaction with the virtual environment and mapping to perform the timely and dynamic adjustment through M2M communication. In this way, data management and production were improved, incorporating the simulation of the process and physical control in real time to execute and trigger actions in the physical equipment simultaneously.

The survey was carried out on university professors with knowledge in the area of Industry 4.0, where the results showed that 87.5% of those surveyed fully agreed with questions R1 and R2 regarding the digital environment used and the feasibility of using the DT as a training system. Meanwhile, in questions R3, R4, and R5, they evaluate the functioning and operation of the DT so that the industrial process complies with the established characteristics, and 91.6% acceptance was obtained.