Real-Time XR Maintenance Support Integrating Large Language Models in the Era of the Industrial Metaverse

Abstract

Recent advancements in Artificial Intelligence and eXtended Reality (XR) have laid solid foundations for the development of a new paradigm in industrial maintenance under the light of Industry 5.0 framework. This research presents the design, development, and implementation of an XR-enabled remote maintenance framework that integrates real-time video collaboration, AI-assisted guidance, and a persistent digital asset knowledge layer based on Asset Administration Shells for Maintenance and Repair Operations (MRO). By combining fine-tuned Large Language Models (LLMs) with immersive XR interfaces, the proposed framework enables technicians to interact with virtual representations of industrial assets, access contextual instructions, and receive expert support remotely in real-time. Through seamless integration of historical MRO data, digital twins, and real-time sensor streams, the system facilitates dynamic fault diagnostics and Remaining Useful Life (RUL) estimation. Therefore, the proposed approach is positioned as a Metaverse-aligned implementation, combining synchronous multi-user collaboration, digital–physical coupling through digital twins, and semantic interoperability. The framework is validated through two industrial case studies, demonstrating its feasibility and practical impact on maintenance efficiency and knowledge transfer. The findings position the Industrial Metaverse as a transformative enabler in the future of AI-driven machinery health monitoring.

1. Introduction

Artificial Intelligence has increasingly shaped the digital transformation of the manufacturing sector, which has been a dominant theme over the last decades [1,2]. This transformation has reshaped the operational landscape of industrial maintenance, introducing increased automation, cyber-physical connectivity, and real-time data acquisition throughout production systems [3]. As industrial assets evolve into complex cyber-physical entities, with the integration of heterogeneous sensor networks (e.g., Wireless Sensor Networks—WSN), advanced control units, and multilayer communication protocols, the corresponding MRO inherit significant challenges related to fault diagnostics, documentation quality, knowledge retention, and the consistent execution of corrective and preventive actions. These challenges are further amplified in high-mix, high-variability shop floor environments, where equipment heterogeneity and limited availability of expert personnel create critical bottlenecks for ensuring operational continuity [4].

Within the broader transition toward a more sustainable, resilient, and human-centric Industry, maintenance is increasingly viewed as a strategic lever for achieving human-centricity, resilience, and sustainable production [5]. Although modern predictive maintenance (PdM) approaches have demonstrated measurable improvements in equipment uptime, they remain heavily dependent on tacit knowledge and the ability of technicians to interpret complex technical documentation [6,7,8,9]. At the same time, the growing complexity of industrial machinery, combined with skill shortages and demographic pressures, emphasizes the need for intelligent digital tools capable of supporting workers proactively, reducing cognitive load, and ensuring safe and reliable task execution [10,11].

In this context, two rapidly maturing technological domains, namely (i) XR and (ii) LLMs, present unique opportunities for transforming industrial maintenance in the era of the Industrial Metaverse [12]. XR technologies enable immersive, spatially anchored visualizations of machine components, holographic step-by-step guidance, and real-time collaboration with remote experts. In parallel, fine-tuned LLMs exhibit strong capabilities in natural-language understanding and retrieval-augmented reasoning, generating task-specific instructions based on maintenance manuals, historical MRO logs, and sensor-derived contextual information [13,14,15,16]. Despite these advancements, existing literature reveals that XR and LLM technologies are rarely integrated within a unified maintenance ecosystem. Instead, current works typically treat them independently, by either focusing solely on XR-based assistance or LLM-based instruction generation, without addressing the need for interoperable data pipelines, synchronized human–machine interaction, and persistent multi-user collaboration environments.

Beyond technological integration, an additional gap can be identified at the methodological level, particularly concerning the systematic evaluation of LLM-assisted functionality within industrial XR workflows. While recent studies demonstrate the feasibility of LLMs for maintenance-related question answering or instruction generation, their assessment is typically limited to offline benchmarks or textual response quality. On the other hand, XR-based maintenance evaluations focus primarily on usability, task execution, and spatial guidance, omitting the contribution of language-based reasoning components. Consequently, there is limited empirical evidence on how LLM-driven cognitive assistance interacts with immersive XR interfaces during real maintenance tasks, how it influences personnels’ behavior and decision-making, and how its impact can be evaluated alongside traditional maintenance performance indicators. The present study explicitly addresses this methodological gap by embedding an LLM within an XR-enabled industrial maintenance workflow and evaluating its contribution in situ, in conjunction with task accuracy, execution behavior, and user experience metrics.

Motivated by these gaps, in this research work the design and development of a framework which integrates immersive Mixed Reality (MR) guidance with a domain-optimized LLM inside an Industrial Metaverse-based maintenance environment. The proposed architecture is based on the implementation of cloud-hosted 3D models, Asset Administration Shell (AAS) structures for standardized data exchange, digital twins for contextual awareness, and Internet of Things (IoT)-enabled monitoring pipelines for real-time operational insights. Through real-time video streaming, spatial mapping, and natural-language interaction, the developed tool provides shop floor technicians with actionable, context-aware maintenance instructions, enhancing accuracy, reducing human error, and contributing to the mitigation of downtime in scenarios where expert presence is not immediately available. The integration of historical maintenance records, sensor streams, and digital twin data enables dynamic fault diagnostics and RUL estimation, laying the foundations for industrial Metaverse as a persistent, collaborative environment for predictive and prescriptive MRO.

Compared to existing approaches, found in the literature, relevant to the topics of (i) XR-based maintenance, (ii) digital twin-based maintenance, and (iii) applications of LLMs in industrial support tasks, this work contributes a unified and operational framework which integrates all the above-mentioned technologies within a single, human-centered architecture. Although prior XR maintenance studies primarily focus on visualization and remote guidance, and digital twin–based approaches emphasize asset modeling and simulation, the proposed framework establishes a persistent, semantically structured asset knowledge layer through AAS, which remains accessible across XR sessions and users. Furthermore, unlike recent LLM for-maintenance works that often explore isolated conversational or diagnostic capabilities, the AI assistant in this work is embedded as a constrained, context-aware decision-support component, grounded in asset-specific data and operating under a human-in-the-loop paradigm. As a result, the presented approach advances beyond isolated technological solutions by demonstrating how XR collaboration, digital twins, and AI assistance can be systematically combined and validated in real industrial maintenance scenarios.

The rest of the manuscript is structured as follows. In Section 2 the most pertinent and relevant literature is investigated, emphasizing the topics of XR, industrial maintenance, and Large Language Models (LLMs). Section 3 presents and discusses the proposed system architecture, including the main modules and the information flow. Section 4 provides technical details regarding the implementation strategy. Section 5 introduces two industrial case studies used to evaluate the applicability and performance of the proposed framework. Finally, Section 6 concludes the manuscript by summarizing the key findings, limitations, and directions for future research.

2. State of the Art

The progressive digitalization of manufacturing has fundamentally reshaped the expectations placed on modern MRO [17,18,19,20]. As industrial systems become increasingly cyber-physical based on the integration of dense sensor networks, interconnected control units, and AI-enabled monitoring infrastructures, the complexity of diagnosing failures and performing corrective actions has become more complex [21,22,23,24,25]. PdM approaches, enabled by Machine Learning, digital twins, and advanced signal analytics, have been shown to reduce unplanned downtime and improve system reliability [26,27,28,29]. Nevertheless, industrial studies report that the economic impact of rare and unpredictable failures remains disproportionately high, often exceeding 29% higher cost per event compared to more predictable scenarios [30]. These findings highlight the persistent difficulty of interpreting complex machine behaviour in real-time and underscore the ongoing dependency on scarce human expertise for accurate fault identification and procedural decision-making. Within the Industry 5.0 paradigm, the problem is reframed beyond efficiency alone, namely maintenance must become human-centric, resilient, and deeply integrated with cognitive support tools that elevate the capabilities of technicians rather than simply automate isolated tasks [31,32,33].

Against this backdrop, XR has emerged as a critical enabler of the next generation of industrial maintenance systems. XR technologies, including Augmented Reality (AR), MR, and Virtual Reality (VR) offer shop floor personnel spatially anchored representations of equipment, holographic overlays of hidden components, and real-time interactive guidance during complex procedures [34,35,36,37,38]. Prior research demonstrates that XR-supported interventions can substantially reduce technician errors, enhance environmental understanding, and improve procedural consistency, particularly when dealing with machines that lack complete documentation or have undergone undocumented retrofits [39]. XR also enables remote expert support through real-time video streaming and holographic annotation, a capability that has proven valuable in both predictive and reactive maintenance scenarios by reducing expert travel time and improving the clarity of remote instructions [40]. When technicians encounter deviations from expected machine configurations or face previously unseen failure modes, conventional XR systems lack the adaptive reasoning needed to adjust guidance dynamically.

LLMs offer a promising solution to this limitation. LLMs have demonstrated outstanding capabilities in natural-language understanding, context-aware reasoning, and the synthesis of procedural instructions from unstructured sources [41,42]. When adapted to industrial contexts, LLMs can process technician descriptions of symptoms, interpret fault indications, and generate repair strategies, structured as actionable steps/recommendations, aligned with domain-specific practices [1,43]. Recent research has explored the potential of fine-tuning LLMs on corpora of maintenance manuals, troubleshooting guides, and historical machine logs, enabling the models to acquire a deep understanding of technical vocabulary, disassembly sequences, and safety constraints [44,45]. Empirical evaluations show that domain-adapted LLMs substantially outperform general-purpose models, which often provide vague or technically inaccurate recommendations. In comparative assessments, fine-tuned LLMs generated detailed, equipment-specific instructions aligned with manufacturer documentation, whereas both pretrained LLMs and conventional AI assistants produced incomplete or overly generic suggestions that were unsuitable for real shop floor usage [46,47]. The potential of LLMs to act as conversational maintenance copilots is therefore significant, yet these models remain largely disconnected from the spatial and contextual information available in XR systems.

The convergence of XR and LLMs represents a cornerstone towards truly intelligent maintenance environments [44]. XR provides embodied interaction, spatial awareness, and visualization of complex machine states, while LLMs provide semantic interpretation, knowledge retrieval, and adaptive procedural generation [46]. When combined, technicians can describe anomalies in natural language and immediately receive AI-generated steps visualized as holograms anchored to the corresponding physical components. Early attempts at XR–LLM integration illustrate the potential of this synergy, but existing systems remain limited in scope. Current frameworks often treat XR and LLMs as loosely coupled modules, lacking the deep interoperability required for contextualized decision-making. Challenges remain in grounding language outputs in 3D geometry, aligning AI-generated instructions with real-time sensor data, and ensuring safety-compliant outputs in time-critical maintenance scenarios. These shortcomings highlight the need for unified hybrid architectures capable of merging XR’s embodied interaction capabilities with LLM-based cognitive reasoning within a coherent digital ecosystem.

In parallel to these developments, the emergence of the Industrial Metaverse introduces a broader conceptual and technological framework for unifying XR interfaces, AI agents, and real-time operational infrastructures. The Metaverse is increasingly interpreted not as a single virtual environment but as a persistent, interconnected digital–physical continuum [12,47]. Within this continuum, users interact with digital twins, virtual workspaces, 3D assets, and intelligent agents through multisensory interfaces, co-exist with other users, while the underlying infrastructure synchronizes continuously with real industrial processes. Recent analyses describe the Metaverse as a layered architecture incorporating network infrastructure, spatial computing, immersive interfaces, decentralized data governance, and application ecosystems that support collaborative engineering, remote operations, and real-time simulation of production activities [48]. XR technologies act as the primary gateway to this environment, enabling workers to perceive and manipulate virtual overlays embedded within their physical surroundings. The Metaverse’s persistent and multi-user nature allows maintenance tasks, training sessions, or diagnostic simulations to continue evolving even when users disconnect, forming a shared, continuously updated knowledge space for technicians and engineers.

In the context of this study, the Industrial Metaverse is defined operationally as a digitally augmented industrial environment that enables synchronous human collaboration, persistent linkage to physical assets, and real-time interaction across the physical–digital continuum. Based on this definition, an Industrial Metaverse implementation is characterized by the following core properties:

• XR-enabled immersion, allowing users to interact with digital content superimposed on real industrial environments;
• Synchronous multi-user collaboration, enabling geographically distributed stakeholders to co-exist and interact within a shared augmented space;
• Digital–physical coupling, supported through digital twins and live operational data;
• Semantic asset representation, enabling structured access to asset knowledge via standardized models (e.g., Asset Administration Shells);
• Cross-platform accessibility, allowing for interaction across heterogeneous devices such as desktop systems and XR head-mounted displays.

To further clarify the operational meaning of the Industrial Metaverse in this work, the main system components are mapped in relevance to commonly recognized Metaverse features. XR interfaces (head-mounted displays and desktop-based visualization) realize immersive interaction by superimposing digital content onto physical industrial environments, supporting user embodiment and spatial context awareness. Multi-user collaboration is enabled through real-time video communication and shared augmented content, allowing geographically distributed expert engineers and shop floor technicians to co-exist and interact within a common reality–virtuality continuum. Digital twins and live operational data streams provide continuous digital–physical coupling, ensuring that virtual representations reflect the current state of physical assets and maintenance activities. Persistence is supported at the application level through structured asset knowledge repositories, enabling maintenance information, annotations, and procedural knowledge to remain accessible across sessions rather than being transient. LLM–based assistance contributes cognitive support by enabling contextual information retrieval, procedural reasoning, and task-specific guidance within the immersive workflow, augmenting human decision-making rather than replacing it. As a whole, these components implement a coherent subset of Industrial Metaverse features in a manner compatible with real industrial constraints. While aspects such as global-scale persistence, open-world interoperability, and extensive multi-user scalability remain outside the scope of the current implementation, the proposed framework demonstrates how Metaverse-aligned concepts can be operationalized in practical industrial maintenance scenarios.

The proposed framework satisfies these properties at an application-driven level and is therefore positioned as a step toward the Industrial Metaverse, while acknowledging that additional characteristics such as large-scale persistence, open-world interoperability, and extensive multi-user scalability remain subjects of future research.

Manufacturing-focused studies increasingly position the Industrial Metaverse as a transformative extension of Industry 4.0 systems, emphasizing its role in enabling immersive visual analytics, hybrid human–AI interaction, and collaborative digital environments that persist across time and location [12]. For instance, virtual replicas of machines can be used to test alternative maintenance strategies, evaluate operational risks, or visualize the consequences of misconfigurations long before technicians intervene on the physical equipment [49]. These capabilities align closely with human-centricity and resilience goals for the industry of the next era, where the emphasis shifts toward empowering workers through augmented cognition, decision support, and intuitive interaction with complex data streams [50]. Despite its potential, however, current Industrial Metaverse implementations remain partial and fragmented. Many existing solutions capture isolated elements, such as XR-based visualization, digital twin simulations, or multi-user collaboration, without achieving full semantic interoperability, persistent synchronization, or real-time intelligent reasoning envisioned in theoretical models. Technical barriers such as limited XR ergonomics, incomplete data standardization, high computational demands, and insufficient integration between AI-driven reasoning and spatial computing continue to impede widespread adoption.

The literature review conducted in the previous paragraphs has revealed key trends in the topic of human-centricity, Industrial Metaverse, Generative AI (GenAI), XR, and other technologically related topics. However, among the key findings lie in the fact that technological momentum has been built towards immersive, intelligent, and human-centric maintenance ecosystems despite the existing substantial gaps. To that end, although XR technologies offer embodiment and spatial visualization, they lack adaptive reasoning. Complementary to that, LLMs provide semantic understanding and procedural generation but lack grounding in the physical world. Industrial Metaverse offers persistent, collaborative, and data-rich environments but lacks fully interoperable cognitive systems which are capable of acting within such environments. Therefore, these limitations motivate the need for integrated frameworks, combining XR engines, LLM-based reasoning, and Industrial Metaverse infrastructures, for supporting real-time, context-aware, safe, and effective MRO. Consequently, the proposed framework focuses on addressing this gap by unifying these domains into a coherent architecture designed to support Operator 5.0 and the next generation of intelligent industrial maintenance.

In the context of this work, the term Industrial Metaverse is not used to denote a fully immersive, large-scale virtual industrial universe, but rather to describe a convergence of enabling capabilities which extend beyond traditional XR-based remote assistance. Concretely, the proposed framework possesses fundamental Metaverse characteristics, including (i) persistent digital representations of physical assets through standardized Asset Administration Shells, (ii) real-time coupling between physical equipment and digital twins, (iii) synchronous multi-user collaboration within a shared digital space, and (iv) cross-platform accessibility across immersive and non-immersive devices. Simultaneously, aspects such as global-scale persistence, fully open-world interoperability, and large numbers of concurrent users remain outside the scope of the present implementation, and are yet to be investigated in the future.

3. Proposed System Architecture

Figure 1 illustrates the overall system architecture developed in this work, integrating XR, cloud-based data services, LLM capabilities, and industrial IoT infrastructures into a unified maintenance-support framework. The architecture follows a layered structure which mirrors the operational flow between shop floor assets, back-end data processing pipelines, and the engineering department. At the lowest layer, machine tools equipped with IoT sensing modules transmit real-time operational data through an IoT Gateway, which acts as the secure communication bridge between the physical shop floor and higher-level digital services. This information is semantically structured and stored within an AAS, which encapsulates sensor readings, asset metadata, and historical maintenance records, ensuring standardized interoperability across heterogeneous equipment.

Figure 1. General system architecture.

Above the AAS layer, cloud services host both the 3D CAD models required for XR visualization and the maintenance manuals used for knowledge extraction and LLM fine-tuning. The LLM module resides within the engineering environment and is responsible for generating maintenance tasks and context-aware instructions based on technician requests, real-time data, or historical records. The right segment of the architecture depicts the MR subsystem, which runs on head-mounted devices and connects directly to the LLM and cloud services. This subsystem provides the core functionalities for remote maintenance assistance, namely real-time video calling, holographic content insertion, IoT data visualization, and spatial mapping, using WebRTC for low-latency communication.

The proposed system architecture enables a continuous flow of information from the physical environment to cloud-based intelligence and back to the technician through immersive XR interfaces. Each module in the diagram is designed so that it can operate both independently and collaboratively, forming an integrated ecosystem which improves human–machine interaction, reduces downtime, and ultimately supports the transition toward human-centric, and resilient Maintenance 5.0 paradigms. In the following subsections, the key components of the architecture are presented and discussed in detail, highlighting their purpose, implementation, and contribution to the overall framework.

From an alternative viewpoint, emphasizing on Metaverse characteristics, the proposed architecture enables digital–physical convergence through distinct functional layers. The AAS repository provides a persistent and semantically structured representation of physical assets, enabling interoperability across tools and sessions. The digital twin layer maintains real-time synchronization with shop floor equipment, while the XR collaboration module enables synchronous co-presence, shared hologram placement, and contextual annotations. The AI assistant (i.e., the LLM) operates on top of these layers in order to support decision-making and procedural guidance, reinforcing the continuity of knowledge across maintenance activities.

3.1. XR Engine

The XR Engine constitutes the core subsystem responsible for the creation, manipulation, and spatial registration of holographic content within the proposed industrial maintenance framework. Its primary function is to bridge the virtual scene (i.e., the augmentations), comprising 3D models, annotations, and interactive overlays, with the physical environment perceived through the HMD (Head-Mounted Display). This is achieved through the development of a pipeline involving spatial mapping, coordinate transformation, real-time data exchange, and 3D asset rendering, following the architectural principles demonstrated in recent XR-enabled maintenance systems.

Upon initialization of the MR application on the HMD (e.g., Microsoft HoloLens 2), the device performs an autonomous scanning of the surrounding environment using its embedded depth sensors, inertial measurement units, and camera-based SLAM (Simultaneous Localization and Mapping). The scanning process constructs a 3D spatial mesh, consisting of triangular surfaces representing real-world geometry. This mesh is compressed, serialized, and transmitted to the desktop application via WebRTC DataChannel.

In order to ensure a consistent coordinate frame between the physical and virtual worlds, the user selects a virtual anchor point on the HMD interface. This anchor acts as the origin of the holographic coordinate system and is encoded as a 4 × 4 homogeneous transformation matrix $T_{anchor}$. In continuation, the HMD exports(i) the spatial mesh, (ii) the anchor position and orientation, and (iii) device pose estimates enabling the desktop application to reconstruct a synchronized virtual scene.

Holographic objects are not stored locally on the HMD (aiming to reduce the computational load). Instead, they are retrieved on demand from a cloud repository, consistent with the AAS-based data exchange mechanisms. When a technician or remote engineer selects a part for visualization, the desktop application sends a request to the cloud server including

• Component identifier;
• Assembly hierarchy position;
• Desired rendering parameters.

The server responds with the appropriate 3D CAD model (typically stored in glTF, FBX, or Unity asset bundle format). The AAS ensures that every component is linked to semantic metadata such as

• Asset ID;
• Maintenance history;
• Manufacturer specifications;
• Contextual AR/XR instructions.

Following retrieval, the model undergoes a lightweight preprocessing step:

• Mesh decimation (if necessary) to ensure real-time rendering.
• Collider generation for physics interactions.
• Material standardization to match MR rendering constraints (e.g., MRTK Standard Shader).
• Coordinate normalization relative to the anchor frame.

After preprocessing, the hologram is instantiated within the virtual scene using the transformation:

$$T_{world}=T_{anchor}\cdot T_{model}$$

where

• $T_{anchor}$ is the spatial anchor transformation;
• $T_{model}$ is the model’s local pose relative to its CAD origin.

The desktop engineer can manipulate the hologram in real-time using GUI controls (rotation, translation, scaling). These interactions are serialized into JSON packets and transmitted to the HMD via WebRTC.

On the HMD, the incoming data are

• Deserialized;
• Applied onto the existing hologram instance;
• Re-rendered within the spatial map.

This ensures frame-accurate registration, allowing holograms to remain stable even during user motion, consistent with the principles of spatial persistence in MR systems.

A key capability of the XR Engine is the real-time collaborative manipulation of holograms during maintenance sessions. When the expert engineer places an arrow, a line annotation, or a CAD component onto the technician’s field of view, the following sequence occurs:

• Desktop users generate an object (arrow, text, tool geometry, or CAD part).
• Object metadata and transformation are serialized.
• WebRTC DataChannels transmit the data to the HMD.
• The HMD reconstructs the object and aligns it with the spatial anchor.
• The technician views the hologram registered to the physical machine.

This mechanism enables precise, intuitive guidance even for complex maintenance tasks that traditionally require physical co-presence of expert personnel. The XR Engine utilizes MRTK interaction models to support hand-tracked gestures, gaze-assisted selection, ray-based interaction, and voice commands, enabling hands-free and intuitive manipulation of holographic content.

Gesture recognition is performed locally on the HMD using MRTK’s hand joint tracking system. The recognized gestures are mapped to hologram manipulation actions (e.g., rotation, selection, step navigation). This ensures minimal latency and preserves operability even under degraded network conditions.

To support predictive and prescriptive MRO tasks, the XR engine can dynamically bind holograms to real-time machine states. When IoT data streams indicate abnormal behavior or imminent failure, digital twin values such as spindle speed, vibration signals, or temperature are visualized as holographic panels co-located with the relevant machine component (discussed in greater detail in the next paragraphs). The mapping is defined through AAS submodels, enabling

• Contextual overlays (warnings, RUL estimates);
• Animated states (component rotations, color-coded stress indicators);
• And real-time synchronization between virtual and physical equipment.

The spatial registration approach adopted in this work is designed to ensure stable and consistent alignment between virtual content and physical assets under real industrial conditions. While the mathematical formulation of spatial transformations supports precise coordinate mapping, the present study does not aim to provide laboratory-grade metrological validation of absolute registration error. Instead, spatial alignment quality was assessed at the task-execution level, focusing on whether holographic annotations, component highlights, and guidance cues remained visually consistent and functionally aligned throughout maintenance activities.

Within the evaluated scenarios, this resulted in high perceived spatial consistency, which was sufficient to support effective holographic guidance, component identification, and remote expert interaction. Claims regarding spatial accuracy are therefore framed in terms of application-level suitability rather than absolute metric precision.

3.2. XR Authoring Tool

The authoring tool is a module which is relevant to the creation of XR content with ease, enabling furthermore the automation of the creation of digital content. Concretely, this module has been developed as an addon for the Unity 3D game engine, which is one of the most widely used tools for the development of XR applications. Through this addon, the users gain access to a variety of functionalities, which are discussed in detail in the following paragraphs. The general data model for this module is illustrated in Figure 2.

Figure 2. MR authoring tool data model (created by the authors).

The XR authoring tool is implemented as a Unity add-on that enables rapid creation of maintenance scenes, work instructions, and tool catalogs through a structured graphical interface. In Figure 3 the main GUI for the authoring tool is presented.

Figure 3. MR authoring tool GUI (created by the authors).

Through the work instruction editor, the users are capable of choosing the suitable components from the expanded list, which is automatically created by the analysis of the assembly (previously created and imported by the user). For that purpose, a JSON file is created automatically when a new assembly is imported to the tool. The JSON file is used so that the assembly components’ names (not the filenames) and the 3D geometries are bound to each other and are to be used in multiple instances at a later point in time. The components list can be seen in Figure 4 on the lower left part of the GUI (highlighted with yellow rectangle).

Figure 4. Work instruction generation menu.

The components list is interactable, so that the user can select multiple components in order to be included in the new scene. Since the assemblies for complex industrial equipment are expected to have large amounts of components, a part preview window has been implemented. This preview window displays a scaled down representation of the selected component, as seen in Figure 4 (highlighted with green rectangle).

The authoring tool additionally performs automated compatibility checks for required XR libraries and project configurations, ensuring consistency and reducing deployment errors before building the application for HMDs. These functionalities can be viewed in Figure 5.

Figure 5. New scene generation.

3.3. Real-Time Video Call

The real-time video call module enables synchronous communication between shop floor technicians and remote expert engineers through bidirectional audio–video streaming. The implementation is based on WebRTC, ensuring low-latency transmission optimized for XR devices such as Microsoft HoloLens 2.

Beyond traditional video calling, the proposed framework extends the communication channel by transmitting spatial data in real-time. In particular, the HMD continuously captures a point cloud representation of the technician’s immediate environment using its integrated depth cameras and SLAM subsystem. This point cloud is serialized and transmitted to the desktop application via WebRTC DataChannels. DataChannels enable low-latency transfer of arbitrary structured data in parallel with the video and audio streams, bypassing the need for server-side relaying.

The significance of point-cloud transmission lies in its direct support for remote hologram placement. By reconstructing the shop floor environment on the desktop side, the expert engineer interacts with a synchronized virtual replica of the technician’s physical workspace. The spatial mesh displayed on the desktop interface enables accurate visualization of environmental geometry, occlusions, and equipment positioning. When the expert places a holographic annotation or CAD component within this virtual mesh, the system transmits the updated transformation parameters back to the HMD. The device then reprojects the hologram into the technician’s field of view using the spatial anchor defined during MR initialization, achieving high perceived spatial consistency within the evaluated industrial setting. This mechanism enables real-time, collaborative manipulation of holograms and seamless registration of virtual objects onto physical equipment.

In order to maintain synchronization between video, point-cloud data, and hologram updates, the communication system uses timestamping and lightweight compression schemes. Point-cloud packets include metadata such as anchor coordinates, HMD pose, and update sequence numbers, ensuring consistent alignment even when frames arrive out of order or under variable network latency. Localization updates from the HMD are throttled to avoid network congestion while preserving sufficient temporal resolution for accurate hologram placement.

In parallel, the video stream provides a complementary perspective by allowing the expert engineer to observe the technician’s body posture, tool handling, and equipment accessibility. The combination of video, audio, and spatial mapping establishes a hybrid MR collaboration environment, where remote guidance approximates physical co-presence. This hybrid perceptual channel is critical for industrial maintenance scenarios, enabling expert oversight even in cases where sensor data or documentation are incomplete.

Overall, the real-time video call module transforms traditional remote support into a multi-modal MR communication pipeline, integrating video, audio, point-cloud streaming, and holographic interaction into a unified, latency-aware architecture. It serves as the backbone of the proposed Industrial Metaverse framework, supporting effective human–machine collaboration and enabling expert-assisted MRO activities without requiring on-site presence.

3.4. Equipment Monitoring

The monitoring module is responsible for tracking industrial equipment and extracting operational data in real-time, processing them, and providing useful visualizations for field engineers and technicians. This module is based on the utilization of QR (Quick Response) code reading for linking the physical machines to the corresponding data on the repository.

With the utilization of timeseries data in combination with LSTM-Autoencoders and Transformer Encoders [6], data recorded from the manufacturing equipment are retrieved from the repository and processed in order to predict eminent failures as well as for evaluating the RUL for specific components. Furthermore, in an attempt to provide a more robust visualization aid to the engineers and the shop floor technicians, a series of interactable MR Graphical User Interfaces (GUIs) have been developed, incorporating the data derived from the predictive AI tools. The first GUI is presented in Figure 6, in which the user can select the equipment of interest (alternatively they can scan the QR installed on the machine) and retrieve its status, and the RUL value calculated and saved on the server.

Figure 6. RUL prediction GUI.

Finally, a notification system has also been set up, so that the engineers and the shop floor technicians are notified in case of emergency events, such as the recording of abnormal values from the sensors installed on the manufacturing equipment. And example case is illustrated in Figure 7. The notification system does not only capture the abnormality, but it records other crucial information as well, including the sensor which recorded the abnormality and the identifier of the manufacturing equipment, since several sensors are installed on each machine.

Figure 7. Notification for urgent events.

The integration between the PdM module and the XR framework is realized through an explicit data interface that enables real-time contextual visualization of asset health information. During the experimental evaluation, the PdM module transmitted a selected subset of variables relevant to maintenance decision support, rather than the full internal model state.

Specifically, the transmitted variables included the current operational state of the monitored asset, predicted RUL estimates, health indicator trends, and threshold-based anomaly flags. These variables were generated by the PdM module based on sensor data streams and forwarded to the XR application through a lightweight communication layer.

Data exchange followed a structured message format (JSON), enabling platform-independent parsing and straightforward mapping to digital twin elements within the XR scene. Update frequency was configured according to the variable type: operational state and anomaly flags were updated at higher frequency, while RUL estimates and health trends were updated at lower frequency, reflecting their slower temporal dynamics.

Alert logic during the experiments was rule-based and threshold-driven. When health indicators or predicted RUL values exceeded predefined limits, alert messages were generated and visualized in the XR environment using contextual cues (e.g., color-coded annotations and warning icons) linked to the corresponding asset components. The PdM module operated as a decision-support system, and no autonomous control actions were triggered. This interface design explicitly bounds the experimental scope of the PdM integration and ensures that only actionable, interpretable information is presented to the user within the XR workflow (Table 1).

Table 1. PdM–XR interface specification used in the experimental evaluation.

Module	Variable/Message	Data Format	Update Frequency	Target Latency	Purpose in XR
PdM module	Operational state	JSON	~1 Hz	<200 ms	Contextual asset status visualization
	Health indicator trend	JSON	~0.1 Hz	<500 ms	Trend monitoring during maintenance
	Remaining Useful Life (RUL)	JSON	Event-based/~0.05 Hz	<500 ms	Prognostic awareness
	Anomaly/threshold flag	JSON	Event-driven	<200 ms	Alert triggering
XR application	Visualization request	JSON	On demand	<200 ms	Digital twin update
	User acknowledgment	JSON	Event-driven	<200 ms	Alert confirmation/workflow continuity

3.5. LLM Recommendation Engine

In addition to the XR-enabled communication and content-authoring modules, the proposed architecture incorporates an LLM Recommendation Engine, which functions as an intelligent, domain-aware conversational assistant for maintenance operations. The primary function of this module is to support shop floor technicians in scenarios where expert personnel may not be immediately available, where equipment documentation is incomplete, and where faults must be diagnosed and resolved in short amount of time under operational constraints. Through speech or text input, technicians can describe symptoms, unusual equipment behavior, alarm codes, or operational anomalies in natural language. The LLM interprets these descriptions and generates clear, contextually relevant, and actionable guidance. This includes fault interpretations, root-cause indications, procedural steps for troubleshooting or component replacement, safety instructions, and recommendations for mitigating equipment degradation until full repairs can be performed. In practice, the model serves as a cognitively aware knowledge medium capable of narrowing the gap between available expertise and real-time on-site decision needs. Therefore, the proposed AI assistant is designed as a decision-support component, rather than an autonomous decision-making agent. Its role is limited to contextual information retrieval, procedural guidance, and clarification of maintenance steps based on predefined documentation, Asset Administration Shell data, and structured knowledge sources. The system does not execute control actions, override human decisions, or directly interact with machinery, ensuring that all safety-critical decisions remain under human responsibility.

To enable such behavior, a comprehensive training data set was constructed from more than one hundred maintenance manuals, service instructions, troubleshooting guides, and historical MRO reports originating from various machine-tool and industrial equipment categories. The documents were preprocessed through a multi-stage pipeline designed to maximize the semantic clarity and procedural fidelity of the resulting corpus. Text extraction was applied to digital manuals and OCR was used for scanned documents. Non-textual artifacts such as figures, complex tables, and page headers were removed to avoid training noise. Sentence-level tokenization preserved the procedural sequence of each maintenance operation, while terminology normalization ensured consistency for component names, tool identifiers, and safety terminology across different manufacturers. The resulting dataset consisted of hundreds of thousands of high-quality, structure-preserving maintenance sentences. These were transformed into instruction-style prompt–response pairs, capturing both the task description and the expected procedural outcome in a manner appropriate for supervised fine-tuning.

To mitigate the risk of erroneous or hallucinated responses, the AI assistant operates under constrained prompting and retrieval mechanisms, prioritizing verified domain knowledge over free-form generation. Responses are grounded in asset-specific data, maintenance manuals, and predefined procedural templates, while uncertainty-aware prompts are used to encourage clarification rather than speculation. When confidence thresholds are not met, the assistant explicitly defers to the human expert, reinforcing a human-in-the-loop interaction paradigm.

For the model foundation, a compact transformer-based architecture was selected, balancing computational feasibility with strong instruction-following capabilities. The chosen base model contains approximately 2.7 billion parameters in its pretrained configuration. To adapt it to the maintenance domain without imposing excessive computational overhead or risking degradation of general linguistic abilities, a parameter-efficient fine-tuning strategy was adopted in which only a small subset of approximately 21 million trainable parameters was updated. This approach preserves the foundational model’s linguistic strengths while enabling it to internalize technical maintenance knowledge with high precision. Fine-tuning was performed on a dedicated workstation with specifications listed in the Implementation Section. The training process was completed within roughly three hours, demonstrating the practicality of local, on-premises fine-tuning without reliance on cloud-based infrastructure.

The training configuration was developed empirically to ensure stable convergence and strong generalization performance. A learning rate on the order of 2 × 10⁻⁴, batch accumulation to compensate for moderate batch sizes, a warm-up phase to avoid early-stage instability, and a series of roughly one thousand optimization steps were used. Mixed-precision training was applied when supported by the hardware to maximize throughput. The final dataset was split into training and evaluation partitions, and the loss curves exhibited a smooth, monotonic reduction in both sets, indicating effective learning without overfitting. Evaluation of model behavior was performed qualitatively by comparing the outputs of the fine-tuned model against those of the pretrained base model and other widely used general-purpose LLMs. The pretrained model frequently struggled to interpret domain-specific terminology and produced incomplete or incoherent maintenance steps, while general-purpose models tended to return generic, non-equipment-specific recommendations. In contrast, the fine-tuned model consistently produced detailed, accurate, and procedurally aligned responses closely matching the authoritative content of the manuals, validating both the dataset quality and the effectiveness of the fine-tuning procedure.

Once trained, the LLM was integrated into the proposed architecture as a backend service interfacing with both the XR subsystem and the engineering desktop application. Technician queries are serialized and transmitted to the LLM, which returns structured outputs designed to be directly interpretable within the mixed-reality environment. The output may include disassembly steps, required tools, torque values, safety interventions, or instructions to temporarily reduce operational load under emergent conditions. These outputs may trigger changes in the XR environment such as highlighting a specific component, isolating a segment of the 3D assembly, or loading a relevant holographic representation. This seamless flow between natural-language reasoning and spatial visualization allows technicians to act on LLM-generated recommendations immediately within the physical workspace.

To ensure operational safety and alignment with industrial standards, the system incorporates a human-in-the-loop mechanism for validating critical LLM outputs. Instructions involving potentially hazardous actions, such as interventions on energized systems, pressurized elements, or thermally stressed components, require explicit approval from a qualified engineer before being presented as operational guidance in the XR interface. Non-critical informational content, such as component identification or general troubleshooting advice, may be displayed automatically. This layered validation maintains the benefits of rapid AI-driven reasoning while ensuring that decisions with safety implications remain under human control.

By acting as a cognitive augmentation layer within the Industrial Metaverse, the LLM strengthens the availability, accessibility, and comprehensibility of maintenance knowledge. It democratizes expertise by making correctly calibrated procedural information accessible to less experienced technicians, supports consistent execution of complex tasks across shifts and facilities, and enhances resilience in environments where documentation may be insufficient or expert availability limited. In combination with predictive analytics, XR-enabled spatial guidance, and real-time remote collaboration, the LLM transforms maintenance activities from documentation-centric routines into dynamic, context-aware processes in which workers are supported at every stage by a continuously available, domain-specialized intelligent assistant.

The language model was adapted to the industrial maintenance domain using a supervised fine-tuning approach on a curated corpus derived from machine-tool maintenance manuals, troubleshooting guides, and procedural documentation. Due to confidentiality constraints associated with proprietary industrial data, raw documents cannot be publicly released; however, the fine-tuning pipeline and evaluation logic are described in detail to ensure methodological transparency.

Prior to fine-tuning, source documents were manually filtered to remove non-procedural content, normalized to a consistent instructional format, and segmented into instruction–response pairs. Instruction prompts were formulated to explicitly request step-by-step maintenance guidance, safety considerations, and diagnostic reasoning. The model context length was configured to support multi-step procedural outputs, and responses exceeding this limit were truncated during training.

Fine-tuning was performed using standard transformer-based optimization, with a fixed learning rate, batch size, and number of epochs selected based on convergence behavior observed on a held-out validation set. The objective was not to maximize benchmark scores, but to improve task relevance, procedural structure, and contextual grounding in maintenance scenarios.

3.6. Data Pipeline, Model Scope, and Evaluation Protocols

The proposed framework integrates multiple AI-enabled components, each operating at a different level of the maintenance workflow. Thus, in order to ensure methodological clarity and reproducibility, the scope and evaluation of each component are discussed in the following paragraphs.

The predictive maintenance module, based on LSTM-Autoencoders and Transformer Encoders, was developed and validated as an independent research contribution prior to the present work. Its data pipeline, training and validation procedures, evaluation protocols, and performance metrics are documented in detail in a dedicated peer-reviewed publication [6]. In the context of this manuscript, the predictive maintenance module is treated as an upstream information provider that supplies condition indicators and alerts to the proposed XR-based maintenance framework, without re-training or re-evaluation within the reported experiments.

The LLM component was fine-tuned to support maintenance task generation and procedural guidance. A custom dataset comprising more than 100 machine-tool repair manuals was curated, digitized, and preprocessed using standard text extraction, cleaning, and tokenization procedures. The dataset was split into training (75%) and testing (25%) subsets, corresponding to approximately 187,000 and 67,000 sentences, respectively. Fine-tuning was performed on a pretrained phi-2 model using supervised instruction-style learning, following best practices for domain adaptation.

Model evaluation focused on qualitative and task-oriented criteria, including correctness of suggested actions, level of technical specificity, and contextual relevance to the target equipment. Comparative evaluation was performed against a general-purpose GPT model and the original pretrained phi-2 model, highlighting the effect of domain adaptation on maintenance-specific responses. Training and evaluation loss curves are reported to monitor convergence and overfitting behavior. A detailed description of the LLM fine-tuning workflow and implementation is available in a complementary publication by the authors [44].

At the system level, evaluation metrics focused on task execution time, error occurrence, and user-perceived usability during maintenance activities. These metrics were operationally defined as the elapsed time required to complete predefined maintenance tasks, the number of human-induced procedural errors, and structured post-task user feedback, respectively. Quantitative improvements were assessed through baseline-versus-supported comparisons, while qualitative observations were used to contextualize system behavior in real industrial settings.

4. System Implementation

The software and hardware used for the development, implementation, and validation of the proposed framework are discussed below. The framework is developed using Unity 2022.3.10f1 (Unity Technologies, San Fransisco, CA, USA) [51], integrating the WebRTC framework (Google LLC, Mountain View, CA, USA) [52] which supports real-time communication, (audio and video) in MR environments. In addition to that, DataChannels from WebRTC have been implemented, in order to enable data transmission besides audio/video (i.e., 3D holograms). For the XR functionalities (i.e., creation and manipulation of holograms), Microsoft’s MRTK v3.2.2 (Microsoft Corporation, Redmond, WA, USA) was used [53]. All the code scripts for the Unity 3D project have been developed in C# v13 using Microsoft’s Visual Studio 2022 IDE (Microsoft Corporation, Redmond, WA, USA) [54]. Additional scripts for the setup of the server and handling the requests have been developed using Python 3.12 64-bit version (Python Software Foundation, Wilmington, DE, USA). In terms of hardware, a desktop PC has been utilized equipped with an AMD Ryzen 5900X 12-core processor (Advanced Micro Devices, Santa Clara, CA, USA), 64 GB RAM, and NVIDIA GeForce RTX 4090 24 GB GPU (NVIDIA Corporation, Santa Clara, CA, USA) has been utilized. Finally, the selected HMD is Microsoft’s HoloLens 2 (Microsoft Corporation, Redmond, WA, USA).

5. Industrial Case Studies and Results

5.1. Research Design and Internal Validity Considerations

The applicability of the proposed framework was evaluated through two industrial case studies conducted in real operational environments. Each case study involved approximately 20 participants (both male and female), including engineers, technicians, and maintenance specialists actively engaged in industrial maintenance activities. Participant inclusion was performed in collaboration with the industrial partners and was based on professional role relevance and prior familiarity with the corresponding equipment or maintenance procedures, ensuring ecological validity.

In an attempt to characterize participant heterogeneity, individuals were grouped according to their corresponding experience level (i.e., junior: <3 years, intermediate: 3–7 years, senior: >7 years). Therefore, comparable distributions across experience groups were maintained within each case study. The evaluation followed a quasi-experimental, within-subject design. Each participant executed the defined maintenance tasks under two conditions:

• A baseline condition using conventional procedures and documentation;
• A supported condition using the proposed XR–digital twin–AI framework.

This within-subject comparison structure was selected in order to reduce inter-participant variability as well as to enable paired comparisons of task performance and perceived usability. The baseline condition was always executed first, followed by the supported condition. Full counterbalancing was not applied due to operational constraints and the risk of knowledge transfer from XR-supported execution to baseline task performance. For the mitigation of potential order and learning effects, participants were already familiar with the maintenance procedures, task definitions were standardized, and baseline and supported executions were temporally separated.

A series of measures were adopted to mitigate learning, novelty, and observer effects. Objective task-level performance indicators (e.g., task execution time and procedural error occurrence) were defined a priori and applied consistently across conditions. Observational assessment was complemented by structured post-task questionnaires rather than open-ended evaluation. Furthermore, remote expert intervention followed predefined guidance protocols for ensuring comparable levels of assistance across participants. An overview of the experimental design is provided in Table 2.

Table 2. Overview of experimental design and bias control measures per case study.

Aspect	Case Study 1	Case Study 2
Number of participants	17	25
Participant roles	Engineers, technicians, maintenance specialists
Experience grouping	Junior (<3 y), Intermediate (3–7 y), Senior (>7 y)
Inclusion criteria	Role relevance and prior familiarity with equipment
Experimental design	Quasi-experimental
Comparison structure	Within-subject
Conditions	Baseline → XR-supported
Counterbalancing	Not applied (operational constraints, knowledge transfer risk)
Learning effect mitigation	Familiar tasks, standardized procedures, temporal separation
Novelty effect mitigation	Task-focused metrics, prior briefing
Observer bias control	Predefined metrics, structured questionnaires
Remote expert intervention	Standardized guidance protocol

The analysis presented in this study is primarily descriptive and exploratory, reflecting the quasi-experimental nature of the evaluation and the constraints of real industrial environments. Reported improvements are summarized using central tendency measures and uncertainty bounds rather than formal hypothesis testing. Wherever confidence intervals are reported, they represent descriptive uncertainty estimates computed using a non-parametric bootstrap procedure. For within-subject comparisons, only complete paired observations were included, resulting in N effective pairs per case study. No imputation was applied for missing data; incomplete task executions were excluded from paired analyses. Bootstrap confidence intervals were computed using 10,000 resamples of the paired differences and the percentile method to estimate 95% confidence bounds around the mean improvement. These intervals are intended to provide an indication of result stability rather than inferential significance. Given this scope, results are interpreted cautiously, and observed improvements are discussed as task-level associations rather than causal effects attributable solely to the proposed system.

5.2. Operational Definitions and Coding of Evaluation Metrics

To ensure methodological transparency and reproducibility, all task-level evaluation metrics were defined using explicit operational criteria. A procedural deviation was recorded when a participant deviated from the predefined maintenance task sequence, omitted a required step, or executed a step in an incorrect order, as specified by the task checklist derived from standard operating procedures.

A rework instance was recorded when an incorrectly executed step required correction, repetition, or reversal before task continuation, regardless of whether the deviation was immediately detected or identified at a later stage. Rework instances were counted per task execution and treated as discrete events.

A measurement error was defined as an incorrect measurement action or value acquisition (e.g., wrong sensor selection, incorrect tool usage, or misinterpretation of displayed values) that could affect diagnostic or maintenance decisions. Measurement errors were recorded only when they were verifiable against task specifications or digital twin ground truth data.

All task executions were evaluated using a structured task-step checklist derived from the corresponding maintenance procedures. Metric recording was performed through a combination of human observation and system-level logging, ensuring cross-validation of recorded events. Ambiguous cases were resolved through consensus between two evaluators following predefined decision rules. An overview of operational definitions and coding scheme is shown in Table 3.

Table 3. Operational definitions and coding scheme for task-level evaluation metrics.

Metric	Operational Definition	Data Source	Validation/Decision Rule
Procedural deviation	Omission, incorrect order, or incorrect execution of a predefined task step	Human observer + checklist	Compared against task-step checklist derived from standard operating procedures
Rework instance	Correction, repetition, or reversal of a previously executed step	Human observer	Counted when corrective action is required before continuation
Measurement error	Incorrect measurement action or value inconsistent with task specification	System logs + observer	Verified against digital twin data or procedure limits
Task execution time	Elapsed time from task start to completion	System logging	Automatically logged timestamps
Error occurrence	Presence of ≥1 deviation, rework, or measurement error	Aggregated	Binary per task execution

5.3. Steel Wire Manufacturer

The first use case involves the removal, disassembly and repair of an industrial gearbox used in steel wire rollers. This gearbox is a complex mechanism, responsible for converting the angle of rotational motion and is capable of driving hot drawn steel wire in order to reduce its diameter even more, following the manufacturer’s specifications. The equipment is depicted in Figure 8. Due to the complex and high frequency oscillations during its operation, the internal bearings suffer from rapid wear. Therefore, the manufacturer has to frequently uninstall them from the production line, disassemble them completely, remove the faulty bearings, renew them, as well as inspect the pinion and crown gear for out spec components. The process of the disassembly is heavily manual and dependent on the technician’s experience due to the fact that this piece of equipment is fairly old. Beyond that, certain details regarding the disassembly steps have not been documented, and in addition the gearbox design has been updated several times (without proper documentation), thus there is no adequate maintenance protocol for the repair of this equipment.

Figure 8. Wire steel production case study, testing the application on field.

Considering the above-mentioned, it becomes evident that new, inexperienced technicians are not capable of performing such maintenance activities. The addition of the proposed tool facilitates documenting the MRO in greater detail as well as in digital format. With the utilization of the video call functionality, expert engineers are capable of communicating their tacit knowledge to the technicians, in real-time.

The bevel gearbox is one of twenty similar units integrated in the SYNCHRO ASHLOW rolling line; however, only five are equipped with condition monitoring and XR-assisted maintenance capabilities. Each monitored gearbox carries four ASH426 industrial vibration and velocity sensors, capturing high-precision measurements of acceleration, misalignment, imbalance, and bearing deformation. These signals are processed by the Acoem MV-x, an AI-enabled edge computing platform that performs real-time vibration analytics, fault detection, and RUL estimation. According to the experimental specifications, data was gathered at 30 samples/day over a monitoring period of six months, producing approximately 10 GB of machine health data categorized across three condition states: Healthy, Intermediate, and Bad.

These predictive insights are then fed into the AR-enabled workflow. Before each maintenance cycle, technicians use the AR headset to inspect the gearbox while viewing live overlays of sensor values, anomaly notifications, and real-time RUL estimates directly on the physical asset. The AR interface projects time-series vibration graphs, threshold indicators, and status flags (e.g., Status: Anomaly or Status: Operational), enabling technicians to determine whether disassembly is warranted. This closes the loop between PdM and hands-on intervention, allowing the system to transition from predictive diagnosis to guided repair.

Once a maintenance action is confirmed, the disassembly and renewal process is facilitated through the XR maintenance guidance module (Figure 9). A high-fidelity 3D CAD model of the bevel gearbox, reconstructed from technical schematics, is displayed within the technician’s field of view, including drive shafts, coupling components, lubrication pathways, electrical conduits, and the full crown–pinion assembly. Trainees can interact with an exploded 3D representation of sub-assemblies, which clearly illustrates bearings, flanges, bolts, seals, and mechanical tolerances. This exploded view is particularly important given the incomplete documentation and the undocumented redesign iterations mentioned earlier; the XR module therefore acts as the only source of up-to-date procedural structure.

Figure 9. Wire steel production case study; view from the HoloLens 2 HMD during testing.

During intervention, the real-time remote video call becomes a central component of the workflow. Technicians initiate a call via the HoloLens interface, transmitting their exact first-person view (including all holograms and sensor overlays) to a remote expert engineer. On the desktop counterpart, the expert can annotate the live video feed, place 3D arrows, draw freehand indicators, or overlay 3D objects sourced from the FTP server (e.g., a specific bearing model or tool geometry). These annotations instantly appear within the technician’s XR environment, providing precise spatial references for disassembly, inspection, or alignment. The simultaneous streaming of the HMD’s environmental point cloud enables the expert to navigate the 3D reconstruction of the workspace, ensuring that all holograms are accurately registered to physical surfaces.

This Metaverse-based, real-time collaborative environment proved critical during the validation of this use case, where maintenance steps such as bearing extraction, shaft alignment, and crown–pinion inspection required highly specific expertise. According to the tracked user feedback metrics, technicians rated the XR system positively regarding safety, ease of following procedures, and reduction in troubleshooting errors, while expert engineers have reported measurable decreases in downtime and improved clarity in maintenance reporting.

5.4. Copper Tubing Manufacturer

The second industrial use case focuses on the disassembly, measurement, and repair of a gear-roller subassembly used in cold extraction (pilgering) processes for copper tubing production. Cold pilgering is a high-precision metal forming process in which the diameter and wall thickness of copper tubes are progressively reduced through continuous reciprocating deformation. The core pilgering machine comprises two forming dies mounted on a reciprocating saddle, a stationary tapered mandrel, a mandrel bar secured in a thrust block, and a complex mechanical drivetrain responsible for synchronizing the reciprocating and rotational motion of the dies. During operation, the rollers and bearings are subjected to severe and repetitive mechanical loading, leading to unpredictable degradation patterns. Maintenance engineers reported significant variability in bearing lifetime (1–3 months) and recurring unplanned failures, with replacement expenses reaching up to €20,000 per workstation and total interruption costs scaling up to three times the replacement cost due to production delays.

The mechanical structure of the pilger mill poses additional challenges for maintenance (Figure 10). Mainly, due to its age and design, the machine lacks built-in sensor instrumentation, while its continuous reciprocating motion renders the installation of wired sensing systems impractical, and dangerous for the seamless operation of the manufacturing processes. Consequently, the PdM configuration adopted a fully wireless sensing architecture, deploying AIRIUS industrial-grade sensors (i.e., two per bearing, eight in total) for the collection of high-frequency vibration, velocity, and temperature data under harsh operating conditions. The acquired signals were routed through a LabJack T7 industrial DAQ (Data Acquisition) module, utilizing its 14 instrumentation-amplified analog inputs, programmable gain settings, and Modbus/TCP support to ensure high-fidelity condition monitoring. The data was streamed to a dedicated IoT platform, stored on the manufacturer’s secure on-premises server infrastructure, and processed through machine-learning fault-detection pipelines to predict anomalies and estimate bearing RUL in real-time. This configuration enabled a shift from reactive maintenance to PdM, reducing failure probability and improving resource allocation for high-wear and failure-prone components.

Figure 10. Copper industry case study; testing with maintenance technicians.

In order to support the maintenance personnel, the proposed framework was applied to the pilger roller subassembly, whose disassembly procedure includes removing the roller shaft, extracting multiple bearing pairs, inspecting the external gear, and verifying the alignment between component pairs. A high-fidelity 3D CAD model of the pilger assembly was created based on 2D mechanical schematics provided by the manufacturer, capturing all critical components such as the reciprocating saddle, roller dies, gears, and frame structure. This 3D model was integrated into the MR maintenance (XR engine discussed in previous sections) guidance module, enabling technicians to visualize the machine’s internal structure, explore exploded views of subassemblies, and follow step-based interactive instructions during repair procedures (Figure 11).

Figure 11. High-fidelity exploded view of bearing sub-assembly.

The holographic model was further linked to real-time PdM data, allowing technicians to overlay vibration or temperature indicators directly onto the machine during inspection tasks.

A structured pilot-testing protocol was established at the copper tubing manufacturer’s maintenance department to evaluate the applicability and ergonomics of the system. Eight trainees, supported by two instructors, used Microsoft HoloLens 2 devices to perform the roller-shaft removal procedure with and without AR assistance (Figure 12). Each participant first executed the task conventionally, followed by a familiarization session with the AR interface, and then repeated the procedure with MR-enabled guidance. Time-to-completion, error rates, and process deviations were recorded via simultaneous point-of-view video capture on a separate PC functioning as a monitoring unit. The AR-assisted workflow contributed to improved task comprehension, safer execution of disassembly steps, and reduced cognitive load for inexperienced technicians. The immersive holographic overlays and manipulation capabilities (zooming, rotating, isolating subcomponents) significantly improved perception/understanding of the rollers’ kinematic relationships and supported accurate component sequencing during reassembly.

Figure 12. Copper industry case study; caption from HoloLens 2 HMD during testing.

5.5. Validation Results

The applicability and performance of the proposed framework were evaluated through the two industrial case studies discussed above. For each case study a structured maintenance procedure protocol has been set up in collaboration with the expert engineers, and were executed by technicians with varying experience levels, both with and without the proposed system. The key objective of the validation activities was to quantify the impact of the system on task accuracy, procedural completeness, execution time, and user experience. Performance was assessed using objective observations (procedural errors, deviations, rework instances) and subjective evaluations (perceived clarity, cognitive load, satisfaction). For extracting the subjective evaluations questionnaires were handed out to the participants during the validation execution.

The experimental evaluation was conducted as an exploratory case study involving a limited number of participants and repetitions, reflecting realistic industrial constraints. Each use case was executed with domain experts and technicians directly involved in the corresponding maintenance activities, and all trials were performed in operational industrial environments. As a result, the collected data are intended to provide qualitative and indicative quantitative insights into system performance and usability rather than statistically generalized conclusions.

In addition to descriptive statistics, we conducted an exploratory inferential analysis to strengthen interpretation of observed improvements. For each quantitative metric, we compared the baseline and supported conditions using paired analyses where the same participants performed both conditions. Mean differences (supported − baseline) are reported together with 95% confidence intervals (CI) estimated via nonparametric bootstrap resampling (10,000 resamples), which provides robust uncertainty estimates for moderate sample sizes. To quantify the magnitude of changes, we additionally report effect sizes for paired designs (Cohen’s d_z, and where appropriate Hedges-corrected estimates). Given the exploratory intent and the practical constraints of industrial evaluations, these inferential results are interpreted as supporting evidence of trends rather than definitive statistical generalization.

Quantitative results are reported as mean ± standard deviation for each condition, together with the mean difference (Δ), the corresponding 95% CI, and effect size. This report supports distinctions between consistent performance improvements and isolated variations.

Across both use cases, the results demonstrate clear improvement in execution quality when technicians used the MR-guided instructions and remote-support capabilities. In the steel-wire gearbox scenario, technicians achieved a higher level of procedural completeness when using the system (Table 4). Specifically, the baseline execution resulted in multiple missing or incorrect steps, with several technicians failing to identify or document key disassembly actions. When supported by MR, procedure adherence increased substantially, with all critical steps successfully completed and a significant reduction in deviations. Observations also indicate improved accuracy during reassembly, especially for steps involving correct bearing placement and alignment of pinion–crown gear components, which were previously prone to human error.

Table 4. Procedural accuracy—steel wire gearbox case study.

Metric	Baseline (No XR)	Proposed System (MR + Remote Support)	Improvement
Procedural steps completed correctly (out of 18)	12.4 ± 2.1	17.6 ± 0.5	+42%
Number of procedural deviations	5.2 ± 1.3	0.8 ± 0.4	−84%
Rework instances required	3.1 ± 1.0	0.4 ± 0.5	−87%
Incorrect bearing seating events	2.0 ± 0.9	0.0	Eliminated
Missing documentation entries	4.5 ± 2.0	0.7 ± 0.6	−84%

Similar improvements were observed in the copper-tubing gear-roller scenario, where disassembly and dimensional inspection tasks were performed by inexperienced operators. The baseline performance was characterized by partial or incomplete adherence to the measurement workflow, particularly in referencing correct datums and validating internal wear. When using the MR-guided workflow, the number of omitted steps decreased dramatically, and technicians executed the required measurements in a more consistent order, as indicated by the procedure-tracking logs. According to Table 5, the technicians committed significantly fewer procedural deviations when using the system, reflecting a more structured understanding of the task sequence and improved confidence during complex interactions such as the extraction of the axle as well as bearing seating verification.

Table 5. Procedural accuracy—copper tubing gear-roller use case.

Metric	Baseline (No MR)	MR-Supported	Improvement
Steps completed correctly (out of 22)	15.1 ± 2.6	21.2 ± 0.7	+40%
Measurement errors (datum misalignment, incomplete readings)	3.8 ± 1.2	0.6 ± 0.5	−84%
Rework instances	2.4 ± 0.8	0.3 ± 0.5	−88%
Incorrect tool selection events	1.6 ± 0.9	0.1 ± 0.3	−94%

The execution time exhibited a more nuanced pattern. Concretely, for both case studies, the MR-enabled procedures required slightly longer completion times for inexperienced users, primarily due to initial familiarization with holographic overlays and interface interactions (Table 6 and Table 7). However, experienced users maintained comparable execution times, and both groups demonstrated a substantial decrease in rework effort. Based on discussions with both the engineers and the shop floor technicians on the filed, the time overhead introduced by MR guidance is offset by the near elimination of procedural errors, resulting in a more reliable and repeatable workflow, which in turn reduces overall downtime during real operations.

Table 6. Execution time—steel wire gearbox case study.

User Group	Baseline (min)	MR-Supported (min)	Difference
Experienced technicians	41.3 ± 4.7	43.0 ± 5.0	+4%
Inexperienced technicians	58.1 ± 7.2	64.5 ± 6.4	+11%

Table 7. Execution time—copper tubing gear-roller case study.

User Group	Baseline (min)	MR-Supported (min)	Difference
Experienced technicians	52.0 ± 5.5	54.4 ± 5.7	+5%
Inexperienced technicians	69.5 ± 8.1	75.8 ± 7.2	+9%

Subjective evaluations further corroborate the effectiveness of the system. According to the user-survey results gathered during the validation of the developed solution, technicians reported higher confidence in understanding task requirements, greater clarity of the visualized steps, and improved situational awareness. It is notable that the majority of participating technicians rated the MR instructions as “clear” or “very clear,” while the remote support functionality was consistently highlighted as essential for bridging knowledge gaps. Furthermore, the participants also noted reduced perceived cognitive load when executing unfamiliar procedures, attributing this to the combined effect of spatial guidance, real-time visual feedback, and the ability to access expert input (via the video call functionality) without interrupting the workflow (Table 8).

Table 8. User experience evaluation (both case studies).

Evaluation Criterion	Mean Baseline Score (1–5)	Mean MR-Supported Score (1–5)	Improvement
Clarity of instructions	2.9 ± 0.7	4.6 ± 0.5	+59%
Perceived cognitive load	3.8 ± 0.9	2.4 ± 0.8	−37%
Confidence in task execution	3.1 ± 0.8	4.7 ± 0.4	+52%
Perceived safety	3.3 ± 0.6	4.5 ± 0.5	+36%
Overall satisfaction	3.0 ± 0.8	4.8 ± 0.4	+60%

Although a full benchmark-style quantitative evaluation of LLM performance is outside the scope of this integration-oriented study, the effectiveness of the fine-tuned model was assessed through controlled comparative task-level evaluation and training convergence analysis.

Specifically, the same maintenance prompt was provided to three models: a general-purpose GPT model, the original pre-trained phi-2 model, and the fine-tuned phi-2 model. As illustrated in Table 9, the fine-tuned model produced a higher number of procedurally complete, equipment-specific, and safety-aware steps, whereas the generic GPT model provided high-level but underspecified guidance, and the pre-trained phi-2 failed to generate a meaningful response. This comparative setup enables a direct qualitative assessment under identical task conditions.

Table 9. Prompt comparison between ChatGPT versus the fine-tuned model.

Model	Response
Gpt v4.0	Access the Computer Module: Locate and open the compartment housing the computer module. Disconnect Cables: Disconnect all cables from the computer module, noting their positions. … Secure the Module: Insert and tighten the screws to secure the module. Reconnect Cables: Reconnect all cables to the new computer module.
Pre-trained phi v2.0	I’ m sorry, but it seems like you may be referring to a different product than the one I am currently assisting with. Can you please provide more information about the specific computer module or LCD/ enclosure replacement that you are looking for?
Fine-tuned phi v2.0	Turn off the power to the machine and control. Unplug all the connectors from the rear of the pendant. Remove the pendant from the pendant arm by removing the bolt and nut that secures it in place. … Secure the computer module back to the LCD/enclosure by following the instructions in reverse order. Make sure that the overlay cable is properly seated before fastening the unit back in place.

In addition, quantitative indicators of model behavior were obtained during training and evaluation. The fine-tuning process employed a 75/25 training–testing split over approximately 254,000 sentences, and the resulting training and evaluation loss curves demonstrate stable convergence and limited generalization gap (Figure 13). These results indicate that the model adaptation improved task relevance without overfitting.

Figure 13. Training and evaluation loss of the model.

Within the context of industrial maintenance support, where correctness, procedural completeness, and contextual specificity are more critical than linguistic similarity scores, the reported evaluation provides evidence-based justification for the improved performance of the fine-tuned model compared to generic or non-adapted alternatives.

5.6. Discussion

The quantitative results from both case studies provide adequate evidence for confirming the trends highlighted in the literature on MR-based maintenance and remote expert support. Prior studies have reported that MR improves procedural adherence and reduces cognitive load by presenting instructions directly in spatial context, allowing technicians to anchor each maintenance step to the corresponding physical component. Past research works, such as in [39] consistently show error reductions between 30 and 50% when AR/MR instructions guide complex technical tasks. The quantitative validation of the current research indicates reductions of 84–94% in procedural deviations and rework, align with and extend these findings by demonstrating similar improvements in two distinct industrial environments.

The inclusion of confidence intervals and effect sizes provides an additional basis for interpreting observed improvements beyond point estimates. In particular, effects that remain directionally consistent with confidence intervals that exclude near-zero differences can be considered more consolidated within the evaluated cohort, whereas wider intervals indicate greater uncertainty and motivate cautious interpretation. This strengthens the distinction between promising trends and more stable evidence within the scope of the present case studies.

The observed increase in step completeness echoes earlier results from MR-assisted disassembly tasks, where inexperienced operators benefit most from holographic overlays and real-time expert support. However, the present work goes further by integrating an LLM-driven assistance module capable of interpreting technical queries and generating context-aware instructions. This hybridization of semantic AI with spatial MR guidance is absent in the previous studies investigated, and provides a strong explanation for the high task-completion scores and the elimination of critical errors such as incorrect bearing seating or misaligned inspections, since the technicians had an additional tool for requesting more detailed instructions.

On the contrary, the execution time indicates a slight increase for inexperienced operators (which has also been well-documented phenomenon in MR literature) attributed to initial cognitive shifts required to adapt to holographic interactions. Yet, as also reported in related industrial studies, this overhead is offset by the dramatic reduction in rework and troubleshooting time. The results from both pilots demonstrate that the system not only improves correctness but also stabilizes workflow variability across technicians, a key challenge in maintenance environments characterized by skill shortages and undocumented machine configurations. However, in order to provide more accurate metrics on the execution time, more testing/using of the tool is recommended, so that the personnel become even more familiar with the tool itself as well as with the technology (i.e., using and interacting with the HMD).

As a final metric, user feedback has been gathered by handing structured questionnaires to the personnel, after testing the tool. User feedback further reinforces the system’s effectiveness. Reported improvements in clarity, confidence, and perceived safety align well with findings from the MR-enabled training and support systems. However, the exceptionally high satisfaction scores observed from the responses indicate that the combination of XR, real-time remote expert annotations, and LLM-generated explanations provides a more comprehensive support environment than traditional AR-only solutions.

Conclusively, the results demonstrate that the proposed framework not only replicates but substantially extends the benefits reported in state-of-the-art XR maintenance studies. By combining spatial computing, PdM metrics, and natural-language assistance within a Metaverse-aligned architecture, this work advances the field toward a more integrated, human-centric, AI-augmented maintenance paradigm envisioned in modern Industry. The interpretations presented above are grounded in the observed outcomes of the conducted case studies and should be understood within their empirical scope. The reported reductions in task duration, improvements in coordination, and positive user feedback constitute direct observations derived from the evaluated scenarios. Inferences regarding broader impacts, such as long-term efficiency gains or scalability across diverse industrial contexts, are reasonable extrapolations based on these observations but are not claimed as empirically validated results within the present study. This distinction is maintained to ensure that conclusions remain aligned with the exploratory nature of the evaluation.

6. Ethical and Privacy Considerations

The proposed framework involves capturing and processing video, audio, and operational data during MRO conducted in real industrial environments. Ethical considerations and data protection were therefore treated as integral aspects throughout the implementation, testing, and validation phases.

All experimental activities were performed within the premises of the participating industrial partners, following their internal data protection policies and operational guidelines. Participation was voluntary, and all involved personnel (i.e., engineers, technicians, and maintenance specialists) were informed in advance about the nature of the data being collected. Informed consent was obtained prior to participation, and participants retained the right to withdraw at any stage of the evaluation.

In order to mitigate privacy risks (if any), no personal identifiable information was stored or processed by the system. Video and audio streams were used exclusively for real-time collaboration and were not retained beyond the experimental sessions unless explicitly required for technical validation and authorized by the industrial partner. Operational data accessed through the framework was limited to task-relevant information and were handled in accordance with existing industrial confidentiality agreements.

7. Concluding Remarks and Outlook

In this manuscript, the design and development of a MR framework for remote industrial maintenance support are presented. The core contribution of the proposed approach lies in the end-to-end integration of immersive MR interaction, real-time remote collaboration, equipment monitoring pipelines, and a domain-adapted LLM-based cognitive assistant into a unified maintenance support ecosystem. This integration enables expert guidance, contextual holographic visualization, and data-informed decision support to be delivered directly within the technician’s field of view, addressing practical needs in real industrial environments.

The experimental evaluation demonstrates measurable improvements at the task-execution level, including reductions in procedural deviations, rework instances, and task completion time, as well as positive user feedback regarding usability, situational awareness, and communication effectiveness. These outcomes are directly observed and empirically supported within the scope of the two industrial case studies and confirm the feasibility and practical value of integrating XR collaboration, digital twins, and AI-assisted support within maintenance workflows.

It is important to maintain a clear distinction between these observed results and broader operational implications. Impact-related outcomes such as reduced downtime or improvements in Overall Equipment Effectiveness (OEE) are not directly measured as system-level indicators in the present study. Rather, they are presented as reasonable implications derived from the observed task-level improvements, since reduced rework, fewer procedural errors, and faster response times are well-established contributors to downtime in industrial maintenance practice. Accordingly, references to downtime reduction and OEE improvement should be interpreted as indicative and context-dependent, rather than as directly quantified outcomes.

Several limitations should be acknowledged. First, the experimental validation was limited to two industrial case studies involving approximately twenty participants each. While sufficient for exploratory in situ evaluations, this restricts statistical generalization across different industrial sectors, organizational structures, and maintenance strategies. In addition, the study focused on short-term task execution and did not include longitudinal observations that would be required to assess long-term learning effects, skill transfer, adoption dynamics, or sustained operational impact.

Hardware- and infrastructure-related constraints were also identified. Head-mounted display ergonomics remain a practical limitation in shop floor environments, as device weight and field-of-view considerations may affect prolonged usage and mobility. Although alternative industrial-grade devices (e.g., helmet-mounted XR solutions) showed improved acceptance, careful integration into safety-critical environments remains necessary. Furthermore, the framework relies on adequate computational infrastructure to support predictive algorithms and LLM-based assistance, which may pose challenges for organizations with strict data locality requirements or limited on-premises resources.

From a software perspective, the current implementation is constrained by the underlying real-time communication stack and the computational capabilities of available XR hardware. Network reliability and latency directly influence the quality of real-time video streaming, hologram alignment, and AI-assisted interactions, particularly in bandwidth-constrained industrial environments. These aspects may affect scalability and user experience and should be carefully considered in future deployments.

From an Industrial Metaverse perspective, the proposed framework should be interpreted as a step toward, rather than the complete realization of, an Industrial Metaverse. While the system supports immersive XR interaction, synchronous collaboration, semantic asset representations, and real-time digital–physical coupling, it does not yet provide global-scale persistence, fully open cross-platform interoperability, or large-scale multi-user coexistence. Current persistence is maintained at the application and asset-knowledge level, and collaboration is limited to a small number of simultaneous users.

Finally, although the LLM-based assistant demonstrated strong practical utility under constrained human-in-the-loop operation, it does not yet provide fully validated safety guarantees for autonomous or high-stakes decision-making. The AI component is intentionally restricted to advisory functions, and further research is required to systematically evaluate robustness, failure modes, hallucination mitigation strategies, and trust calibration before broader deployment in safety-critical maintenance scenarios.

Future work will focus on larger-scale and longitudinal evaluations across additional industrial domains, enhanced multi-user collaboration, improved XR ergonomics, and deeper integration of trustworthy AI mechanisms. In parallel, continued attention to data quality and digital maturity remains essential, particularly for small and medium-sized enterprises, where the benefits of advanced maintenance support frameworks are closely tied to the completion of foundational digital transformation initiatives.