Augmented Reality-Based Training System Using Multimodal Language Model for Context-Aware Guidance and Activity Recognition in Complex Machine Operations

Abstract

Augmented Reality (AR) and Large Language Models (LLMs) have made significant advances across many fields, opening new possibilities, particularly in complex machine operations. In complex operations, non-expert users often struggle to perform high-precision tasks and require constant supervision to execute tasks correctly. This paper proposes a novel AR-MLLM-based training system that integrates AR, multimodal large language models (MLLMs), and prompt engineering to interpret real-time machine feedback and user activity. It converts extensive technical text into structured, step-by-step commands. The system uses a prompt structure developed through an iterative design method and refined across multiple machine operation scenarios, enabling ChatGPT to generate task-specific contextual digital overlays directly on the physical machines. A case study with participants was conducted to assess the effectiveness and usability of the AR-MLLM system in Coordinate Measuring Machine (CMM) operation training. The experimental results demonstrate high accuracy in task recognition and feature measurement activity. The data further show reduced time and user workload during task execution with the proposed AR-MLLM system. The proposed system not only provides real-time guidance and enhances efficiency in CMM operation training but also demonstrates the potential of the AR-MLLM design framework for broader industrial applications.

1. Introduction

Augmented reality (AR)-based systems have emerged as promising solutions for industrial training [1]. AR has significantly improved training efficiency by providing real-time guidance, helping operators learn new skills quickly, and reducing training time and material waste [2]. In the context of complex machine operations, AR training systems enable users to visualize step-by-step guidance in their actual work environments.

Despite these advancements, most AR-based training solutions remain ineffective. In these systems, the training flow assumes that users follow instructions exactly in the same order [3]. However, in actual operations, users frequently rely on technical manuals or written instructions to perform the procedures. These documents are often lengthy, densely worded, and not always intuitive, particularly for novices who are unfamiliar with machine interfaces. Even when the text is understood, execution in a real environment is difficult because the manuals lack concrete visual guidance or the spatial context of each step. For example, a manual might state, “set the tool offset before starting the process”, without indicating where the offset controls are located on the machine or how to confirm that the task has been performed correctly. Similarly, another issue arises when trainees encounter a Human–Machine Interface (HMI) that displays errors or machine feedback, further confusing users in determining the correct action to take [4]. Thus, extracting operational steps from raw textual data and integrating them with live machine feedback in training systems to generate actionable guidance remains a complex challenge [5].

To overcome the limitations of static guidance, researchers have integrated various computer vision methods into AR. However, these methods remain rigid, requiring extensive training and massive labeled datasets for every new machine interface or operational scenario. While the integration of LLMs in AR opens new possibilities for complex procedures [6,7,8], they lack the real-world awareness necessary for industrial environments. This creates a semantic gap where the system understands the instruction text but cannot perceive the physical machine states or the user actions required to execute them as it lacks the inherent ability to process visual context. The emergence of MLLMs provides a solution by jointly processing textual and visual data in a unified framework, enabling the system to perform complex multimodal reasoning. Because these models are pre-trained on vast datasets, they can handle diverse machine behaviors and varying procedural tasks through zero-shot reasoning capabilities, eliminating the need for task-specific retraining. Therefore, we propose an AR-based training system that can read dynamic manual instructions in real time and incorporate visual input in an AR environment to interpret both machine states and user activities in complex machine operations. The core contributions of this work are:

• We propose a system that utilizes ChatGPT (vision-enabled configuration) to interpret real-time machine feedback and technical instructions, bridging the semantic gap between physical environments and digital instructions.
• We introduce the TARCO framework, a structured interface layer designed to map the stochastic outputs of MLLMs to the deterministic requirements of AR environments. This framework eliminates instructional drift by transforming complex technical instructions into precise, stepwise AR commands anchored directly to the physical machine.
• We provide empirical evidence demonstrating the effectiveness of the proposed model. Our results indicate a significant increase in usability, alongside a measurable reduction in cognitive load and task completion time during complex industrial operations.

This research presents an innovative AR-based training architecture that integrates an MLLM with the TARCO framework to facilitate context-aware procedural guidance. This proposed model mitigates the challenges of understanding machine feedback and interpreting technical instructions in complex industrial operations, providing valuable insights and supporting the application of AR-MLLMs in diverse fields.

The remainder of this paper is organized as follows. Section 2 presents a comprehensive literature review that critically analyzes existing research on AR, LLMs, VLMs, and MLLMs in operational planning. It also highlights the key developments and gaps in the current literature. Section 3 introduces our proposed system, including the design and integration of the MLLM in AR. Section 4 details the implementation of the proposed system. Section 5 presents the operation of the proposed system. Section 6 outlines the method used to evaluate the system’s performance. Section 7 reports and discusses the results of the study. Finally, Section 8 concludes the paper and outlines directions for future research.

2. Related Work

2.1. Augmented Reality in Industrial Training

Augmented reality (AR) has been widely reported to improve industrial training and assistance by placing instructions in the worker’s field of view, reducing time-to-competence and errors in Industry 4.0 scenarios [9,10]. Several studies have explored the potential of AR across various industries [11]. AR was mostly used in the gaming and entertainment industries; however, in recent years, AR technologies have become increasingly popular in professional training and industrial settings [12]. Unlike other training methods, such as virtual reality (VR), AR overlays digital information over the user’s real-world view, allowing users to interact with both physical and virtual aspects simultaneously [13,14]. AR has significantly improved production efficiency by providing real-time guidance [15], helping workers learn skills quickly, and reducing training time and material waste [2]. For instance, an AR-assisted guidance system for the assembly of avionics equipment was developed to provide dynamic assembly instructions overlaid onto a real-world environment using real-time pose tracking [16]. This method increased worker performance and reduced errors in assembly tasks. The advantages of AR have been further evaluated in material-forming and machining processes, where AR training resulted in faster completion times and fewer errors than traditional video or paper-based manuals [2]. AR also offers significant advantages in railway maintenance training and task execution by providing realistic and engaging content [17]. Similarly, in the tool change process, AR reduced cognitive load by overlaying visual cues and instructions on the real-world environment. This step-by-step guidance of AR enables users to perform tasks easily and efficiently, improving their overall effectiveness [18]. AR is particularly effective for initial task exposure because it reduces cognitive load compared to other technologies [19,20,21].

Despite these advancements, existing AR applications predominantly focus on predefined training content and assume an ideal process flow. However, these systems often struggle to keep pace with the nature of real-world operations, where the machine status and user activity vary in real time, creating a bottleneck for their widespread implementation. This gap highlights the opportunity to leverage AR for complex machine operations by integrating adaptive, context-aware mechanisms that interpret live conditions and provide real-time, task-specific guidance.

2.2. Integration of MLLMs

For many years, industrial operations, maintenance, and training have relied heavily on static manuals and the experience of skilled workers. Although this approach has served its purpose, it often slows work, especially on tasks that require precision or pose safety risks. The integration of Artificial Intelligence (AI), particularly LLMs, has introduced a transformative paradigm for industrial operations [22]. Recent literature underscores this shift, highlighting that LLMs fundamentally reshape task interpretation, execution, and planning across complex workflows [23]. This transformation is exemplified by the “RoboGPT” system, which employs LLMs to generate automated step-by-step instructions for robot-based assembly tasks in construction [6]. These models can optimize assembly sequence planning by decomposing tasks into logical task steps. Beyond assembly sequences, LLMs, such as GPT-4, have been utilized to convert textual instructions into physical steps for tasks such as electric panel maintenance. The system employs optical character recognition (OCR) to process text instructions that are then sent to the ChatGPT server. The server processes the input and transforms complex instructions into simple, sequenced operational commands. These commands are subsequently sent back to the AR system to invoke virtual objects and display relevant prompts on the physical machine [7]. Another study demonstrated that ChatGPT-3.5 could successfully generate the underlying code for Web-based Augmented Reality (Web-AR) applications from natural language prompts, suggesting that LLMs can automate parts of the development workflow, thereby making AR technology more accessible to individuals without specialized programming skills [24].

Despite these significant advancements, LLMs remain limited in their understanding of physical tasks, which often require visual perception beyond language. Visual inputs provide essential context about the environment, such as the machine state and user actions, which cannot be reliably inferred from text alone. To address this limitation, recent work has leveraged multimodal large language models (MLLMs) that jointly process vision and language for more efficient scene understanding [25]. For instance, Text to Automated General-purpose Guidance in AR (TAGGAR) utilizes LVLMs to create general-purpose AR task guidance without requiring expertise or complex computer-aided design [26]. This system integrates GPT-4V to process natural language instructions and images, generating visual guidance and accurately anchoring visuals in the physical environment using “GroundingDINO,” an object-detection model that locates targets and renders appropriate AR visuals. In another study, a similar vision–language model, GPT-4V, was used to provide cognitive assistance and feedback to users. This system employed a reality encoder to capture audio, images, and 3D space data and convert them into inputs for Multimodal Large Language Models (MLLMs). These inputs were processed using GPT-4V and Ferret for reasoning and object detection. Subsequently, a reality decoder converts the MLLM output into AR overlays and anchors them in real-world environments [8].

2.3. Prompt Engineering

The above discussions show that MLLMs can interpret responses in both textual and visual contexts. This is because these models are based on the transformer architecture and focus only on the instructions and parts that matter most in the context. This enables them to understand complicated instructions and generate appropriate responses [7,27]. However, when these models are used without additional input or external tools, such as structured prompts or domain-specific data, they produce inaccurate or incomplete outputs. Some studies have pointed out that existing MLLMs face challenges in interpreting correctly and struggle to apply domain rules that are obvious when deployed for general-purpose tasks [28]. This limitation makes it difficult to rely solely on MLLM outputs in real-world environments, particularly in industrial operations that require precision, safety, and consistency.

To overcome these issues, researchers in AI and engineering have explored various methods to make MLLMs more reliable and produce more accurate responses. Two main strategies have emerged in this regard. The first is the fine-tuning of LLMs on domain-specific tasks using resources such as equipment manuals, training datasets, and maintenance logs. This method has increased accuracy because the models become familiar with the domain’s workflow. However, fine-tuning can improve task relevance and accuracy, particularly in specialized industrial domains. However, it requires large, high-quality datasets, considerable training time, and ongoing updates whenever the domain changes, which is not always practical in industry. Additionally, it reduces the flexibility of the model in specific contexts [29,30].

The second increasingly popular strategy is to use prompts rather than change the model itself. This approach can guide the model step by step without modifying its core architecture. Research shows that even small adjustments in a prompt can greatly improve a model’s reasoning ability and reduce errors [31]. This is further confirmed by recent comparative studies that show that prompt tuning outperforms fine-tuning, particularly in tasks that require flexibility and contextual inference [32].

Studies on maintenance, construction, and AR-based training have shown that without clear prompts, multimodal models may focus on the wrong objects, misunderstand a user’s intention, or misinterpret visual details that humans would instantly recognize [26]. Thus, various prompt techniques, such as asking the model to think step-by-step, assigning it a specific role, or defining the specific output format, have been designed and have shown significant improvements in many tasks [33,34,35,36,37]. Another advantage of the prompt technique is that it is simple to apply, requires fewer resources than fine-tuning, and can be reused in different domains. These benefits make prompt engineering an attractive choice for real-world operations, particularly in industrial environments, where efficiency and adaptability are essential [38].

In conclusion, multimodal LLMs have shown great potential and are becoming increasingly common; however, to the best of our knowledge, existing studies have not yet deeply explored their integration for complex machine operations. The current system primarily focuses on general-purpose applications and lacks implementation for inferring machine states and recognizing real-time user activities. Moreover, industrial environments often contain many overlapping objects and cluttered backgrounds, which can confuse MLLMs. Thus, a structured prompt design is necessary to ensure rapid deployment and help MLLMs deliver predictable, safe, and reliable outputs in real-world industrial workflows [7].

3. System Design

3.1. Overview

The proposed system is designed to assist users in performing complex machine operations in real time by combining visual recognition and large-language models. It is connected to a vision–language model (ChatGPT) and deployed on the Microsoft HoloLens 2. The proposed method is built to interpret both instructions and dynamic machine states, providing real-time feedback in the user’s physical environment.

3.2. System Architecture

This AR-based training system is developed using the Unity game engine (6000.0.29f1) [39], which serves as the primary component for integrating AR with AI assistance. To maintain precise alignment between virtual instructions and content on the physical machine, we use local spatial anchoring with Microsoft’s Mixed Reality OpenXR (1.11.2) and AR Foundation (6.0.3) plugins. The core of the system is the multimodal ChatGPT (gpt-5-chat) model deployed via the Azure OpenAI Service, as shown in Figure 1.

Communication between the HoloLens application and ChatGPT is handled through a C# API implemented in Microsoft Visual Studio 2022, which sends multimodal requests and receives structured responses, which are then parsed and rendered as AR guidance in the Unity application. At the start of training, the user captures an image with the HoloLens camera [40] to recognize the current activity and sends an input to the ChatGPT model via a secure API request [6,7,8,24,27]. Once the input is received by the ChatGPT model, it interprets the image using prompt structure (TARCO) and performs multimodal reasoning to recognize the nature of the task. Upon receiving a response from the ChatGPT online server, the output is returned in structured JavaScript Object Notation (JSON) format. The JSON is parsed in the Unity application using C# script, which allows the AR application to activate the AR interface. If the input is purely textual, such as a manual instruction, the structured prompt with the extracted content asks system to convert it into clear, stepwise instructions. Similarly, if the input recognizes the user activity and machine’s HMI feedback, ChatGPT interprets the image and triggers the corresponding anchor. Finally, the output is rendered in the user’s field of view using the spatial anchor method. This system pipeline continuously infers both the machine state and operator activity from a live scene. Whenever the user encounters a new on-screen instruction, ChatGPT recaptures and reprocesses the frame. This process ensures that visual cues and actions are synchronized during training. Moreover, for real-time interaction and user engagement, the training system incorporates the Microsoft HoloLens 2 AR headset (Microsoft Corp., Redmond, WA, USA). This device facilitates intuitive user interactions and allows trainees to engage with virtual instructional content and manipulate digital elements in an AR environment.

4. System Implementation

4.1. AR-MLLM Workflow

In the proposed system, a multimodal inference pipeline is implemented for complex machine AR-based training. This includes local heuristic validation to filter the low-quality captures, followed by semantic validation to receive accurate output from the MLLM, as illustrated in Figure 2. This dual approach ensures only clear visual data is processed, which results in deterministic spatial anchor activation.

4.1.1. Stage 1: Heuristic Validation

Industrial lighting variations and hardware-level luminance constraints are identified as major challenges that can adversely impact the reliability of high-precision tasks in AR environments [41]. To mitigate this issue, the AR–MLLM workflow initiates when a user captures a high-definition frame through HoloLens 2 PhotoCapture API in the AR environment. The system immediately executes the heuristic validation and calculates the mean luminance (${\mathrm{L}}_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}$) of the frame by averaging the weighted RGB components of all n pixels as defined in Equation (1),

$${\mathrm{L}}_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}={\displaystyle \frac{1}{\mathrm{n}}}{\sum }_{\mathrm{i}=1}^{\mathrm{n}}(0.2126{\mathrm{R}}_{\mathrm{i}}+0.7152{\mathrm{G}}_{\mathrm{i}}+0.0722{\mathrm{B}}_{\mathrm{i}}),$$

(1)

where R_i, G_i and B_i represent the red, green and blue colors for the ${\mathrm{i}}^{\mathrm{t}\mathrm{h}}$ pixel. The coefficients for these components are derived from the ITU-R BT.709-6 standard to calculate the relative luminance, ensuring the visual data is suitable for semantic interpretation [42]. The system accepts the frame only if it satisfies the heuristic quality criteria defined in Equation (2),

$${\mathrm{L}}_{\mathrm{m}\mathrm{i}\mathrm{n}}{<\mathrm{L}}_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}}{<\mathrm{L}}_{\mathrm{m}\mathrm{a}\mathrm{x}},$$

(2)

where ${\mathrm{L}}_{\mathrm{m}\mathrm{i}\mathrm{n}}$ and ${\mathrm{L}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ represent the lower and upper luminance boundaries, respectively. During experiments, it was observed that frames falling below 0.08 resulted in insufficient object-to-background contrast. Similarly, the fine details of the machine part were lost when frames reached to 0.92. Thus, these thresholds were set to ensure the MLLM receives frames with sufficient feature contrast for accurate interpretation. Furthermore, if a frame falls outside this threshold range, the system is configured to reject it locally to prevent unnecessary MLLM costs and API token waste. This rejection initiates a local retake loop to capture a new frame. If the first frame is rejected, the system performs up to two recursive retakes before proceeding.

4.1.2. Stage 2: Semantic Validation

Once the frame is confirmed as valid during the local validation stage, it is forwarded to the MLLM via an HTTP request. The system uses ChatGPT (gpt-5-chat version: 2025-08-07) integrated into the workflow via the Azure OpenAI service. To achieve an accurate interpretation of the physical environment, the model is initialized with a fixed system prompt of “You are a CMM assistant. Respond with only the requested format”.

Subsequently, to reduce the model search space and ambiguity in output, the visual input is processed through the TARCO prompt framework as shown in Equation (3). This framework facilitates domain-specific reasoning through two strategies: Technical Instruction Extraction for transforming manual text into structured command output, and User Activity and Machine Feedback Recognition for interpreting operator actions and displaying feedback:

$$\mathsf{\Phi }=(\mathrm{T},\mathrm{A},\mathrm{R},\mathrm{C},\mathrm{O}),$$

(3)

where T is the task definition, A represents allowed actions, R is the reasoning procedure, C denotes constraints and rules, and O is the output format. To ensure industrial-grade reproducibility and deterministic output, the model is configured with a temperature of 0.0, and a maximum response limit of 200 tokens. These parameters ensure that the final output $\left(\mathrm{Z}\right)$ is a stable, reproducible function of the visual input (I) and the structured TARCO prompt $\left(\mathsf{\Phi }\right)$. The relationship is expressed in Equation (4):

$$\mathrm{Z}={\mathrm{f}}_{\mathrm{M}\mathrm{L}\mathrm{L}\mathrm{M}}(\mathrm{I},\mathsf{\Phi }),$$

(4)

If the model perceives the visual input but lacks sufficient feature clarity for a high-confidence inference, it issues a “RETAKE” command to re-initialize the cycle until a deterministic state is reached.

4.1.3. Stage 3: Spatial Mapping and Prefab Activation

The final stage performs a semantic-to-spatial mapping. The AR system parses the MLLM output and attempts to match each label with its corresponding prefab on the machine. When the mapping is successful, the corresponding prefab is activated in the AR scene, guiding the user to the next step. The user can then move forward or revisit previous steps, creating a continuous, interactive loop throughout the training process. If the output does not conform to the expected schema, or if the user wants to proceed to the next operation, they trigger a correction loop and recapture an image to ensure the response is valid either for execution or for the next operation.

Figure 3 shows when the system detects that a captured frame is blurred or low-quality during operation, it automatically re-captures the image. It repeats this process up to 2 times. After the Unity application confirms that the image meets the required quality, it sends the image to ChatGPT along with a structured prompt for visual interpretation. This quality control step prevents the AR-MLLM from generating responses based on unreliable visual input and avoids triggering an incorrect prefab in the AR scene.

4.2. TARCO Prompt Structure

MLLMs are primarily trained on general-purpose datasets and lack the domain-specific contextual knowledge needed to interpret complex machine operations accurately. In addition, technical instructions and machine feedback often contain symbolic characters, logical dependencies, and repetitive terminology, making it difficult for models to generate appropriate responses when these raw inputs are processed directly [6]. Similarly, many existing VLMs often rely on open-ended prompting and are prone to instructional drift. In industrial settings, such general models are unreliable for deployment as they may omit critical procedural steps or inappropriately rephrase technical terminology, which can lead to operational errors [7,26,28].

To mitigate this stochastic behavior, we designed the TARCO framework, which utilizes a rigid hierarchical structure to map AI reasoning into a constrained JSON-output schema. This framework operates as a secondary input instruction layer to guide the model, enabling it to produce outputs that are clear, structured, and directly usable within the AR training workflow [43]. Furthermore, this approach ensures that model responses are not only contextually accurate but also functionally anchored to the machine state, providing actionable guidance within the AR environment. The framework consists of the following five core components.

• Task Definition (T): This specifies the role of the MLLM and instructs it to extract the required information. It defines the model’s objective so that all its responses remain aligned with the specific machine.
• Allowed Action (A): This part restricts the model to a predefined set of allowable actions and provides a strict JSON schema that must be followed, preventing models from producing their own actions that do not exist in the AR system.
• Reasoning Procedure (R): It guides the model to follow the execution order as defined in the technical instruction and ignore any shortcut steps from the technical instruction.
• Constraints and Rules (C): These rules instruct the model to use only those commands that match the technical instructions and prevent others that do not exist. It also instructs the model not to rephrase or invent labels on its own.
• Output Format (O): This part guides the model to convert the output to the required format without additional explanation. This strict format ensures that the output is executed reliably.

Table 1. Prompt structure for understanding technical instructions and user activity recognition.

Components	Technical Instruction Prompt	User Activity Recognition and Machine Feedback Prompt
Task Definition	1. You are an AR command extractor for a Coordinate Measuring Machine (CMM). The machine includes components such as the probe head, stylus, workpiece, and measurement features (i.e., circles, cylinders, spheres).	1. You are an AR model for a Coordinate Measuring Machine (CMM) operation. The machine includes components such as the probe head, stylus, workpiece, and measurement features (i.e., circles, cylinders, spheres).
	2. Your task is to convert the technical instructions into a single JSON object and ignore all keyboard shortcuts in the manual instructions.	2. Your task is to detect machine feedback displayed in the image.
		3. Identify the type of feature being measured by the CMM (circle, cylinder, or sphere) and calculate its diameter and/or length in inches.
Allowed Action	1. Translate the step-by-step instructions into AR UI actions using ONLY manual/touch interactions.	1. When the CMM displays numbered instructions for operation, return only: Number: <digit> OR
	2. Follow the strict JSON schema: {"commands": [{"action": "open_menu", "menu_path": ["<Top>", "<Sub>"]}, {"action": "click", "target": "<ButtonLabel>"}, {"action": "select", "target": "<OptionLabel>", "group": "<ControlNameOptional>"}]}.	2. When the user sees the workpiece, follow the required output schema: Diameter: <value or N/A> Length: <value or N/A> Feature: <Circle|Cylinder|Sphere>.
	3. For menu notation in instructions such as "Menu: A then B", follow this schema: {"commands": [{"action": "open_menu", "menu_path": ["A", "B"]}]}.
Reasoning Procedure	1. Preserve the exact step sequence as written in the technical manual.	1. Examine stylus–surface contact (probe tip contact points), curvature, edges, and feature geometry.
	2. Map textual instructions directly to the pre-defined AR UI actions.	2. When the probe/stylus is visibly in contact with the workpiece feature, infer the feature type from the visible contact pattern and measurement context.
Constraints and Rules	1. Use only exact UI labels from the technical instructions (i.e., “Qualify”, “Stylus Manager”, “Ball”, “Measure”, “Circle”, “Cylinder”, “OK”, “ID”).	1. If the diameter cannot be determined, return N/A for diameter.
	2. Do not invent or rephrase UI labels. Labels must match the real interface exactly.	2. Ensure the chosen feature type matches visible CMM contact points and shape cues.
	3. Never produce actions that do not exist in the instruction set.
Output Format	1. The output must contain only JSON objects, with no explanations, notes, or additional text.	1. Output must contain only the required format: Number: <digit> or Diameter: <value or N/A> Length: <value or N/A> Feature: <Circle|Cylinder|Sphere> (if analyzing a feature dimension).
		2. The output must not contain explanations, notes, or additional text.

presents the prompt structures used in this study for CMM operation. These prompts were developed and refined through an iterative testing process across multiple machine operation scenarios. They enable the system to convert lengthy and complex instructions into executable actions that can be processed in real time, while also supporting the recognition of user activities. Furthermore, the prompts are adaptable to various types of input, improving the model’s ability to generate accurate, contextually relevant outputs.

4.3. Spatial Anchoring

To maintain the alignment of virtual prefabs on a physical machine, a local spatial anchor is used. An anchor key is assigned to each virtual prefab, as shown in Figure 4a, allowing the AR anchor manager to save and reload anchors and maintain the content fixed at the same position in the real world. After spatially anchoring the prefabs, a vision-to-action detector component is applied, as shown in Figure 4b, which sends the captured images to ChatGPT via the Azure OpenAI endpoint. Based on the model output, the system determines which prefab should be triggered and activated in the AR scene, as shown in Figure 4c. Together, these steps facilitate seamless synchronization between the virtual and physical machine states, which is essential for delivering a realistic training experience.

5. AR-MLLM CMM Training

An AR-MLLM-based training system for Coordinate Measuring Machine (CMM) operations (Helmel Engineering Products, Inc., New York, NY, USA) is developed for a case study of our proposed method [1]. Training operators to master CMM workflows remains a critical challenge in industries that require zero-defect manufacturing [43]. CMM operation requires precise configuration and calibration to guarantee accurate measurements, which is essential for ensuring that the final product meets the required specifications [44]. As shown in Figure 5, the workflow includes stylus qualification, establishment of the part coordinate system, feature measurement, and data analysis, which presents challenges for novice operators [1].

To provide a realistic simulation of CMM training and an accurate representation of real-world scale and spatial relationships [45], a virtual CMM is precisely designed and deployed on a real machine, as shown in Figure 6. This includes probe, sphere, vertical bridge, horizontal bridge and joystick controller. For the stylus calibration task, virtual prompts are anchored on the sphere and CMM display. These prompts are activated whenever users wear the AR HMD and use the MLLM to provide guidance.

5.1. Manual Instruction and Machine Feedback

AR-MLLM operation training begins when users capture technical instructions from machine manuals using an MRTK hand interaction menu button. These virtual buttons provide the input for taking pictures and command navigation, allowing users to confirm a step or advance to the next one without using the machine’s physical controls. The AR-MLLM then performs visual–textual reasoning to interpret the captured instructions and returns the step-by-step prompts to the AR device. Figure 7 shows a stylus calibration task in which virtual arrows with labels are activated on the physical machine display to assist the user during the operation. The same workflow can be activated whenever users capture instructions for feature measurements, allowing the model to interpret the content and immediately render the corresponding guidance.

In complex machine operations, the HMI serves as the primary source of feedback on the machine’s current state. Executing steps solely based on feedback leads to incorrect actions, as these displays do not provide visual guidance directly on the machine. This increases user workload and task completion time. To reduce these risks, the proposed AR-MLLM system also allows users to monitor machine feedback in real time and forward it to a ChatGPT-based reasoning model to interpret the current state and render AR guidance to the machine for operator assistance. As depicted in Figure 8, the sphere prompts during stylus qualification are dynamically updated in response to CMM display feedback, providing step-by-step guidance and keeping the user aligned with the required actions.

5.2. Activity Recognition

MLLMs can interpret physical space by processing visual inputs, but they still lack the ability to reliably recognize user activity as quickly as humans can understand task context. Thus, to help the AR-MLLM accurately recognize user intent and reduce ambiguity in inferring the task, a user activity recognition prompt is used, as discussed in Section 4, to support human-like task interpretation during operation. For example, during feature measurement activity, the prompt specifies the operating context is “feature measurement on CMM”, the entities of interest “stylus tip and target feature in contact”, and the required inference “identify the feature and estimate its dimension”.

Figure 9 shows the response generated by ChatGPT in an AR environment during the CMM feature measurement task. The AR-MLLM identifies user activity by observing the stylus tip’s position and computing its approximate diameter in real time, without prior measurement data. As shown in Figure 9, the model identifies the targets as a circle, a sphere, and a cylinder, and computes their approximate diameters in real time without prior measurement data.

6. Evaluation

A user study was conducted to compare AR-MLLM training against a traditional AR baseline (without AR-MLLM) in terms of usability and perceived workload. In addition to this, the Bland–Altman analysis was used to compare the AR-MLLM feature measurements with CMM measurements [46]. Two well-established assessment measures are utilized: System Usability Scale (SUS) and NASA Task Load Index (NASA-TLX) [19,47]. The SUS consists of 10 questions that yield scores ranging from 0 to 100, providing an overall usability score. Each question measures a different aspect of usability, such as ease of use, confidence in using the system, complexity, and consistency. Participants rated each item on a 5-point Likert scale (1 = Strongly Disagree to 5 = Strongly Agree). The SUS score is computed using Equation (5): first transform the item responses, sum the transformed scores, and then multiply the total by 2.5 to convert it to a 0–100 scale. Higher scores indicate better usability, whereas lower scores indicate poorer usability.

$$\mathrm{S}\mathrm{U}\mathrm{S} \mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\left(\sum \mathrm{t}\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{s}\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{d} \mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}\mathrm{s}\right)\times 2.5,$$

(5)

The NASA-TLX was used to assess the workload of users during CMM operation tasks. It consists of six dimensions (mental demand, physical demand, temporal demand, performance, effort, and frustration). The NASA-TLX score was calculated using Equation (6) by averaging ratings across the six dimensions, where higher scores indicate a higher perceived workload, and lower scores indicate a lower perceived workload during the operation.

$$\mathrm{N}\mathrm{A}\mathrm{S}\mathrm{A}-\mathrm{T}\mathrm{L}\mathrm{X} \mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\left({\displaystyle \frac{\sum (\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{o}\mathrm{f} \mathrm{a}\mathrm{l}\mathrm{l} \mathrm{d}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{6}}\right),$$

(6)

Similarly, to evaluate the efficiency of the AR-MLLM, the task completion time during the qualification process was measured. A shorter completion time indicates a more efficient system. The performance metrics were compared to the traditional AR method using the Wilcoxon rank-sum test to assess statistical significance. Furthermore, to verify the accuracy of the proposed model, task recognition and feature measurement of the workpiece were validated during CMM operation. The task recognition accuracy of AR-MLLM was validated using Recall (R), which measures the model’s ability to identify all features, and Precision (P), which measures the accuracy of those identifications:

$$\mathrm{R}=\left({\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}\right),$$

(7)

$$\mathrm{P}=\left({\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}\right),$$

(8)

where TP, FN and FP represent the true positive outcomes, false negative outcomes and false positive outcomes, respectively. Furthermore, feature measurements were validated against CMM measurements using the Mean absolute percentage error (MAPE) and measurement accuracy as follows:

$$\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E}={\displaystyle \frac{1}{\mathrm{n}}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\left({\displaystyle \frac{\left|{\mathrm{x}}_{\mathrm{i}}-{\mathrm{y}}_{\mathrm{i}}\right|}{{\mathrm{y}}_{\mathrm{i}}}}\times 100\right),$$

(9)

$$\mathrm{Accuracy}=100-\mathrm{M}\mathrm{A}\mathrm{P}\mathrm{E},$$

(10)

where ${\mathrm{x}}_{\mathrm{i}}$ and ${\mathrm{y}}_{\mathrm{i}}$ represents the AR-MLLM predicted dimensions and CMM measured dimensions. Moreover, dimensional agreement was further assessed using Bland–Altman analysis, which evaluates the mean bias and limits of agreement between the two measurement methods.

A total of 15 novice participants were recruited from undergraduate and graduate students majoring in Mechanical Engineering (aged 20 to 32 years). The experiment employed a within-subjects crossover design, in which all participants completed both conditions: traditional AR and AR-MLLM methods. Traditional AR (without AR-MLLM) provided spatial cues to participants, but it lacked the multimodal reasoning capabilities required to interpret real-time text and evolving machine states. Consequently, in this condition, the user had to manually read the instructions and mentally map them to their actions while wearing the headset.

To mitigate the learning effect, the order of conditions was shuffled for each participant, and the task content was varied between the two conditions. For example, one task involved extracting instructions from a technical manual to perform a CMM setup, while the other required extracting instructions from the CMM display (HMI) for stylus qualification. To keep task complexity equivalent, both tasks were designed with exactly five operational steps. Participants completed a brief pre-training session on the HoloLens 2 prior to data collection. This included interactions with holograms in the AR environment and menu buttons using the built-in Microsoft Tips app (1.2208.101.0) to ensure that all participants were aware of how to use the device before executing actual tasks and obtaining accurate results. AR-MLLM training starts when a user clicks the capture button and ends when probe calibration is successfully completed. Similarly, for traditional AR operations, the training time was counted from reading the instructions to completing the probe calibration. The time-to-completion results and recognition accuracy of the AR-MLLM system were recorded using the HoloLens 2′s built-in camera.

7. Results and Discussion

The task completion time, cognitive load, and usability of both methods for each participant were evaluated using a non-parametric Wilcoxon rank test. Two hypotheses were formulated: the null hypothesis (H₀), stating that there is no significant difference between the AR-MLLM and the method without AR-MLLM training; and the alternative hypothesis (H₁), proposing that there is a difference between the two methods, with a significance level of $p=$ 0.05.

As shown in Figure 10a, the results indicate that the task completion time for participants trained with the AR-MLLM method is lower than that for participants trained without it. The average time for AR-MLLM is 176.12 s, compared to 211.24 s for those who trained without the AR-MLLM method, with a difference of $p=8.03\times {10}^{-4}$, indicating a statistically significant reduction in task duration (

Table 2. Average values of AR-MLLM and without the AR-MLLM method.

Method	Task Execution Time (s)	Usability (Score)	Task Load (Score)
AR-MLLM	176.12	81	30.12
Without AR-MLLM	211.24	25	65.22
Difference	35.12	56	30.34

). Furthermore, we recorded an average end-to-end latency of 4.6 s for the transmission of visual data and the receipt of responses from the AR-MLLM. This response time is achieved through direct integration between the AR headset and the MLLM server, which minimizes the communication overhead often found in multi-layered intermediary solutions [7]. Despite this latency, the total task completion time remained significantly lower with the AR-MLLM system, confirming its practical viability for industrial applications where procedural accuracy and overall time efficiency are important.

Figure 10b shows the NASA-TLX workload analysis for the AR-MLLM application. The survey further reveals that the AR-MLLM training conditions produced significantly lower workload scores across all subscales, with a significant difference between the two methods $(p=6.51\times {10}^{-4})$. Participants also rated the AR-MLLM system significantly higher in terms of usability, with an average score of 81 (Table 2). However, some of the participants provided neutral responses, but still results remained statistically significant $(p=1.42\times {10}^{-3}$) as shown in Figure 10c.

To further validate the system performance, we evaluated both task recognition and feature measurement accuracy. Task recognition was assessed across 45 total trials (15 trials for each of the three feature types) as shown in

Table 3. Performance metrics for AR-MLLM feature type recognition.

Feature Type	Sample Size	Precision	Recall	MAPE (%)	Accuracy (%)
Circle	15	0.882	1.000	0.068	99.932
Cylinder	15	1.000	0.933	0.464	99.536
Sphere	15	1.000	0.933	0.048	99.952

. The AR-MLLM utilizes an activity recognition prompt that clearly defines the operating context and entities of interest. The real-time observation of the stylus tip’s position in contact with the target enabled the model to identify these features with high accuracy. This context-aware approach allowed the model to reduce ambiguity and correctly identify the target feature in nearly all trials. However, as shown in Figure 11, circle recognition achieved a precision of 0.882, with two false positives: one cylinder and one sphere were misclassified as circles. This occurred when the camera was aligned parallel to the longitudinal axis, capturing only the circular cross-section and obscuring the feature’s length. Similarly, the sphere misclassification resulted from the external placement of the probe tip. This caused the MLLM to perceive only the 2D circular boundary, making it impossible to tell the difference between a sphere and a flat circle. After this identification, the subsequent data were then collected carefully to mitigate such ambiguities in the results.

Similarly, a Bland–Altman analysis was performed to compare AR-MLLM measurements with CMM measurements. It plots the difference between the two methods against their mean for each trial, including the mean bias and 95% limits of agreement. Figure 12 demonstrates that the differences for AR-MLLM and CMM measurements mostly fell within the limits of the agreement for all features, showing a close agreement between the two methods across trials with LOA $(-2.123\times {10}^{-3}$ to $1.218\times {10}^{-3} \mathrm{i}\mathrm{n})$ for circle, LOA ($9.496\times {10}^{-4}$ to $5.972\times {10}^{-3}$ in) for cylinder and LOA $(-5.537\times {10}^{-4}$ to $1.363\times {10}^{-3}$ in) for sphere.

As depicted in Figure 13a,b, the AR-MLLM slightly underestimates the circle diameter, with a mean difference of $(-4.527\pm 8.522)\times {10}^{-4}$ in. In the cylinder measurements, the AR-MLLM method shows a positive mean difference of $(3.461\pm 1.281)\times {10}^{-3}$ in. This suggests that the AR-MLLM measured the cylinder diameter approximately 0.0035 in larger than the CMM on average. Similarly, sphere measurements follow the same trend, showing a slight positive bias with a mean difference of $(4.047\pm 4.890)\times {10}^{-4}$ in, as shown in

Table 4. Feature measurement results.

Feature Type	Standard Deviation (SD)	Mean Bias Difference	MAPE (%)	Accuracy (%)
Circle	$8.522\times {10}^{-4}$	$-4.527\times {10}^{-4}$	0.067	99.932
Cylinder	$1.281\times {10}^{-3}$	$3.461\times {10}^{-3}$	0.463	99.536
Sphere	$4.890\times {10}^{-4}$	$4.047\times {10}^{-4}$	0.047	99.952

. While the cylinder exhibited the largest mean bias, this deviation was found within the original equipment manufacturer (OEM) design tolerance of $\pm 0.005 \mathrm{i}\mathrm{n}$. This shows that the system’s errors are functionally low for precision-critical industrial tasks.

Overall, all these results demonstrate good agreement between the two methods and show that errors remain small and bound within the limits of agreement, indicating consistent performance over the observed measurement range. Figure 13c,d further highlights the measurement accuracy of the AR-MLLM and shows that the model achieves accuracies of 99.932%, 99.536%, and 99.952% for circles, cylinders, and spheres with corresponding MAPE values of 0.067%, 0.463% and 0.047%. These results confirm the practical capability of the proposed AR-MLLM solution in complex precision measurement operations.

8. Conclusions

This research developed an advanced AR-MLLM-based training system for complex machine operations, where tasks are critical and require constant supervision. The use of ChatGPT and prompt engineering in this research provides greater expertise and enhances operational efficiency by interpreting real-time machine feedback and recognizing user activity. Unlike traditional AR training systems that rely on fixed, pre-scripted instructions, our system provides robust training methods by automatically updating the steps in the AR environment and directly superimposing digital overlays on the machine.

This AR-MLLM training system allows the user to capture live visuals of the working environment and send them to ChatGPT for processing and generating the output. These outputs are then returned to the AR application and provide guidance directly on the machine. We validated the proposed system using Coordinate Measuring Machine (CMM) operations. During the operation, the user captures images from the machine manual and the CMM display. These images are sent to ChatGPT using carefully designed prompts, enabling the model to understand the current step and provide accurate, context-aware guidance through digital overlays on the CMM.

We also evaluated the system by comparing the CMM task completion time with and without AR-MLLM training. The results show that participants using the proposed method completed the operation faster, indicating improved efficiency. To further assess accuracy, we estimated the workpiece feature dimensions using the AR-MLLM system and compared them with CMM-measured dimensions. This comparison demonstrates that the system can support users by recognizing their actions and providing feature measurements. Finally, we compared the user experience with traditional AR-based training and found that the participants reported higher satisfaction and lower workload when using our approach. Together, these results show that AR-MLLM training can be effective for complex operations that normally require constant supervision, while also improving productivity, reducing time and resource use, and increasing worker satisfaction in the manufacturing industry.

The proposed AR–MLLM training system utilizes ChatGPT as the multimodal reasoning platform; however, the core contribution lies in the system architecture rather than reliance on a specific language model. The spatial anchoring, AR guidance, and prefab activation mechanisms are implemented deterministically in the Unity engine, with the LLM serving as a modular decision-support component. Moreover, interactions with the MLLM are governed by the TARCO framework, which reduces variability and ensures consistent responses across repeated runs. The proposed approach is model-agnostic and can be extended to other multimodal LLMs supporting image–text input, while the overall AR–MLLM training workflow remains unchanged.

Although the user study was conducted with a relatively small sample size (N = 15), the results yielded high statistical significance across all evaluation metrics. The proposed system demonstrates significant performance improvements compared to the traditional AR method, with a 51.5% reduction in task load and a 35.12 s improvement in task execution time. These results provide strong preliminary evidence of the effectiveness of the proposed approach.

Despite these results, several limitations remain to be addressed in future research. As the study had a relatively small sample size and participants were primarily engineering students, the generalizability of our results may be influenced by this specific study group. Future research could expand the participant pool and include a more diverse range of users to ensure broader generalizability across industrial sectors. Additionally, voice-based interaction could be integrated alongside text and visual inputs to create a more hands-free experience. Lastly, the machine feedback could be directly connected to the MLLM in future work. By feeding real-time machine data directly into the model, the need for manual instruction capture can be eliminated, allowing the AR system to autonomously trigger digital overlays based on live machine states.