On-Premise Multimodal AI Assistance for Operator-in-the-Loop Diagnosis in Machine Tool Mechatronic Systems

Abstract

Modern machine tools are safety-critical mechatronic systems, yet shop floor maintenance from abnormal events still relies heavily on scarce expert know-how and time-consuming manual searches across heterogeneous controller documentation. This paper presents an on-premise multimodal AI assistant. It integrates large language models with retrieval augmented generation and real-time machine signals to support operator-in-the-loop fault diagnosis. The proposed system provides three tightly coupled functions: (1) alarm-grounded guidance, which answers controller alarms and recommends corrective actions by grounding generation on manuals, maintenance procedures, and historical alarm cases; (2) parameter-aware reasoning, which injects live process and health indicators (e.g., spindle temperature, vibration, and axis states) into the reasoning context through an industrial data pipeline, enabling context specific troubleshooting; and (3) vision enabled support, which retrieves similar visual cases and generates concise visual instructions when text alone is insufficient. The assistant is deployed within an intranet environment to satisfy industrial security and privacy requirements and is orchestrated via lightweight tool calling for seamless integration with existing shop floor systems. Experiments on real machine tool alarm scenarios demonstrate that the proposed system achieves 82% answer correctness for alarm Q&A and improves response consistency and time-to-resolution compared with baseline keyword search and template-based guidance. The results suggest that grounded, multimodal chatbot assistants can act as practical AI-based feedback and decision support mechanisms for mechatronic production equipment, bridging human skill gaps while enhancing reliability and maintainability.

1. Introduction

Machine tools are safety and quality-critical mechatronic systems that underpin modern manufacturing. Despite continuous advances, day-to-day operation and recovery from abnormal events still depend heavily on operators. They must interpret vendor-specific alarm codes, search heterogeneous manuals, and decide corrective actions under time pressure. This challenge is particularly acute for small and medium-sized enterprises [1,2,3]. In these settings, a shortage of skilled engineers and lack of systematic training frequently lead to prolonged downtime.

In other words, the bottleneck is no longer only the machine capability itself but the ability to integrate actionable intelligence into the mechatronic system at the human machine interface [4].

Recent studies have explored the use of LLMs in manufacturing for tasks such as quality defect analysis [5] and document-based question answering [6]. These approaches predominantly rely on text-based interactions and outputs, reflecting the current trend of LLM deployment in industrial environments.

However, most existing solutions exhibit two primary limitations. First, they are generally restricted to predefined question–answer patterns, which limits their ability to handle diverse and unforeseen scenarios that require adaptive reasoning. Second, they are predominantly text-based, despite the fact that many maintenance and setup tasks in mechatronic systems inherently involve visual elements, such as component localization, panel state interpretation, and verification of tooling or fixture conditions.

These limitations reduce the effectiveness of DIAs as AI-based feedback mechanisms in mechatronic environments, where decision-making should be grounded in both technical documentation and real-time machine states.

Large language models (LLMs) provide a promising path to move beyond scripted assistants by enabling natural language interaction, multi-step reasoning, and tool-augmented responses. Yet, deploying LLMs in industrial mechatronics raises practical concerns: (i) hallucinations and ungrounded recommendations can be unsafe; (ii) sensitive production data often cannot leave the factory network; and (iii) purely text-grounded responses may ignore the “physical context” captured by smart sensors and controller signals.

Therefore, to align with the vision of “Integrating AI into Mechatronics”, an LLM assistant should be (a) grounded on authoritative machine knowledge such as official documents, (b) connected to real-time mechatronic feedback (PLC/CNC/sensors), (c) capable of visual guidance, and (d) deployable on-premise within a LAN environment.

To address these needs, this paper presents a multimodal digital assistant for machine tools that integrates generative AI directly into the mechatronic operation loop. The proposed method consists of two main components. First, a textual assistant uses a fine-tuned LLM grounded via RAG and prompt engineering. Second, a visual assistant supports troubleshooting by retrieving similar cases or generating explanatory images. Critically, the proposed method also incorporates real-time machine parameters, such as spindle temperature, vibration indicators, and controller states, into the reasoning context through an industrial data pipeline. Unlike existing LLM-based industrial assistants that focus primarily on text-based interaction, the proposed system integrates multimodal guidance and real-time machine feedback into the reasoning loop.

The key contributions of this work are as follows:

• Mechatronic AI integration at the HMI: We propose an LLM-powered operator interface that transforms legacy manuals and alarm lists into step-by-step, actionable guidance for machine tool operation and maintenance.
• Grounded and privacy-conscious deployment: We design an on-premise architecture using private datasets and grounding mechanisms to protect confidential industrial information.
• Multimodal assistance for practical troubleshooting: We integrate textual and visual support, enabling operators to receive both procedural explanations and intuitive imagery aligned with real maintenance tasks [7].
• Real-time feedback-aware reasoning: We incorporate live machine parameters into the assistant’s context, enabling situation-specific diagnosis and recommendations that reflect the current mechatronic state.

The remainder of this paper is organized as follows. Section 2 reviews related work on industrial DIAs, LLM-based assistants, and multimodal guidance. Section 3 describes the overall system architecture and key modules of the proposed method, including data grounding, parameter integration, and visual support. Section 4 reports implementation details and experimental evaluation on real machine tool alarm scenarios. Section 5 discusses limitations and future directions toward safer and broader AI integration for mechatronic production equipment. Finally, Section 6 concludes the paper.

2. Related Work

2.1. Digital Intelligent Assistants in Manufacturing

DIAs have been increasingly studied in the manufacturing field for their potential to improve operational efficiency and reduce dependence on external technical support. Historically, expert systems formalized domain expertise into actionable logic for industrial troubleshooting, evolving from basic rule-sets to complex frameworks for predictive maintenance and quality control [8,9]. These systems have been implemented as chatbots or rule-based programs to guide workers through standard procedures or address basic equipment issues. For example, the adoption of AI-based chatbots in SMEs was examined by Panigrahi et al. [10], who reported improvements in supply chain responsiveness. A multimodal chatbot that processed voice, gesture, and text input for manufacturing support was proposed by Chen et al. [11]. A digital assistant supporting predictive maintenance was proposed to facilitate natural language interaction and human AI collaboration [12]. Cortana demonstrated early voice-based personal assistance using speech recognition and dialogue management [13].

However, current DIAs have been limited by their reliance on predefined workflows. Most systems have been constrained by fixed knowledge bases and have lacked the flexibility to manage unexpected or complex situations. In addition, their support has been primarily text-based, which has reduced usability in scenarios that require visual understanding, such as equipment operation or error handling. Bousdekis et al. [14] noted that the absence of domain-specific training has hindered these systems from delivering personalized assistance.

These limitations have demonstrated the need for DIAs that are adaptive. A system capable of handling diverse operational conditions is essential for supporting non-expert users in manufacturing environments.

2.2. Large Language Models in Industrial Applications

To overcome the limited adaptability of legacy DIAs, recent efforts have focused on applying LLMs to industrial domains. These attempts aim to improve the flexibility and contextual understanding of assistant systems through advanced natural language processing. An LLM-based assistant improved assembly quality and reduced cognitive load in manufacturing tasks [15].

LLMs such as GPT, BERT, and LLaMA have been recognized for their ability to understand and generate domain-specific content when fine-tuned with relevant data. These models have shown strong performance in natural language tasks, but their general purpose design has introduced risks including hallucinations and factual errors [16,17].

To mitigate these issues, RAG has been developed as a method that supplements LLMs with external knowledge retrieval. In this approach, relevant context is retrieved from a trusted source and supplied to the LLM, which improves factual accuracy and reduces the likelihood of hallucinations [18]. RAG has been especially valuable in applications that require high accuracy, such as industrial maintenance and diagnostics.

Despite the growing interest in LLMs, their application in industrial settings has remained highly limited. Most existing systems have focused solely on textual information and have lacked integration with machine interfaces or real-time data sources. As a result, their practical utility in manufacturing environments has been constrained.

The proposed system has addressed these limitations by incorporating both textual and visual information to enhance user comprehension. In addition, real-time machine parameters have been utilized to generate context-aware responses, enabling more practical and operationally relevant support. By combining structured manual fine-tuning, RAG-based retrieval, and visual assistance, the proposed system demonstrates how LLM-based assistants can overcome the generalization issues of rule-based DIAs and offer greater flexibility in real-world applications.

3. System Architecture

This section explains the structure of the proposed system and its components with the procedures of processing the contents of the given manuals. The overall system architecture is illustrated with the processing pipeline and is followed by both processes generating text outputs and image outputs. Text outputs come from the textual assistant module using a large language model trained with the text dataset composted of the manual descriptions. Image outputs are extracted from the image retrieval module or generated by the image generation module trained with the image dataset.

The proposed system is designed to assist operators in manufacturing environments by providing accurate and context-specific answers through a combination of text and image outputs. The architecture is structured to handle user queries, process relevant information, and deliver comprehensive responses consisting of text information, image information, and image descriptions.

The process begins when the operator submits a query regarding a machine’s operation or troubleshooting.

Based on this query, the system executes the RAG pipeline [18,19] (indicated in green in Figure 1). It retrieves relevant context from a dataset derived specifically from machine tool manuals, and the LLM generates the text information based on this context. This response serves a dual purpose: it is provided directly to the operator and acts as the input for the image retrieval process.

Figure 1. System architecture overview of the ManuAI, showing the flows of generating text information, image information, and image description.

In the image retrieval pipeline (orange line in Figure 1), the generated text information is used to search a pre-labeled image dataset. Using a ranking algorithm, the system identifies the most relevant image and its corresponding labels. If the similarity score exceeds a predefined threshold of 0.6, the image is included as the image information. However, if the score falls below this threshold, the Stable Diffusion model generates a corresponding image, which is then added to the response.

Finally, the system generates an image description [7] (blue line in Figure 1) by synthesizing descriptive text from both the text information and image information. This ensures the operator fully understands the relevance of the graphic. To ensure credibility and traceability, the system provides the specific source alongside the text information in the final response.

The completed response delivered to the operator includes up to three components:

• Text Information: The primary textual response to the operator’s query.
• Image Information: A relevant image retrieved or generated by the system.
• Image Description: A detailed textual explanation of the image.

4. Textual Assistant

The textual assistant component focuses on the textual data processing and text generation capabilities that form the foundation of the system. To achieve accurate and contextually relevant responses, the system relies on two critical datasets: one designed for fine-tuning the LLM and the other structured to enable RAG. These datasets are carefully constructed to capture both general manual formats and specific information from machine tool manuals, ensuring the system is versatile and adaptable to diverse manufacturing environments.

The fine-tuning dataset is developed to train the LLM to understand and respond to queries based on general and specialized manuals. This dataset preparation involves text extraction, preprocessing, and organization into a format compatible with training objectives. Similarly, the dataset used for RAG is built to support efficient retrieval of context-specific information, allowing the system to access and integrate relevant content dynamically during user interactions. The construction and organization of these datasets are vital for achieving both generalization and precision in the system’s responses.

In addition to the datasets, the textual assistant incorporates advanced text generation capabilities. Fine-tuning the LLM enables it to produce responses tailored to the manufacturing domain, while user-specific prompt engineering ensures that the generated outputs align with the operator’s needs and preferences. These mechanisms work in tandem to refine the quality, accuracy, and relevance of the system’s textual outputs, forming a crucial part.

4.1. Text Dataset

As mentioned in the previous section, two distinct datasets were collected: the Manual Book Format dataset for fine-tuning the LLM and the Target Manual dataset to enable the system’s RAG capabilities.

Industrial manuals are specialized documents designed for systematic troubleshooting, and thus they follow a highly standardized and formal framework. To establish the foundation of the proposed framework, manuals from four major machine tool manufacturers were analyzed [20,21,22,23]. The analysis revealed that all manuals share a highly consistent structural organization, including hierarchical components such as chapters, headings, subheadings, contextual descriptions, and page number-based segmentation. This structural consistency supports the generalizability of the proposed approach across different manufacturers. Our data preprocessing pipeline was specifically architected to leverage these structural commonalities, ensuring that the proposed method remains applicable across different vendors despite variations in specific linguistic content.

Both datasets were designed to address the identical structure of the given machine tools, utilized to improve the accuracy and contextual relevance in the generated responses. The “Manual Book Format dataset” was constructed to fine-tune the pre-trained LLM, ensuring it could generate accurate, domain-specific responses tailored to the manufacturing environment. The preprocessing pipeline handled unstructured files (PDF, Word, and TXT) by extracting text and cleaning formatting artifacts [24].

During the tokenization process, the manual content was categorized according to this standardized hierarchy. As shown in Figure 2, the largest headings representing sections or chapters were labeled as Title. Subheadings indicating specific topics were labeled as Heading, while detailed explanations or procedural steps were grouped as Contents. Each Title/Heading/Contents grouping was then encapsulated as a single Context token to form a structured dataset.

Figure 2. Preprocessing workflow and example, considering structured format of the manuals.

To enhance the model’s ability to generate question–answer pairs, general text paragraphs were transformed into Q&A data. This transformation facilitated fine-tuning of the pre-trained LLM, supporting customized prompts specifically designed for this purpose. The example prompts used in this process are shown in Figure 3, Figure 4 and Figure 5. Due to the privacy policy, anonymized sample of the used alarm dataset is given in Appendix A.

Figure 3. Prompt for generating questions.

Figure 4. Prompt for generating answers.

Figure 5. Prompt for creating question–answer dataset from alarms.

Additionally, the manual appendix, which contains detailed descriptions of machine tool alarm causes and their solutions, was processed into Q&A format using similar methods. This ensured that the dataset comprehensively covered both operational instructions and troubleshooting scenarios, making it highly suitable for fine-tuning the LLM.

The Target Manual dataset was developed to support the RAG process. While RAG inherently relies on embedding models to perform text embeddings, which typically do not require extensive preprocessing, we hypothesized that the response quality could be improved if the dataset were standardized before embedding. Specifically, relying solely on raw formats such as PDF, Word, or TXT may result in suboptimal embedding quality. Therefore, utilizing the prompts developed for the Manual Book Format dataset, the Target Manual dataset was standardized in a similar manner to ensure high-quality retrieval.

Beyond official manuals, this proposed method allows for the seamless integration of diverse text-based industrial data, such as after-service (AS) logs and historical maintenance records, further enhancing the system’s practical utility.

The details of the RAG process and the specific embedding model used will be discussed in the subsequent Section 4.2. At this stage, the standardization of the Target Manual dataset is highlighted as a key step to enhance the consistency and quality of responses generated by the system

4.2. Text Generation

The fine-tuning process of the LLM was a critical step in enabling the proposed system to generate accurate and domain-specific responses. Preparing the model for fine-tuning required preprocessing the dataset into a specific format, as described in Section 4.1.

The input data format incorporated the following structure:

• The question section was enclosed within “[INST]” and “[/INST]” tags.
• The question and its corresponding answer were combined into a single unified column for training.
• The entire dataset was wrapped with “<s>” and “</s>” tags to mark the start and end of each entry.

The dataset was split into 80% training data and 20% validation data. The split was performed randomly to prevent content bias and ensure diverse data representation during training [25]. This preprocessing ensured that the model could effectively learn patterns and context specific to manufacturing environments.

The system facilitates query–answer interactions by combining retrieval-based augmentation with generative capabilities. A custom embedding mechanism was used to represent textual data in a high dimensional space, enabling efficient retrieval of relevant context from the Target Manual dataset [26]. This retrieval process dynamically integrates additional information into the system’s responses, enhancing their accuracy and relevance.

The language model is deployed on a dedicated server, allowing for real-time interactions through a robust API interface. Operators can seamlessly query the system and receive precise, contextually appropriate answers tailored to their needs.

5. Visual Assistant

Generally, when visual information is added to linguistic information, comprehension and processing speed improve [27]. In industrial settings, where technical and specialized terms are frequently used, novice operators may struggle with textual-only assistance, potentially leading to confusion.

For example, “tool offset” refers to “the value used to compensate for the difference between the physical dimensions of a tool and the reference value set in the machine.” To correct compensating for the tool offset requires measuring the difference between the set value and the reference point, which can sometimes be ambiguous. However, this issue can be easily resolved when images are provided alongside the text.

Therefore, official instructions from equipment manufacturers usually include visual aids along with textual information to enhance comprehension.

Unlike existing DIAs, the proposed system provides image-based information to complement textual responses, offering more intuitive and comprehensible assistance to the users. This feature ensures users receive visual context that enhances their understanding of machine tool operations and troubleshooting scenarios.

To build this capability, an image dataset was constructed by extracting and labeling images from official machine tool instructions. And the system utilizes text information generated by LLM Module 1 to query the relevant image from the dataset. Additionally, when no relevant image is available, the system uses text information as a prompt to generate the necessary images.

The following context explains the key elements required to provide the visual assistant: image dataset, image retrieval, and image generation.

5.1. Image Dataset

The image dataset is based on official instructions from machine vendors [28].

The dataset construction followed a general sequence of data extraction, preprocessing, and labeling. To provide images that support the textual assistance generated in the earlier stages, images from the official instructions were extracted as target data.

The image extraction process was conducted using Python (3.9) libraries. Specifically, the Y-coordinate of the paragraph immediately above the image was set as the image’s top boundary, and the Y-coordinate of the paragraph immediately below the image was set as its bottom boundary. This ensured that each image was accurately segmented and stored in a designated directory. Connected images were extracted as a single image, as shown in Figure 6, and stored along with the corresponding page.

Figure 6. Examples of the constructed image dataset.

Once the images were extracted, they underwent a labeling process to provide structured metadata. Official instructions include captions accompanying the images. Inspired by this, the proposed image dataset uses captions to label images, along with chapter information. Chapter information was calculated based on the page from which the image was extracted. As shown in Figure 2, by recognizing the table of contents in the manual, it was determined which chapter each page belonged to.

The labeling followed a specific format: Image File Name, Chapter Information, and Caption. Figure 6 shows an example of how the image data is stored in the directory:

• Image File Name: File path of saved image, assigned based on the page number and figure sequence (ex) “page5_figure1”).
• Chapter Information: Name of the chapter where the image was extracted (ex) “6_MANUAL GUIDE_i_SIMULATOR_FOR_THE_PERSONAL_COMPUTER“.
• Caption: Image caption, text located closest to the image, typically within the nearest text box.

This structured labeling ensures that the image dataset is well-organized and easily accessible for further processing in the system, particularly for image retrieval and generation tasks. Figure 6 illustrates an example of the image labeling results. Furthermore, the image dataset is inherently expandable to include real-world photographs with expert-annotations. Integrating such additional data types would further enhance the system’s visual reliability and provide a practical alternative to purely generated images in complex maintenance scenarios.

5.2. Image Retrieval

The image retrieval functionality, as shown in the architecture in Figure 1, utilizes the text information generated by LLM Module 1 and the image dataset. This feature aims to identify the image most relevant to the text information and thereby augment the textual responses with images, making the answers more comprehensible.

At this point, a ranking algorithm is used to assess the relevance between the text information and each image in the dataset. Ranking algorithms, commonly used in search engines, are designed to identify the document most similar to a given query among multiple candidates [29].

In this paper, the queries utilized for the ranking algorithm were the text information generated by LLM Module 1, while the comparative documents were the captions of the image dataset. In other words, the proposed system identifies the caption most relevant to the previously generated text information.

Then, it evaluates again the similarity between the top ranking caption and the text information. The similarity check is performed using the Sentence-BERT model, which measures cosine similarity between two sentences [26]. In essence, it evaluates the similarity between the text information and the selected caption, assigning a similarity score.

Ultimately, if the similarity score exceeds a predefined threshold, the image corresponding to the selected caption is adopted as the image information.

In this study, the similarity threshold was set at 0.6. This value was determined through experimentation, since setting the threshold higher than 0.6 significantly reduced the image provision rate.

Figure 7 illustrates an example of the image retrieval process. For a user query related to “tool installation direction,” LLM Module 1 generates a textual response, which is then used to rank the captions in the image dataset. The selected caption’s similarity to the text information is evaluated, resulting in a score of 0.651. Since this score exceeds the threshold of 0.6, the corresponding image is adopted as image information.

Figure 7. The process of image retrieval.

Additionally, the selected caption corresponding to the image information is passed to LLM Module 2 along with the text information as input. LLM Module 2 generates a new image description based on the text information and image caption, which is then provided as part of the response. LLM Module 2 utilizes the same model as LLM Module 1 in the textual assistant.

As a result, the final response includes the most relevant image alongside the newly generated caption based on the image.

5.3. Image Generation

This paper leverages high-quality images from official instructions to make the textual assistant more comprehensible. However, not all content in the official instructions includes corresponding images.

In the previous image retrieval chapter, cases where the similarity score does not exceed 0.6 are defined as instances where there are no images in the dataset related to the textual assistant. To better support operators in such scenarios, an image generation function was developed.

The image generation function creates illustrations to supplement the textual assistant. This is especially useful when official manuals lack the necessary visual aids to enhance operator understanding.

A diffusion model, which repeatedly adds and removes noise to learn image features, is employed as the basis for this image generation [30]. Recent diffusion model-based approaches have shown significant success in generating high-quality images from text prompts, and various methods to simplify their usage are being actively explored.

The image generation functionality in this paper utilizes a diffusion model pipeline defined to generate images that support the textual assistant.

In cases where no image information is selected during the image retrieval phase, the proposed system provides only the textual assistant. Novice operators may find it challenging to rely solely on text-based responses and may request visual aids.

The text information generated by LLM Module 1 is passed to the diffusion module and used as a text prompt. The diffusion module generates an illustration that describes the text information, which is then utilized as image information and provided to the operator as part of the final response.

Figure 8 illustrates an example of the image generation process. For a user query related to “tool offset,” LLM Module 1 generates a textual response, which is used to rank captions in the image dataset. The highest similarity score of 0.181 does not exceed the threshold of 0.6, so the textual response is passed to a diffusion model as a prompt to generate a new image. The generated image is then provided to the user as image information.

Figure 8. The process of image generation, when no relevant images are in the dataset (similarity score < 0.6).

Through this feature, operators are always provided with both textual and visual assistants, significantly improving their understanding and operational efficiency.

6. Results of Prototype Implementation

This section describes the technical integration and deployment of the proposed system. This includes setting up the development environment, deploying the fine-tuned LLM and related modules, and integrating various components such as RAG and the diffusion module. The section also outlines the system’s workflow and the challenges addressed during development. By detailing these processes, this section highlights how the proposed system was constructed to provide seamless and efficient support for manufacturing environments.

6.1. System Overview and Setup

The proposed system integrates multiple components to provide seamless support for manufacturing operations through text and image processing. The core system consists of the following elements:

• LLM: Handles all user interactions by generating contextually relevant responses.
• Fine-Tuning: Customizes the LLM using the Manual Book Format dataset for domain-specific accuracy.
• RAG: Dynamically retrieves relevant information using the Target Manual dataset.
• Image Retrieval: Utilizes BM25 ranking [29] and cosine similarity to find the most relevant images based on text inputs.
• Diffusion Module: Built on the generative AI [31] platform, it enables users to refine or generate new images.
• API Framework: Combines the web servicer platform [32,33], to connect the LLM server with other modules.
• History APP: Developed using the dashboard platform [34] to visualize and manage a history of user queries and responses.

Development Environment

The system was developed on a high-performance local server environment to handle computationally intensive tasks such as model fine-tuning and image generation. The development environment includes the following:

• Hardware: AMD Ryzen 9 7950X3D 16-Core CPU (Advanced Micro Devices, Santa Clara, CA, USA) and NVIDIA GeForce RTX 4090 GPU (NVIDIA Corporation, Santa Clara, CA, USA).
• Operating System: Windows 11.
• Programming Language: Python (3.9).
• Libraries and Tools: LangChain [35], PyTorch [36], HuggingFace [37], SQLite [38], Node-RED [34], ComfyUI [31], and Transformers, among others.

Server and Data Storage

The system is deployed on a local server, combining Flask and FastAPI frameworks for efficient API communication [33]. Data is organized into separate directories for ease of access and management:

• Manual Book Format: Text data for fine-tuning the LLM.
• Target Manual (Text): Text data used for the RAG process.
• Target Manual (Image): Image data used for image retrieval and generation.
• History: Interaction logs stored for query–response visualization.

For real-time visualization in the History APP, the system uses an SQLite database connected to Node-RED. This setup ensures that user interactions are logged and can be reviewed efficiently.

6.2. Model Deployment and Integration

The proposed system integrates fine-tuned LLMs, RAG processes, image retrieval, and generation modules to provide comprehensive and contextually relevant responses. The workflow connects these components seamlessly, ensuring accurate and efficient interactions between the user and the system. Additionally, prompt engineering techniques were applied to improve the accuracy and relevance of the generated outputs. While alternative models (e.g., other open-source LLMs or image generation models) could be considered, adopting different models would require additional evaluation and prompt re-optimization to achieve comparable performance.

LLM Deployment and RAG Integration.

The Llama3 model was selected as the language model due to its wide availability as an open-source LLM that can be deployed locally, which aligns with the system requirements in the development stage.

The fine-tuned Llama3 model operates on a local server, utilizing Flask and FastAPI frameworks for API-based communication. When a user submits a query, it is processed by the LLM, which generates a textual response. To enhance the relevance of the generated answers, a RAG process is implemented using LangChain [35]. The RAG module employs the nomic-embed-text-v1 [39] embedding model to perform text embedding and retrieval from the Target Manual dataset.

To validate the generated response, the system conducts a similarity evaluation with an exemplary answer using the Sentence-BERT model. This ensures that the textual response aligns with domain-specific knowledge and user expectations. The result of the evaluation, along with the generated response, forms the text information of the final output.

Image Retrieval and Diffusion Module

The generated textual response undergoes further processing in the image retrieval module. The response is compared with labels from the Image Dataset using the BM25 ranking algorithm. A similarity evaluation is then performed between the top-ranked image label and the textual response.

For image generation, Stable Diffusion was adopted as it demonstrated the most suitable performance for generating machine-related components among the evaluated models.

• Similarity Threshold: If the similarity score is 0.5 or higher, the corresponding image is deemed relevant and included in the response.
• Below Threshold: If the score is below 0.6, the image is discarded as irrelevant, and the system generates a new image using the diffusion model. The diffusion module, built on the ComfyUI platform, creates custom images based on the textual response as a prompt.

The final response provided to the user includes text information (the generated textual answer with the source path), image information (a relevant or newly generated image), and image description (a description synthesizing the image label and textual content).

Data Storage and Visualization

All query–response interactions are stored in real time in an SQLite database, ensuring traceability and enabling users to review their interaction history. The History APP, developed using Node-RED, retrieves this data and visualizes it in a tabular format, allowing users to efficiently access their records. The Node-RED configuration, illustrated in Figure 9, connects the SQLite database with a dynamic dashboard interface. This setup ensures that new entries are visualized instantly, providing users with an intuitive real-time view of their interaction history.

Figure 9. Real-time database saving using Node-RED.

User interface

The finalized system was prototyped as a mobile application using App Inventor [40] and FastAPI. This prototype allows users to interact seamlessly with the proposed system through a user-friendly interface on their mobile devices. By integrating the server-side functionality of FastAPI with the visual development tools of MIT App Inventor, the system ensures real-time query–response interactions and data visualization. Figure 10 and Figure 11 illustrates the prototype’s interface, showcasing its ability to retrieve, process, and display interaction history efficiently.

Figure 10. User interface of ManuAI Smart phone App.

Figure 11. HMI app of Machine tools Supported by LLM Edge device.

6.3. Case Study on Similarity Evaluation

To further validate the effectiveness of the proposed system, we conducted a case study using sentence-level semantic similarity analysis. Specifically, we employed the Sentence-BERT model to measure the cosine similarity between the system-generated answers and the exemplary reference answers provided by the machine tool vendors.

A similarity threshold of 0.5 was adopted. Although cosine similarity reflects vector closeness in semantic space, a high similarity score does not always guarantee semantic correctness. For example, two sentences may appear almost identical in vector space, but a minor change such as replacing “yes” with “no” could reverse the meaning entirely. Moreover, since LLM-generated responses are not always verbatim matches to reference answers, it was necessary to empirically determine a threshold at which the generated responses could still be regarded as semantically acceptable. Based on pilot testing, a value of 0.5 was found to be appropriate for distinguishing semantically aligned answers from irrelevant ones. This threshold balances robustness against minor linguistic variations while avoiding overly lenient evaluation.

The evaluation was conducted on two datasets: an NC Alarm List comprising 224 alarm cases and a PLC Alarm List comprising 24 alarm cases. Among a total of 248 test queries, 204 answers achieved similarity scores greater than 0.5, yielding an overall success rate of 82.26%. Table 1 summarizes the detailed evaluation results.

Table 1. Similarity evaluation results.

Dataset	Total Cases	Successful Cases (>0.5)	Success Rate (%)
NC Alarm List	224	183	81.70
PLC Alarm List	24	21	87.50
Total	248	204	82.26

To ensure the technical reliability of the evaluation, the two mentioned datasets were pre-validated and refined by domain experts from the machine tool manufacturer.

A qualitative review of unsuccessful cases identified revealed three primary failures. Contextual ambiguities in the original manuals occasionally led to incomplete RAG retrieval, resulting in less grounded answers. Despite the use of RAG, the system still exhibited intrinsic hallucinations, producing technically plausible but incorrect maintenance steps.

6.4. Real-Time Parameter-Based QnA Integration

To enhance the ability of the proposed system to respond dynamically to changing operational conditions, we implemented a real-time query/answering module that utilizes live machine tool parameters. This addition enables the system to provide context-sensitive responses based on the current state of the equipment, improving both precision and relevance.

6.4.1. Real-Time QnA Workflow

The real-time QnA pipeline is designed to operate entirely within a local intranet, removing the need for external internet or cloud infrastructure.

• Data Acquisition: Machine parameters such as spindle speed, temperature, vibration, and alarm codes are collected from PLC-connected sensors and controllers.
• MQTT Communication: A lightweight MQTT broker (e.g., Mosquitto) is used to publish and subscribe to parameter updates over LAN. No cloud or external Ethernet connection is required.
• Prompt Generation: Parameter updates trigger prompt templates within Node-RED, generating natural language queries such as the following:

“The spindle temperature is currently 85 °C, which exceeds the normal operating range. What actions should be taken to prevent damage?”

• Language Model Interaction: The prompt is sent to the locally hosted LLM server via a FastAPI interface. The model generates a domain-specific, context-aware response.
• Logging and Visualization: Queries and responses are stored in a local SQLite database. Node-RED retrieves and displays this data in a dashboard for operator review.

6.4.2. Operational Benefits

This module brings several operational benefits:

• LAN-based operation: Entirely functional in closed networks, which enhances security and reliability.
• Real-time reactivity: Prompt generation and model response occur within seconds of receiving new parameter data.
• Scalable architecture: MQTT topics can be configured for multiple machines and data types, allowing for easy expansion.
• Low implementation cost: Built on open-source tools (Mosquitto, Node-RED, and SQLite), the system is cost-effective and customizable.

6.4.3. Validation

We validated the system using machine tools equipped with live parameter monitoring. Real-time parameters were successfully captured, encoded into prompts, and used to generate reliable, context-aware responses. In both normal operation and alarm scenarios, the system exhibited accurate and relevant behavior.

To quantify the system’s reactivity, we measured the end-to-end inference time across 248 test cases (NC and PLC alarms combined). As shown in Table 2, the system achieved an average response time of 4.83 s using a local edge device equipped with an RTX 4090 GPU and 6.20 s under a CPU-only configuration. This latency includes query processing, RAG-based retrieval, and visual synthesis. The CPU-based evaluation reflects a more realistic deployment scenario in industrial environments where high-end GPUs may not be available.

Table 2. Inference performance and response time evaluation.

Number of Tests	Hardware Environment	Avg. Response Time (s)
248	NVIDIA GeForce RTX 4090	4.83
248	CPU only	6.20

7. Discussion

This section discusses the limitations of the proposed system and potential areas for improvement, focusing on image generation, system feasibility, and the accuracy of the textual assistant.

While the proposed system presents a robust framework for supporting manufacturing operations, there are clear areas where improvements are necessary. Addressing the challenges in image generation, ensuring the system’s adaptability to real-world complexities, and enhancing the accuracy of the textual assistant will be critical steps for future iterations. These enhancements will not only solidify the system’s utility but also pave the way for broader adoption in industrial settings.

7.1. Challenges in Image Generation

While the diffusion model-based image generation feature significantly enhances the system by providing visual support when no relevant images are found in the dataset, its accuracy and practicality remain a concern. Generated images, though helpful in certain scenarios, often lack the precision and reliability. This limitation stems from the inherent challenge of generating an accurate visual representation based solely on a single text prompt.

To address this issue, the system displays a clear disclaimer, “This image is generated via AI,” whenever a generated image is given as a response. Future work could explore multimodal prompt engineering, where additional contextual information, such as related text blocks or multiple descriptive inputs, is included to guide image generation. Another potential solution is to fine-tune the diffusion model with domain-specific images to improve its relevance and utility for industrial applications. Incorporating user feedback loops to iteratively refine generated images could also enhance their effectiveness.

7.2. Feasibility in Real-World Applications

The current study demonstrates promising results within the constraints of a controlled dataset and a specific set of machine tools. However, extending this system to handle the diverse and complex environments of real-world industrial settings poses significant challenges. Many industrial facilities operate with heterogeneous machine types and varying operational conditions, which may require extensive customization and scalability beyond the scope of this research.

Future studies should aim to validate the system in broader industrial environments, collaborating with various manufacturers to create standardized datasets and models. Developing modular components that can adapt to different machine configurations and operational contexts will also be crucial for achieving practical feasibility. And, a feedback-driven QnA pipeline is being developed to allow operators to regenerate, expand, or compress answers in real time. This human-in-the-loop mechanism ensures that responses meet specific contextual and credibility requirements.

In addition, confidentiality risks are mitigated by our system’s design choice of relying exclusively on proprietary datasets and fully localized deployment. All modules, including fine-tuned LLMs, retrieval pipelines, and visualization components, operate on a dedicated local server within a closed LAN environment. This ensures that sensitive industrial data never leaves the local network, thereby eliminating dependency on external cloud infrastructure and providing a robust level of data security suitable for industrial applications.

7.3. Limitations in Textual Assistant Accuracy

Although the incorporation of RAG has reduced hallucination issues commonly observed in LLMs, the system still falls short of delivering near perfect accuracy. This limitation is intrinsic to LLMs, which may occasionally produce responses that are incorrect or lack domain-specific relevance.

As LLM researchers, it is imperative to continue investigating methods to mitigate these inaccuracies. Advanced techniques, such as fine-tuning with larger and more diverse domain-specific datasets, incorporating external knowledge bases, and real-time validation of outputs against trusted sources, could improve accuracy. Additionally, integrating continuous learning mechanisms to adapt to user interactions and evolving industrial requirements could further enhance the system’s performance.

To mitigate the issue of hallucinations, the system relies on proprietary datasets extracted from official machine manuals and incorporates RAG to validate outputs. This integration reduces the dependence on the model’s pre-trained knowledge alone, ensuring domain-grounded and factually consistent responses.

In addition, a case study using Sentence-BERT-based semantic similarity evaluation was conducted on two datasets (NC Alarm List with 224 cases and PLC Alarm List with 24 cases). The system achieved an overall success rate of 82.26% at a similarity threshold of 0.5, which demonstrates the robustness of the proposed approach in aligning generated answers with vendor-provided references. The alarm datasets were pre-validated and refined by domain experts from the collaborating machine tool manufacturer. While semantic similarity serves as a useful baseline for evaluation, more sophisticated methodologies for assessing technical accuracy is currently being conducted as part of follow-up studies.

7.4. Summary

While prior studies have proposed digital assistants for industrial domains, the proposed system presents a distinct advancement in several dimensions that are not concurrently addressed in existing research.

First, the proposed system is designed as a domain-specific assistant tailored explicitly for machine tool operators. Unlike generic chatbots or rule-based systems, our approach leverages fine-tuned large language models trained on actual machine manuals and alarm data, resulting in more context-aware and operator-relevant guidance.

Also, Manthe-proposed systemuAI incorporates real-time machine tool parameters, such as spindle speed, temperature, and vibration, into its query generation and response logic. This integration enables the system to generate dynamic, situation-aware answers that reflect the actual state of the equipment. Such real-time contextualization has rarely been addressed in prior DIA systems for manufacturing.

Our system is fully operational within a local, offline environment. The system operates entirely offline to ensure data privacy. By eliminating cloud-based processing, it achieves network independence. This makes it suitable for security-sensitive industrial sites with restricted external connectivity.

Taken together, these features distinguish the proposed system from earlier approaches that often rely on static knowledge bases, lack real-time adaptability, or depend on external servers. This architecture combines domain specialization, real-time parameter integration, and secure local deployment. Together, these features represent a significant step forward for intelligent assistants in the manufacturing sector.

8. Conclusions

This paper presents the proposed system, a digital assistant designed to address the challenges faced in the machine tool industry, including the lack of expert operators and the complexity of diverse interfaces. By integrating fine-tuning, retrieval-augmented generation, and diffusion-based image generation, the system provides both textual and visual support to enhance operators’ efficiency and understanding.

From the case study, the proposed system demonstrates 82.26% accuracy of answering trouble shootings against the 248 alarms of commercial machine tools by combining LLMs with domain-specific datasets and advanced image generation techniques to create a robust assistant framework. The integration for real-time interaction logging and visualization further highlights the system’s practicality in real-world applications.

Despite its contributions, this research acknowledges certain limitations. The feasibility of applying the system to heterogeneous industrial environments remains a challenge, and the accuracy of textual responses still requires improvement to ensure near perfect reliability. Future studies should focus on expanding the system’s applicability to diverse machine types, refining image generation processes, and enhancing LLM accuracy through continuous learning mechanisms.

In conclusion, the proposed system provides a significant step forward in leveraging AI to support complex industrial tasks, paving the way for smarter, more efficient manufacturing processes.