Content and AI-Delivery Analytics of Semantically Enriched Content in Engine Manufacturing ^†

Abstract

Standard Retrieval–Augmented Generation (RAF) often fails in safety-critical manufacturing and reveals the inherent problem of additional and inappropriate retrieved context topics. In engine manufacturing, where topic-based documentation is widely variant-driven, semantic similarity no longer correlates with technical relevance, assembly procedures for engine variants are linguistically often almost identical yet operationally incorrect. Recent RAG systems, relying on the LLM’s ability to discern this context noise during the generation phase, might fail to account for this extreme content-wise overlap. This paper proposes a dual-optimization strategy: ensemble sizing and semantic level enhancement. Instead of relying solely on vector similarity, we partition the retrieval space into sub-ensembles and apply a metadata scoring function. Our findings demonstrate that this hybrid approach might help to transform RAG-based systems into more variant-appropriate delivery systems essential for safety-critical environments.

1. Introduction

The integration of Large Language Models (LLMs) into technical communication has recently changed how complex product information is accessed and utilized. While these AI-driven systems offer efficient, dialog-based retrieval for general documentation, their application in highly specialized industrial context reveals significant limitation.

Specifically, engine manufacturing documentation presents a critical challenge for Retrieval-Augmented Generation (RAG) systems operating in safety-critical environments. Unlike standard RAG processes for AI-based delivery, where semantic similarity typically correlates with relevance, engine documentation is highly variant-driven, leading to a manifold of similar topics in the framework of topic-based documentation [1]. A single engine platform may encompass dozens of configurations, each with distinct torque specifications, assembly sequences, and regulatory compliance requirements. Documents describing these variants are often semantically nearly identical—differing only in specific numerical values, conditional procedural branches, or regulatory annotations—yet these differences carry safety-critical implications.

Recent RAG systems, which rely primarily on vector similarity to select relevant context, systematically fail in this environment because high cosine similarity does not guarantee contextual correctness. A procedure retrieved from an adjacent engine variant or outdated regulatory edition may be highly semantically similar yet operationally incorrect or legally non-compliant. This phenomenon occurs when semantically proximate but contextually incompatible content enters the RAG context, potentially degrading answer quality and, in the research project domain, introducing potential safety risks.

2. Project Goals

The research work addresses the challenge of ensuring variant correctness and content integrity when applying RAG in semi-large data, i.e., content corpora from technical documentation. The objective is to analyze how semantically enriched but variant-dense content behaves under similarity-based retrieval, to understand how the amount of retrieved context topics is influenced [2], and to identify possible mechanisms that can prevent contextually incorrect topics from entering the RAG workflow. The overarching aim is to characterize the retrieval behavior of variant-dense technical content within RAG processes to determine the conditions under which more precise and variant-appropriate information delivery can be ensured in safety-critical engine manufacturing environments.

3. Methodology

We present AI-based delivery analytics, a framework called the PIAI!-Lab [2], designed to study variant behavior and content integrity in semi-large, heterogeneous engineering content corpora (topic ensembles) [2]. The approach introduces three components.

3.1. Ensemble Sizing Method

The necessity of a granular, multi-tiered approach to Retrieval-Augmented Generation (RAG) architectures is corroborated by recent advancements such as the HIRAG framework (Hierarchical-Thought Instruction-Tuning) [3]. While HIRAG focusses on enhancing the model’s internal capabilities for filtering and reasoning through instruction turning, our concept of ensemble sizing addresses these challenges at the microscopic topic level of the content corpus.

Unlike model-centric approaches that rely on the LLM’s ability to discern noise during the generation phase, ensemble sizing mitigates additional and inappropriate context topics a priori.

In the specific context of engine manufacturing, the search space is partitioned into three hierarchical tiers of sub-ensembles. This pre-emptive narrowing of the retrieval window serves as a strategic filter to ensure that the retrieved context is both technically accurate and contextually relevant.

3.1.1. Three Levels of Ensemble Sizing

To map the complexities of engine production data, we defined three distinct ensemble sizes by extrinsic product and information metadata (Figure 1):

• Size 1 (Global): This represents the maximum search window, encompassing the total volume of all extrinsic data products. It includes all modules and document types across all engine models within the organization.
• Size 2 (Product-Specific): The retrieval window for this sub-ensemble is restricted to a single engine series or model type. This is achieved by filtering the corpus based on product-extrinsic metadata, which allows the system to isolate the relevant asset class before the vector retrieval begins. While this narrowed scope includes all associated document types (e.g., maintenance manuals, assembly instructions), it effectively excludes data from unrelated engine architectures, thereby preventing cross-model interference.
• Size 3 (Information-Specific): This is, in our case, the most granular level of sub-ensembles, where the retrieval is limited to a single specific document type for a specific engine model. This high-precision refinement is based on information-extrinsic metadata, which allows the system to filter by specific document types, such as repair manuals, user manuals, etc.

Figure 1. Ensemble sizing 1–3.

Figure 1. Ensemble sizing 1–3.

3.1.2. Mitigation of Semantic Noise?

The primary objective of ensemble sizing is the reduction in similarity width. In technical domains like engine manufacturing, many documents share a high degree of terminological overlap, for example, assembly instructions for different cylinder head variants may appear nearly identical to a vector-based search.

Under a global ensemble search approach (Size 1), there is a significant risk that the cosine similarity metric identifies topics as relevant that are linguistically similar but technically incorrect for the specific request. By narrowing the search window a priori, we prevent these “false-positive” modules from ever entering the comparison process. This preemptive filtering should ensure that the RAG pipeline is not diluted by irrelevant or wrong context, increasing the reliability of the generated output.

3.2. Semantic Levels Method

While ensemble sizing acts as a macro-level strategy to restrict the search corpus a priori, Semantic Level Enhancement are intended to serve as a micro-level optimization of the vector space itself. Even within a focused search window (e.g., Size 3), topics within technical documents still exhibit high terminological overlap. To further refine retrieval precision, we expand the standard RAG process by enriching the similarity measurements through the inclusion of semantics.

3.2.1. Three Semantic Levels

Traditional RAG systems typically vectorize only the raw body text of a document, resp. topic, which often leads to a poor semantic understanding of domain-specific terminologies. In variant-dense environments like engine manufacturing, this results in a phenomenon where standard embedding models cannot distinguish between highly similar topics, leading to the retrieval of weakly related documents that share a high perceived semantic similarity but are operationally incorrect [4]. We counter this by implementing three distinct levels of data enrichment:

• Semantic Level 1 (Content and Topic Title): At this stage, the topic title is integrated with the topic body text during vectorization.
• Semantic Level 2 (Structural XML-Tagging): This level leverages the inherent structure of technical documentation. By injecting XML tags (e.g., <warning>, <technical_specification>, <assembly_step>) into the vectorization process, we add functional intent to the vector itself. The model no longer just calculates linguistic similarity; it distinguishes between a description of a part and a safety instruction regarding that part.
• Semantic Level 3 (Intrinsic Metadata Integration): The most granular level involves the inclusion of intrinsic product and information metadata by adding them to the payload along the vector. This creates a unique “semantic fingerprint” for each topic, ensuring that even if two topics are linguistically close to identical, they can still be differentiated through their metadata.

3.2.2. Semantic Level 3 Calculation and Weigthing

The distinctive feature of Semantic Level 3 is the augmentation of standard cosine similarity with a metadata scoring function. This hybrid approach was used to address a specific limitation of vector-based retrieval in the domain of technical documentation.

In an engine manufacturing context, components such as an ‘Oil pump’ and an ‘Oil filter’ share a high degree of linguistic and thematic overlap (both belong to the lubrication system). Consequently, their vectors are positioned closely in the vector space. However, in physical reality, these components are distinct entities located at different positions within the engine architecture. Relying solely on the further vectorization of metadata would fail to separate these topics effectively, as the RAG process may view both as contextually relevant even if they are not.

To enforce a clear distinction, a simple discrete scoring has been implemented, the step-function metadata (x,y). This function evaluates the metadata of two objects: product-intrinsic metadata p and information-intrinsic metadata i. This scoring function is initialized at a maximum value 1.0 and undergoes a linear penalty for each mismatching metadata, resulting in three possible cases when comparing two topics, x and y:

1. Full-match: metadata(x,y) = 1.0 if both metadata sets are identical, i.e., p(x) = p(y) and i(x) = i(y).

2. Partial-match: metadata(x,y) = 0.5 if only p or only i metadata-type values are identical.

3. No-match: metadata(x,y) = 0 if no metadata values are identical.

The final mixed Semantic Level 3 cosine similarity score is calculated as a weighted combination of the standard cosine similarity and the output of metadata (x,y). For the purpose of showing the basic effect, a balanced 50/50 ratio is applied, so that the metadata has equal influence compared to the linguistic similarity. In business applications, this ratio should be evaluated and can be adjusted at lower values.

$$CosSim\_semanticlvl3(x,y)=(0.5 \times \mathrm{CosSim}(\mathrm{x},\mathrm{y}\left)\right)+(0.5 \times \mathrm{metadata}(\mathrm{x},\mathrm{y}\left)\right)$$

(1)

3.3. Metrics

To evaluate the semantic structure and retrieval performance within the vector space, we employ the following three metrics:

• Cosine Similarity (Cos Sim) measures the cosine of the angle between two vectors to determine their semantic proximity. It serves as the fundamental metric for our retrieval operations. Our RAG processes are executed with a similarity threshold of 0.85 to ensure high relevance of the retrieved context [2,5].
• Similarity Width (Sim Width) describes the quantitative extent of thematic similarity within the vector space. Sim Width measures how many topics are similar with respect to a threshold, serving as an indicator for the density of potentially ambiguous candidates in the retrieval set [4].
• Mean Similarity Difference (MSD) is the averaged cosine similarity between topic embeddings to quantify the semantic divergence between distinct topics [2,5].

4. Findings

4.1. Ensemble Sizing

The initial evaluation focused on the impact of ensemble sizing on retrieval precision, specifically measuring its influence on the Similarity Width.

Under the ensemble size 1 (Global) configuration, the retrieval system operated across a corpus containing 684 topics separated by colored lines as seen in Figure 2. To evaluate the effectiveness of the vector-based comparison, a cosine topic similarity threshold of 0.85 was established. Within this framework, any document module exceeding this threshold is supposed to be identified as a relevant match and passed to the RAG pipeline for response generation, as seen in Figure 3.

Figure 2. Global cosine similarity.

Figure 3. Similarity width with a table on the side.

The table on the right side of Figure 3 shows a summary of the similarity width graph on its left side. The column “Bycatch” shows how many additional and inappropriate context topics would be retrieved during the RAG process of finding the target topic. The table on the right side of Figure 3 shows a summary of the similarity width graph on its left side. The column “Bycatch” shows how many additional and inappropriate context topics would be retrieved during the RAG process of finding the target topic, with 0 meaning that the target topic has no similar neighbors. The total number of retrieved topics is calculated by summing the products of each bycatch count and its corresponding frequency. For instance, if a bycatch of 18 occurs twice, it contributes 36 units to the aggregate. This total sum (1868) is then divided by the number of compared topics (684), resulting in a mean SimWidth of 2.73 at the specified threshold (0.85).

The findings indicate that in a global search environment, the high number of distinct topics leads to a significant “crowding” of the vector space. With 684 topics being compared simultaneously, many other modules reached or exceeded the 0.85 threshold.

To validate the impact of the method ensemble sizing, the retrieval performance was compared across all three ensemble sizes. The empirical data, visualized in the cosine similarity graphs Figure 4 and similarity width tables Figure 5, reveals two primary effects:

• Reduction in Topic Density: As the ensemble size decreases from Size 1 to Size 3, the density of candidates exceeding the 0.85 threshold is drastically reduced. In Size 1, the high number of overlapping topics creates a “crowded” retrieval window. By transitioning to Size 3, the vast majority of these “distractor” topics are eliminated a priori. This results in a significantly cleaner retrieval set, where only contextually relevant topics remain.
• Lower Similarity Scores: A critical observation is the downward shift in similarity scores for the remaining topics in smaller ensembles, as seen in Figure 4. While in the global ensemble many topics clustered above the 0.85 threshold, the more granular focus of Size 2 and Size 3 tends to “push” irrelevant or tangentially related topics below this critical limit.

Figure 4. Cosine similarity graph, ensemble size 1–3 from top to bottom. (a) N = 684, (b) N = 242, (c) N = 77.

Figure 4. Cosine similarity graph, ensemble size 1–3 from top to bottom. (a) N = 684, (b) N = 242, (c) N = 77.

Figure 5. Similarity width compared in tables, ensemble size 1–3 from top to bottom. (a) N = 684, (b) N = 240, (c) N = 77. The resulting mean SimWidth is reduced accordingly.

Figure 5. Similarity width compared in tables, ensemble size 1–3 from top to bottom. (a) N = 684, (b) N = 240, (c) N = 77. The resulting mean SimWidth is reduced accordingly.

The empirical results demonstrate that ensemble sizing is a highly effective mechanism for enhancing retrieval precision, reducing similarity width and increasing the mean similarity difference (MSD) [6].

4.2. Semantic Levels

As shown in the cosine similarity graph (Figure 6), progressing from Semantic Levels 1 to 3 exerts significant downward pressure on the similarity scores of irrelevant candidates. While the graphs on levels 1 and 2 look quite similar, Semantic Level 3 creates a visible hole in the similarity graph. This is a direct result of the metadata scoring, which effectively pushes topics with mismatching metadata toward a score of zero and while maintaining higher values for topics with metadata matches.

Figure 6. Cosine similarity graph, Semantic Levels 1–3 from top to bottom. (a) level 1, (b) level 2, (c) level 3.

The similarity widths in Figure 7 statistically validate this trend, showing a consistent reduction in the similarity width and their resulting average as the semantic level increases. This indicates a more homogenous and relevant result set, as semantically proximate but possibly contextually false topics are eliminated.

Figure 7. Similarity width in table, Semantic Levels 1–3 from left to right.

Furthermore, the Mean Similarity Difference (MSD) analysis in Figure 8 confirms that Semantic Level 3 achieves the highest degree of unique distinctness. The MSD values for Semantic Level 3 remain significantly higher than those of the lower levels. This isolation confirms that the logic provides the necessary contrast to distinguish between technically similar but functionally different topics. Figure 8 also shows that Semantic Level 2 barely improves the distinctness of the topic, which can be confirmed in Figure 7, where similarity width lightly improves to that in the Semantic Level 1 table.

Figure 8. Similarity width in graph by engine type, Semantic Levels 1–3 from (1 = red, 2 = blue, 3 = green).

4.3. Combination

The final analysis evaluates the simultaneous application of both ensemble sizing and semantic level enrichment. The empirical data reveals a powerful synergy between these two methods, resulting in a retrieval environment with nearly non-existent similarity widths.

As visualized in Figure 9, the combination of a focused retrieval window (ensemble size 3) and metadata scoring (Semantic Level 3) effectively reduces similarity width to a maximum of 3 and, at best, even 0, which is significantly better than the results in the beginning, shown in Figure 3.

Figure 9. Similarity widths in table, Semantic Level 3 and ensemble size 3 combined.

5. Conclusions and Outlook

The results of this study confirm that the combination of ensemble sizing and semantic levels significantly optimizes retrieval performance in engine manufacturing domains. By transitioning from a global search space to a focused ensemble, and augmenting vector similarity with metadata scoring, the similarity width was strongly reduced. This creates a high-contrast retrieval environment where the target topic is mathematically isolated from irrelevant context, providing a foundation for more reliable RAG outputs in engine manufacturing.

Building on these findings, future research could explore alternative vectorization processes, as suggested in the related literature [7], to further refine Semantic Level 2. It remains to be investigated whether different vectorization models can improve the second semantic level, if it can also expand the third level to higher degrees of semantics [2].

Furthermore, the transition from static topic-to-topic comparisons to dynamic, user-driven interactions presents a significant area for future exploration. The primary challenge lies in how a user can leverage this framework through a simple natural language query to receive a precise and technically correct answer.

In conclusion, this investigation confirms that the integration of extrinsic and intrinsic metadata, starting at the source within the technical documentation and topic-based content, serves as a vital foundation for retrieval optimization. The empirical evidence demonstrates that when these extrinsic and intrinsic metadata are preserved and leveraged within the RAG context, they have a profound impact on reducing similarity width. By transforming a dense vector space into a high-contrast environment, this approach might bridge the gap between structured content management and AI-driven information retrieval in complex technical domains.