Semantic Information Architectures and Topic Ensemble Properties in AI Delivery of Product Information in the Domain of Water Treatment ^†

Abstract

This paper examines how semantic information architectures within technical communication shape topic ensemble properties being relevant for retrieval-augmented generation (RAG) systems. They allow for investigating the AI-readiness of technical content for AI-based delivery in the present domain of water treatment. The analysis characterizes the dependence of AI-driven delivery on similarity-based metrics and metadata completeness. Topic content management is operationalized within the PI-Class framework by analyzing topics with respect to content variants and versioning, and by assessing their implications for retrieval precision. Building on previous research in technical communication, the paper further investigates the role of large language models (LLMs) in supporting the content engineering phase through automated analysis of legacy documents, derivation of metadata classifications, and curation of the resulting semantic structures in knowledge graphs.

1. Introduction

Digitalization is reshaping how product information is accessed and delivered. User expectations increasingly shift from static PDF-based documentation toward interactive, personalized access to technical knowledge [1]. LLM chatbots enable dialog between users and systems. They translate natural language input into machine-readable queries and can deliver targeted responses. Beyond simple question-answering, they support functions such as on-demand content translation and role-based access control, ensuring that users receive only the information relevant to their context [2].

This development raises the question of how technical content must be structured to support reliable Artificial Intelligence (AI)-based delivery such as in chatbots. This topic has been explored in related research work, which provides the foundation for the work presented in this paper [3].

RAG has emerged as a relevant architecture for technical communication and documentation-driven assistance systems [2]. RAG is a technique that augments LLMs with external knowledge sources in order to improve the accuracy and reliability of generated responses [2,4].

The empirical basis of this study is a representative content corpus provided by Grundfos (Bjerringbro, Denmark) providing eco-conscious water solutions designed to transport and treat liquids across the building, utility, and industrial sectors [5].

2. Project Goals

The overall objective of this study is to assess content regarding its suitability for AI-based retrieval applications. AI-readiness is evaluated through a set of metrics focusing on the semantic similarity of the content defined by one of the authors [3].

Based on this assessment, the suitability of the proposed method is evaluated as an architectural and content pre-assessment approach for RAG-based delivery scenarios.

Given the central role of metadata in content governance, the work additionally specifies an LLM-supported process for deriving metadata classifications and applying them to legacy documentation at scale.

Finally, the content corpus is analyzed with regard to topic variants, and reuse behavior to determine whether observed similarity primarily stems from versioning effects or from true content variants.

3. Method and Process

The method used in this research is based on the PIAI!-Lab environment [3]. The analysis framework is built in Python 3.12.10 while using an LLM-assisted developing approach. GPT-4.0 is accessed via API integration provided by the research partner to support secure content analysis.

The topic classification method used by the company and serving as the basis for the analysis in this study is PI-Class, which organizes metadata along two dimensions: intrinsic and extrinsic. Intrinsic metadata characterizes the topic’s uniquely contained information within the information architecture, whereas extrinsic metadata captures its assignment to one or more end-product types delivered to customers [6].

The overall process is structured into four phases: content management analysis, AI-readiness analysis, content engineering process definition, and method evaluation.

AI-readiness is assessed using similarity-based metrics derived from vector representations of topics. Two main properties were measured. Similarity Width (SimWidth) quantifies how many other topics are similar to a given topic above a predefined cosine similarity threshold and is therefore interpreted as an indicator of redundancy or reuse. Cosine similarity measures the angular similarity between two topic embeddings, with higher values indicating greater semantic similarity. Mean Similarity Difference (MSD) measures the average distance between a topic vector and the remaining topic vectors and is operationalized using cosine distance. Values close to 0 indicate that a topic is highly similar to many other topics, whereas values closer to 1 indicate higher uniqueness [3].

The underlying hypothesis of this study is that lower content redundancy (i.e., fewer highly similar topics) leads to more reliable retrieval in RAG systems and thus indicates higher “AI-readiness”. If redundancy in topics is high, several highly similar topics may appear simultaneously among the top-ranked results in the retrieval process, introducing additional irrelevant context. This might happen especially in the case of variant-rich products as it is the case of the considered domain. Therefore, a set of similar, yet non-appropriate topics will increase the probability of decreasing answer accuracy.

3.1. Content Management

The first phase addresses the content management of the topic ensemble by analyzing topics with respect to their variants. The ensemble consists of 26 bookmaps comprising 1173 DITA topics in XML format. In the DITA standard, a topic represents a modular unit of information focused on a single, specific subject. A DITA bookmap serves as a container that organizes topics and enables the transformation of a topic collection into a publication; individual topics are referenced in the bookmap via links [7].

Metadata for each topic is extracted. Intrinsic metadata is obtained from the topic-level prolog, whereas extrinsic metadata is derived from the bookmap-level prolog of the corresponding bookmap in which the topic is referenced.

The extracted metadata is stored in a relational database for further analysis. The XML source files are stored in a local file system.

To provide an overview of the dataset, the distribution of topic sizes, topic reuse and metadata coverage are examined.

Since intrinsic metadata is largely missing, the analysis is pre-filtered by topic type, comparing product-intrinsic (p_int) metadata and topic types. An LLM-assisted analysis is then applied to candidate variant groupings to distinguish duplicates (i.e., substantively identical content) from true variants (i.e., content that is structurally similar but differs with respect to product-, context-, or configuration-specific information).

3.2. AI-Based Content Delivery

The second phase evaluates the AI-readiness of the content using similarity-based metrics derived from vector representations of topics.

To enable reliable embedding and subsequent analysis, the content is normalized prior to vectorization by stripping topics of all XML markup. Topics are then vectorized using the sentence-transformers model all-mpnet-base-v2 and stored in a vector data base. During vectorization, a small number of topics could not be embedded because they contain no textual content, resulting in 1168 vector representations out of 1173 topics.

The metrics are then computed over the topic ensemble to identify topics with extreme values, including topics with high and low SimWidth and topics with low MSD. These cases are more closely examined to determine which content characteristics drive high similarity or high uniqueness and to assess implications for retrieval.

To test the practical impact on RAG, probe queries are formulated for topics exhibiting high MSD values and to topics exhibiting low MSD values to prove how redundancy and uniqueness influence retrieval behavior.

In addition, the influence of topic size on MSD is assessed by relating word counts to MSD values. Finally, the similarity analysis is performed both on the complete topic ensemble and on reduced ensembles at bookmap level. By this one can simulate prefiltering in delivery environments and thereby calculate ensemble properties within the operational context in which topics might be consumed.

3.3. Content Engineering

The third phase corresponds to the content engineering stage, which is typically positioned prior to operational content management and content delivery [3]. It comprises a pre–content-management assessment aimed at preparing the transition to scalable delivery by deriving PI-classifications and curating these classifications in a knowledge-graph-compatible form. Although content engineering is not part of the original project plan, it is identified as necessary because intrinsic metadata is largely missing from the topic ensemble. Consequently, a comprehensive assessment of metadata effects is not possible. This gap can be attributed to the legacy documentation process, in which full PI-metadata-classification was not mandatory.

Nevertheless, full and consistent classification is considered beneficial beyond analytical purposes. Consistent PI-classification is a prerequisite for AI-based delivery, as it improves discoverability and enables systems to interpret topics in context [8].

For this reason, the paper defines an LLM-supported process to reclassify legacy content using the existing ontology provided by the project partner. While the ontology already specifies the relevant metadata categories, these labels were not applied consistently across all topics.

3.4. Method Evaluation

In the method evaluation, the proposed approach proves suitable for quantifying semantic similarity and characterizing the thematic distribution of the topic ensemble. It successfully identifies redundant topics and clusters and predicts behavior of topic-based ensembles in RAG processes derived from microscopic properties.

At the same time, similarity does not assess clarity, structural soundness, correctness, or usefulness of the content. Highly distinct topics may still be poorly written or otherwise unsuitable for delivery. While ensemble-level similarity and redundancy measures are informative indicators, AI-Readiness may also depend on the linguistic quality of the underlying documentation. On the other side, the introduction of topic-based information architectures is mostly performed to address and improve exactly the mentioned user-centric aspects of information management. We therefore treat this as a parameter of the investigated content ensembles being not assessed in the present research work.

4. Findings

4.1. Content Management

The findings in the content management phase summarize the structural characteristics of the topic ensemble, including topic size, reuse patterns, metadata coverage, and observed topic variants.

Topic length (measured in word count) is concentrated within a typical, operationally manageable range, with a mean topic size of 137.20 words (Figure 1). This indicates a relatively consistent level of detail and granularity across the documentation, with few outlier topics that are exceptionally long or unusually short. Given this distribution, embedding truncation is unlikely to be a major factor for most topics in the present analysis. This is relevant because the used embedding used truncates topics beyond a maximum input length (384 token) [9].

Figure 1. Distribution of topic sizes of global ensemble.

The highest reuse rates are observed for the topics: Factory Settings Tables, Operating Conditions, Security Settings, Address Information, Disposing, General Information and Hazard Statements.

The variant analysis identifies true variants in the sense that no completely identical content was observed across different product intrinsic and topic type combinations through a generalized variant heatmap and the LLM content analysis of the classification combinations (p_int–Topic Type). Due to the variant-rich nature of topics, a risk of semantic redundancy remains, particularly for RAG-based retrieval systems, because some variants differ only in elements such as links, QR codes, or images. Traditional RAG-systems, however, are often constrained by their reliance on text-only inputs, although recent work has extended RAG to multimodal settings [4]. The semantic content can therefore appear effectively identical to the RAG-systems, increasing the likelihood of ambiguous retrieval results.

4.2. Ensemble Properties

Figure 2 shows the cosine similarity of the global ensemble. Each line represents a single topic. Topics whose curves drop rapidly toward 0.0 are comparatively more unique, whereas topics whose curves remain at higher similarity values over a larger rank range are more similar to other topics in the ensemble. For almost all topics, the curves decrease steeply and then flatten gradually.

Figure 2. Cosine Similarity of the global ensemble.

Figure 3a shows the analytically compressed distribution of MSD of the global ensemble. The MSD values are generally within a narrow range, with distinct maxima and minima. This suggests that topics in the ensemble are overall relatively distinct, while some individual topics are substantially more similar to others.

Figure 3. (a) MSD of the global ensemble; (b) SimWidth of the global ensemble.

Figure 3b shows the SimWidth of the global ensemble with a threshold of 0.86. Most topics exhibit only a small number of neighbors above the threshold, whereas a limited number of topics produce pronounced spikes. This pattern suggests that redundancy is concentrated in specific topic groups.

4.3. Content Delivery

For topics with high MSD (MSD = 0.9828), a representative case (i.e., concept-PquCwoE) is queried using a question whose answer is contained within the target topic (i.e., What is the correct format for entering a gateway address?). The target topic is ranked first and shows a clear similarity margin compared to the remaining candidate, as shown in Figure 4a. This separation indicates that the retriever is able to identify the relevant context with high specificity.

Figure 4. Top 10 matches comparison topic with high MSD (a) and low MSD (b).

In contrast, for a query (i.e., What is the difference in net weight between an oval flanged pump and the standard ANSI version?) for a topic with a low MSD (MSD = 0.5818), similarity scores across the top-ranked candidates show a less distinct separation, suggesting limited discrimination between relevant and non-relevant results; in this case, the answer-bearing topic (i.e., reference_ve6_oe7_xba) does not appear within the top-10 retrieved topics as seen in Figure 4b. Overall, these results are consistent with the underlying hypothesis.

When examining topics with very low MSD relative to the rest of the topic ensemble, it becomes apparent that many of the lowest-scoring topics are tables or lists.

To assess the effect of topic size on MSD, the metric was analyzed in relation to topic word count. Results indicate a length effect. Extremely large topics (word count > 7000) consistently exhibit lower MSD values and fall below the global average of 0.7301 (Figure 5). In contrast, very high MSD values (MSD > 0.9) occur exclusively among small topics, indicating that short, focused units are more likely to be highly distinctive (Figure 5). However, a distinct cluster of small topics also shows extremely low MSD values, in some cases approaching the minimum observed values. Manual inspection of these low-performing small topics indicates that they are predominantly tables.

Figure 5. MSD compared to topic size.

When relating SimWidth to topic word count, no comparable length effect can be observed. SimWidth values are widely dispersed among short topics and remain generally low across longer topics. However, the topics with the most extreme SimWidth values (i.e, SimWidth = 20) contain tables (Figure 6). We observe an overrepresentation of topics containing tables among cases with low MSD and high SimWidth values, although other topic properties may also contribute to this pattern.

Figure 6. SimWidth compared to topic size.

4.4. Minimal and Maximal Threshold

The results show that MSD values are bounded within a domain-specific range. No topic reaches MSD = 1.0, as all topics share semantic overlap due to their common subject area (i.e., pumps), and no topic falls below MSD = 0.5818, since MSD is computed against the full ensemble, which introduces inherent variation even for identical topics.

As a consistency check, a thematically unrelated topic was injected into the ensemble (a short narrative about “Mickey Mouse’s adventure: The Discovery of the Secret Cheese Map”). Its MSD value (1.0241) exceeded the upper threshold, confirming that values near or above 1 are only attainable for content with no semantic overlap with the ensemble (Figure 7).

Figure 7. MSD of distinct topics.

MSD values therefore must be interpreted relative to the ensemble-specific thresholds: a value of 0.5818 may represent the least distinctive topic, while MSD ≈ 0.90 may already approach maximum distinctiveness within this domain.

When analyzing the bookmap-level ensembles and comparing them, no single global lower threshold can be identified. Specifically, the minimum MSD observed within individual bookmap ensembles varies across bookmaps and does not coincide with the minimum of the global ensemble, indicating that the lower threshold is context-dependent and influenced by the specific composition of topics of each bookmap.

When comparing arithmetic means (±1 SD across topic-level values) of MSD and SimWidth for the global ensemble with those computed for reduced, bookmap-level ensembles (i.e., only the topics referenced by a single bookmap, with the metric first computed per topic and then averaged within the bookmap), the mean MSD is lower for all bookmaps. The SimWidth values of the bookmaps are also generally lower, although a small number of bookmaps exhibit markedly higher values (Figure 8).

Figure 8. Mean values compared across the global ensemble and bookmap ensembles.

This is likely driven by the narrower thematic scope of bookmap-level ensembles compared to the heterogeneous global ensemble (e.g., installation procedures versus safety instructions). Topics within a single bookmap typically share product context, terminology, and stylistic conventions, reducing average semantic distances.

Subsequently, bookmap size is analyzed in relation to MSD values. The lowest mean MSD value over all bookmap ensembles is associated with a bookmap containing only 15 topics (Figure 9).

Figure 9. MSD of bookmaps compared to the bookmap size.

4.5. Content Engineering

To address the limitation of lacking metadata, a process for LLM-based intrinsic metadata classification of legacy content was defined in [3].

LLM-Classification is performed iteratively using small batch runs of normalized topics (50 topics per iteration). For each batch run, the topics and the current ontology are provided to the LLM, which is instructed via a prompt to assign the most appropriate class labels from the existing ontology to each topic or propose a new class. Newly introduced metadata classes are incorporated into an updated ontology and used as additional options in subsequent iterations. The procedure is repeated with subsequent batch runs until all topics are processed (Figure 10).

Figure 10. Defined LLM metadata classification process.

For each assigned class, the LLM is advised to capture the model’s confidence score and whether the selected class already exists in the research project partner ontology or was newly introduced by the LLM.

Classification is performed in two steps. First, topics are classified at the class level (i.e., Information Subject and Component), then at instance level, using the previously assigned class as a fixed constraint to only use its associated instances.

To increase reliability, a domain expert reviews newly proposed labels after each iteration, approving or deprecating them before the next run, thereby preventing the reuse of non-fitting classes in later iterations. Deprecated candidates are then manually mapped to the preferred Information subjects, enabling automated replacement of deprecated terms in future runs and reducing the proliferation of near-duplicate subjects by prioritizing reuse of approved terms (Figure 11).

Figure 11. Mapping process.

Overall, LLM-classification works reliably for topics that align with existing information classes, and the proposed classifications are generally meaningful. A remaining limitation is that identical topics can receive inconsistent classifications across repeated runs likely due to the non-deterministic behavior of the LLM [10].

LLM-based classification enhances interpretability, allowing peaks and troughs in MSD and SimWidth to be detected and examined based on intrinsic metadata as shown in Figure 12.

Figure 12. Peak-level metadata inspection for the topic Gateway before (left) and after (right) LLM-classification.

5. Summary and Outlook

5.1. Summary

In conclusion this research work shows that topic size, topic structure and ensemble reduction impact similarity metrics, and that an LLM-based workflow can support metadata classifications for legacy content. Topic size and structure are observed to influence Mean Similarity Difference values of topics within a specific topic ensemble. Very large topics tend to merge multiple concepts, which can dilute embeddings and increase similarity to other topics. Smaller topics are typically more modular, which may lead to more precise vector representations with higher MSD.

Topics with lower MSD values and high Similarity Width values often contain tables, which may be due to normalization reducing tables to similar text patterns, though other topic properties may also play a role.

The request examples support the hypothesis that high redundancy leads to the simultaneous retrieval of highly similar topics, which can introduce irrelevant context in RAG-systems.

Ensemble reduction, as performed through pre-filtering in modern delivery applications, affects the metric values. The global ensemble shows a higher average MSD value, likely because it compares topics across more heterogeneous content (e.g., different products). In contrast, bookmap-level ensembles tend to show lower average MSD values, likely due to their narrower thematic scope and shared vocabulary within a publication context.

PI-metadata classification completeness is a prerequisite for comprehensive AI-readiness and variant analyses and therefore constitutes a key requirement in the documentation process when content is intended for AI-based delivery use cases.

LLM-based reclassification can support legacy metadata completion but requires governance and human expert validation.

5.2. Outlook

For future AI-readiness analyses, the relationship between metadata classifications and retrieval performance should be examined to determine which PI categories are associated with systematically higher or lower similarity. In addition, the impact of incorporating semantic metadata into the vectorization process (e.g., embedding metadata-enriched representations) should be evaluated similar to the research work in [11]. Further work should also compare alternative embedding models and vectorization strategies following the approach in [9].

Given the observation that topics with a lower MSD value often contain tables, future studies should systematically evaluate table-specific preprocessing and its effect on the metrics.

Finally, the evaluation should be extended to multimodal RAG configurations for topics relying on images or QR codes to validate the generalizability of the proposed AI-readiness pre-assessment.