AI-Powered Analytics in Technical Communication: Advancing Information Management Toward Content Data Science ^†

Abstract

We introduce a similarity-based ensemble framework for assessing the AI-readiness of topic-based technical content in Retrieval-Augmented Generation (RAG) contexts. Rather than evaluating RAG performance through query-and-answer measurements, the proposed approach characterizes content corpora through microscopic similarity metrics—mean similarity difference (MSD) and similarity width (SimWidth)—derived from vector representations of individual topics. These metrics quantify the distinctiveness and contextual precision of topic ensembles, providing a corpus-specific indicator of expected RAG behavior prior to deployment. The method is validated across four separately published industry-academic research projects in the PIAI!-Lab framework in different industrial domains. Results consistently confirm that modular, PI-Class-based information architectures with concise metadata improve ensemble-level AI-readiness, while variant-rich content without metadata pre-filtering reduces context precision. Beyond the technical findings, the paper demonstrates that Content Data Science methods can be effectively integrated into advanced academic research in technical communication, bridging information management and data science methodology.

1. Introduction

The impact of recent artificial intelligence (AI) technologies on technical communication (TC) is evident across an increasing number of its sub-processes and underlying methodologies. One of the strongly affected areas is the delivery of information in chatbot scenarios, partially substituting traditional search and retrieval technology and changing user interaction [1]. In an increasing number of cases, this leads to competition between core TC methodologies—such as topic-based, highly semantic content creation and delivery—and content processing by large language models (LLMs). In the latter case, documents are processed irrespective of their logical structure, source formats, and semantic degree of order. A special and particularly relevant case in this domain is the chunking of documents and Retrieval-Augmented Generation (RAG) [2] of answers to user requests, which will be considered in detail in the subsequent sections.

In order to measure the quality of RAG implementations, scientific research typically focuses on measurements of response quality with respect to queries. For this purpose, ideal responses to a selected set of queries are compared with the responses obtained through the RAG process. From this comparison, standard numerical measures of response quality—such as precision and recall, as well as derived quantities or entropy-based quantities—can be determined. In some approaches, synthetic question sets are generated by AI-driven processes and compared with statements contained in the corpus in order to assess these performance characteristics of the RAG application.

In the present publication, an alternative approach is proposed, which has been implemented in research projects and published in the present conference proceedings. It is described in detail below. The approach rests on two conceptual foundations. First, it draws on the transitivity of vector similarity of modularized content units—referred to in technical communication as topics. Second, it involves an ensemble-theoretical treatment of microscopic and derived macroscopic quantities that exhibit conceptual correspondences with statistical aspects of thermodynamics. Beyond the introduction of the method, the essential findings of the parallel research contributions [3,4,5,6] are summarized.

Beyond formal theoretical treatment, the publication also addresses the academic dimension of applied research in technical communication. For this purpose, experiences from the aforementioned research projects are presented in the context of a master’s-level course. The AI technologies employed and the project settings are described with the view to enabling graduate students to conduct research with real-world data in collaboration with industry partners.

This research aims to bring together information management in technical communication—both in its teaching and its scientific dimensions—with the methodological approach of data science, thereby establishing a framework referred to as Content Data Science.

2. Conceptual Framework and Methodology

2.1. Information Architectures in Technical Communication

Over the past three decades, technical communication has developed into an interdisciplinary field that spans a wide range of activities in both industry and education. The practitioners in this field—content engineers or technical writers—operate across domains ranging from text authoring to the broader areas of language, information, and media management, including the underlying technologies and user-centric aspects.

Beyond this, the field has undergone a substantial technical evolution, such that information architects with a background in technical communication are now involved in the fundamental architecture of information, information processes, and digitalization initiatives, extending to cross-organizational data governance. This development now covers applications in the area of artificial intelligence as well. A central concern at present, however, remains the standardization and structuring of information—and, within system-supported environments of component content management systems (CCMS), the modularization of content. The resulting topic-based content is indispensable for collaborative authoring, controlled reuse, variant management for complex product ranges, and targeted information delivery.

A methodology that is necessary—or at least highly beneficial—in the present work as well as in many real-world implementations is the detailed classification of modular content. Without it, the automation of many processes within CCMS is not feasible, and targeted information retrieval remains limited. Crucially, however, the classification of modular content is a concept that must be established at the start of the editorial process, as it enables the planning of documentation scope, provides evidence of normative and technical completeness, and defines the delimitation between individual content topics. Modern information architectures therefore develop a classification schema in parallel with the modular content structure and use it to enrich the content accordingly. In principle, the classification concept should serve as the leading instrument for topic planning.

The PI-Classification method displayed in Figure 1, defined and described previously by the author [7], is widely used across industries and has become also the basis of the iiRDS standard for content delivery exchange in TC [8]. It covers four principal dimensions within the information space of modular topics (basic PI-Class), as well as two additional dimensions covering variant characteristics and functional metadata (extended PI-Class). In CCMS and delivery systems, classifications for modular content appear most generally as taxonomic metadata trees. In semantic systems, such as knowledge graphs, they appear as classes and instances within an ontological information architecture.

For the present initial work on topic ensemble properties, it is important to distinguish between intrinsic and extrinsic metadata. Intrinsic metadata defines the content of a topic in conceptual and editorial terms, with reference to the objects (physical, virtual or software components) described and under a finely differentiated specification of the information type. Extrinsic metadata, by contrast, describes the use of content with respect to products and document types, as well as target audiences and markets. Since this perspective always involves both the product dimension and the information character, the abbreviation PI has become established for this classification approach.

In practice, the metadata spectrum actually implemented is frequently incomplete, with assigned metadata being partially missing or incorrectly assigned. The underlying reason is often the absence of automated publication processes or not yet fully established topic-based delivery processes, which leads authors to regard metadata assignment as dispensable within the editorial workflow. This in turn limits the analytical potential of the content. On the other hand, it renders AI-driven processes an attractive option for post hoc classification. Both aspects are also addressed in the research contributions [3,4,5,6].

2.2. AI-Driven Content Delivery with RAG: From Chunks to Topics

As a complement or alternative to document- and topic-based delivery processes for technical information, chatbot applications have been increasingly adopted in recent years. Early purely LLM-based approaches suffered from a context problem, where queries about specific documents resulted in extensive content being passed through prompting. This was significantly improved by Retrieval-Augmented Generation (RAG) methods, in which chunking of content and subsequent similarity matching between chunks and queries leads to more precise answers and reduced hallucinations; see Figure 2. Chunks are extracted as text content from documents and segmented according to various strategies.

In scenarios where topic-based content already provides the basis for document production, it appears natural to use these self-contained units directly as chunks. In particular, the classification-driven modularization with well-separated information classes described above should provide a well-suited starting point in the RAG context. This, in turn, requires methods to assess the quality and behavior of topic-based content within RAG pipelines.

2.3. Analysis Methods and Metrics

RAG implementations are commonly assessed by measuring the response quality of chatbot systems, using either real-world question sets or AI-generated query collections [9]. The approach proposed here takes a complementary direction: a topic ensemble is examined with respect to its microscopic similarity behavior, and statistical measures are used to characterize ensemble properties relevant to RAG performance. Information-theoretic measures have been proposed as algorithm-independent indicators for system performance comparison [10]; the ensemble metrics introduced in our research extend this paradigm by assessing AI-readiness as an inherent property of topic-based content prior to RAG processing. Furthermore, recent work confirms that chunk quality is the dominant design decision in RAG pipelines [11,12]; our approach addresses this at the content level, where topic-based modules with well-defined information classes provide semantically coherent chunks without additional preprocessing.

The starting point of our proposed analytic scheme is the fundamental mechanism of RAG systems: matching queries against content—chunks or topics—through similarity comparison. Similarity is typically computed as the cosine similarity between content vectors generated by embedding models. Figure 3 shows a similarity distribution of a query matched against topics from a test ensemble drawn from a technical documentation environment.

The topics passed to the LLM as context for answer generation can be selected through a threshold and various retrieval strategies and reranking methods. This builds the retrieval stage in RAG, finally selecting one or more topics—or chunks—as the most similar and relevant content for a given query. The following analysis assumes a transitivity of similarity: if two or more topics exhibit a high degree of mutual similarity, a query that scores highly against one of them is expected to score comparably against the others.

This motivates an examination of the mutual similarity among topics from an industrial environment within an ensemble. Figure 4 shows the similarity distributions of ensemble topics arranged in descending order, where each curve represents a topic i whose cosine similarity has been computed against all other topics j in the ensemble.

This representation reveals a characteristic pattern typical of topic ensembles in technical documentation. Some curves drop steeply from the maximum of 1, leading to a sharply defined LLM context in a RAG process. Others remain flat near the maximum, potentially selecting, in this case, more than 20 topics depending on the similarity threshold applied—suggesting diffuse retrieval results and less precise answers. The underlying cause is typically the presence of module variants associated with different extrinsic metadata (most commonly different products) or near-duplicate modules with minor content differences. Taken together, all the individual curves provide an overall picture of the ensemble’s behavior in potential retrieval processes within AI-based content delivery.

2.4. Similarity Metrics for AI-Readiness

The qualitative microscopic behavior of individual topics can be described quantitatively in a measure-per-topic i, calculated as the mean similarity difference between topic i and all other topics j in the ensemble. The similarity difference (${\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{D}\mathrm{i}\mathrm{f}\mathrm{f}}_{\mathrm{i}\mathrm{j}}=d_{ij}$) between two embedding vectors $v_i$ and $v_j$ is given by

$$d_{ij}=1-cos(v_i, v_j),$$

(1)

and the mean similarity difference MSD accordingly by

$$MS{D}_i={\displaystyle \frac{1}{N-1}}{\sum }_{j=1,j\ne i }^N{d}_{ij}.$$

(2)

The MSD value quantifies the distinctiveness of a topic—its degree of separability from all other topics in the ensemble. Classically, this distinctiveness was achieved through editorial work in content creation. In thermodynamic analogy, $MS{D}_i$ can be interpreted as the microscopic energy of a topic, serving as the basis for further ensemble-level quantities. It is also related to the mean vector distance in the embedding space but is treated throughout the paper as a measure of linguistic and informational separability.

Figure 5 illustrates the individual similarity difference contributions of topics in descending order of similarity to the MSD of a given topic i.

While ${\mathrm{M}\mathrm{S}\mathrm{D}}_{\mathrm{i}}$ is the central element of the ensemble metric, a further quantity of interest is the similarity width (${\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{W}\mathrm{i}\mathrm{d}\mathrm{t}\mathrm{h}}_{\mathrm{i}}$), defined as the number of topics selected alongside topic i when matched by a query at a given similarity threshold. In Figure 5, this is indicated as a horizontal threshold line; its intersection with the similarity curve of topic i yields the topics potentially passed to the LLM as context. It should be noted that similarity transitivity holds reliably only at high similarity thresholds. The lower the applied similarity threshold, the less reliably similarity transitivity holds, and consequently ${\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{W}\mathrm{i}\mathrm{d}\mathrm{t}\mathrm{h}}_{\mathrm{i}}$ will no longer reflect the actual number of topics selected in a RAG retrieval process. A threshold of 0.80 or above should generally be considered reliable. In the projects examined [3,4,5,6], varying thresholds are applied, but 0.86 were used in many test cases. Accordingly, the transitivity threshold does not need to coincide directly with the selection threshold of the RAG implementation.

As with ${\mathrm{M}\mathrm{S}\mathrm{D}}_{\mathrm{i}}$, ${\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{W}\mathrm{i}\mathrm{d}\mathrm{t}\mathrm{h}}_{\mathrm{i}}$ is a microscopic quantity characterizing the group size of topics selected alongside topic i—in other words, the number of topics that would jointly form the LLM context when topic i is retrieved. The corresponding ensemble-level quantity, the averaged similarity width aggregates this across all topics and provides a macroscopic indicator of context multiplicity in the ensemble. As will be shown in the following, the distribution of SimWidth values across the ensemble is better represented by its median and outlier structure.

2.5. Distribution Functions and Analytic Compression of Ensemble Properties

Figure 6 shows the MSD distribution for the ensemble from Figure 4. High values above 0.6 indicate well-distinctive topics, while a broad cluster of lower values reflects topics with limited differentiation and distinctiveness. This illustrates that the selection of test queries for evaluating RAG implementations may introduce a sampling bias, as the distinctiveness and group size of the topics actually retrieved remain unknown without prior corpus analysis. Moreover, when combined with the intrinsic metadata of correspondingly enriched content, the MSD distribution provides a highly analytical tool for systematically identifying both distinctive and undifferentiated topics, and for working toward improved ensemble-level properties.

In addition to ${\mathrm{M}\mathrm{S}\mathrm{D}}_{\mathrm{i}}$, the Mean Square Similarity Difference (${\mathrm{M}\mathrm{S}\mathrm{S}\mathrm{D}}_{\mathrm{i}}$) is shown, given by

$${\mathrm{MSSD}}_i=\sqrt{{\displaystyle \frac{1}{N-1}}{\displaystyle {\sum }_{j=1,j\ne i}^N}{d}_{ij}^2}.$$

(3)

However, for the purposes of the performed evaluation, it does not yield deviating behavior compared to ${\mathrm{M}\mathrm{S}\mathrm{D}}_{\mathrm{i}}$.

The SimWidth quantity introduced above can likewise be represented as a distribution. Figure 7 shows the unsorted SimWidth values for the ensemble under examination. At a transitivity threshold of 0.85, group sizes per topic range from 0 to more than 50, with a notable number of topics exceeding 20. High SimWidth values correspond to the undifferentiated topics with low MSD_i, while values of 0 or near 0 indicate highly distinctive topics that contribute few or no additional topics to the LLM context.

So far, the analysis has focused exclusively on ensembles and the internal characterization of their topics. In industrial applications, however, the examination of sub-ensembles is equally relevant. Sub-ensembles are defined by extrinsic metadata and allow for the comparison of ensemble properties across different product groups, target audiences, document types, and similar dimensions.

To enable comparison across sub-ensembles, a further analytical compression of the data is applied. As shown in Figure 8, boxplot representations are used to aggregate (in this case) the MSD distributions of separately computed sub-ensembles into their median values. These are typically also compared against the overall ensemble of all topics. The boxplot representation additionally captures standard deviations and outliers. By comparing sub-ensembles, clear statements can be made about mean ensemble properties and thus about the behavior of, for example, product-specific content in the RAG process. For a more detailed assessment of AI-readiness, however, a closer examination of the microscopic quantities together with the intrinsic topic-level metadata is recommended for each sub-ensemble.

The method introduced so far enables systematic investigations of the AI-readiness of topic-based technical content from arbitrary source systems. Beyond industrial applications, it also opens new directions for academic work in the field of Content Data Science.

3. Academic Research and Content Data Science Projects

The following section presents selected research projects conducted with students and industry partners within the PIAI!-Lab framework as part of the Content Data Science program, demonstrating the application of the proposed ensemble metrics to real-world technical content corpora.

Academic Setting and the PIAI!-Lab

Student researchers were members of a Master’s program in communication and media management with a focus on technical communication. The majority held a Bachelor’s degree in the same field and had prior knowledge in content management. A basic understanding of computer science was present, though only partially at the level of advanced programmers. Project partners were business professionals in technical communication and information architecture. Content was exported from various CCMS environments as XML files, comprising either aggregated documents or separated topics, with metadata at varying levels of maturity. Project teams consisted of two to three students working in conjunction with an industry partner.

As part of the industry-academic cooperation, the following steps were adapted from the Data Science Lifecycle:

• Problem Definition: Business understanding of content-related processes and research questions was introduced during Bachelor studies, providing the domain foundation for the project.
• Data Acquisition: Data collection exports were technically and conceptually prepared by the course instructor in advance and extended at the start of the project in collaboration with the industry partner.
• Data Cleaning and Preparation: Data cleaning of XML export formats from content management systems was performed to enable selective access to topic-level content. Preparatory analyses, such as module sizes and variant frequencies, provide structural characterization prior to vectorization and ensemble computation.
• Data Integration and Transformation: Data transformation through normalization and semantic preparation of topics and ensembles for similarity analysis forms the core phase of the project.
• Exploratory Data Analysis: Analysis of ensemble properties includes visualization and preparation of data for subsequent investigations.
• Modeling: Data modeling based on standard ensemble metrics is extended by students with independent research contributions, developing additional models, applying project-specific perspectives, and deriving conclusions for the ensemble under investigation.
• Deployment: Operationalization is not part of the academic project scope.
• Communication: Documentation and knowledge transfer using a Python-based Jupyter environment (“PIAI!-Lab”) and presentation of results to academic and industry partners.

Project teams used AI-based vibe coding to build up a Python environment. All phases of the above lifecycle had to be completed within 15 weeks, followed by a final presentation and conference participation. Collaboration with industry partners typically took place in weekly online coordination meetings. As indicated in the data cleaning step, preliminary content management analyses were also required to characterize the data—i.e., the topics and documents. The present publication focuses on the actual ensemble property determination for AI-readiness. In the projects, the preliminary analyses were important for addressing individual research questions of the industry partners and for gaining familiarity with products, content, and the organization. The PIAI!-Lab was developed and provided in advance by the course instructor as a template with basic functions for computing ensemble properties. Upon request, the environment is made available to project partners along with the collected data and visualizations.

As shown in Figure 9, the logical and programming components were sliced within the PIAI!-Lab and executed as separately callable modules. As long as only ensemble properties were investigated using a standard vectorizer, no LLM access was required—which was relevant for data security reasons. However, hosted LLMs could also be integrated and called via API. The latter was important when, for example, chatbot responses were prototypically evaluated against real content.

In addition, RAG pre-tests were possible, demonstrating which topics were selected as RAG context for a given query. This allowed the validity and practical relevance of the ensemble method to be examined in an illustrative manner for both high and low similarity differences. As a standard embedding model, all research projects used the all-mpnet-base-v2 sentence transformer [13].

With this technical and didactic research setup, four project teams of two to three students each were able to conduct Content Data Science investigations on documentation content in the field of technical communication.

4. Results and Discussion

4.1. Ensemble Properties for AI-Readiness Measurements

By deriving ensemble-level metrics for RAG-based delivery implementations from microscopic, i.e., topic-based similarity analysis, an instrument for assessing the AI-readiness of content becomes available. It can be applied in a variety of ways to compare the expected behavior of content in chatbots or other RAG-based applications or agentic systems [14] with topic processing. Comparisons can be made, for example, between product domains, between different information architectures, or between chunking strategies that do not follow a topic-based approach. The primary metric is the similarity difference, i.e., the distinctiveness of content units.

Retrieval behavior is characterized by the sharpness of the LLM context—that is, the number of selected context topics or chunks relevant for answer generation. The logical foundation is the similarity transitivity described above: if two topics are highly similar to each other, a query matching one of them is expected to match the other with comparable similarity. While this assumption has inherent limitations, the RAG pre-tests conducted across the research projects provide practical evidence that the introduced similarity width at high similarity thresholds reliably reflects actual retrieval behavior.

Characterization can be performed in compressed form at the ensemble level or at the resolution of individual topics. This is where the significance of classification methods becomes apparent: analytical tracking and evaluation of topic content behavior—or the underlying topic concept—is most effective when intrinsic classifications are available. For purely analytical purposes, however, a retrospective classification within the same environment can also be used [4]. As the investigations demonstrate, extrinsic classifications should be used for pre-filtering ensembles, for example by product or document type. Unfiltered global ensembles—particularly for variant-rich products—otherwise exhibit low distinctiveness and less precise retrieval contexts.

4.2. Projects: Validation of Ensemble Properties

The ensemble property framework was applied and validated in four industry-academic research projects conducted within the PIAI!-Lab, covering distinct domains and extending the methodology in complementary directions.

A study examining content reuse and AI-readiness in the maritime component supply domain [3] revealed:

• Reuse behavior measurements analogous to the earlier introduced REx analytics [15,16] can be used to characterize ensembles and content management environments. Higher reuse rates are generally associated with shorter topics; large topics exhibit low reuse, consistent with variant-specific content creation in manufacturing environments.
• Product-level sub-ensembles show higher median MSD than the global ensemble, confirming that metadata-based pre-filtering improves retrieval sharpness.
• RAG pre-tests demonstrate that for low-MSD sub-ensembles, the correct topic fails to appear in the top 5 results in the global ensemble, while sub-ensemble filtering restores correct retrieval—confirming that ensemble sizing and extrinsic pre-filtering are strongly recommended.

In the research using content from the water treatment domain [4], the following was found:

• At small topic sizes, embedding truncation is unlikely to affect most topics in well-modularized TC corpora, supporting the use of all-mpnet-base-v2 [13] for this content type.
• Lower MSD thresholds detected in the data are ensemble-relative: the observed range must be interpreted within the specific ensemble context, as domain-specific semantic overlap imposes inherent bounds.
• Topics containing tables systematically exhibit low MSD, likely due to normalization reducing tabular content to similar text patterns—identified as a dedicated preprocessing challenge.
• An LLM-based metadata classification process with human-in-the-loop validation is usable for legacy content reclassification, enabling retrospective coverage of PI-Class or other classification schemes.

Comparing software documentation from different information architectures [5] yielded the following findings:

• PI-Class-based new information architecture achieves higher median MSD than legacy architectures, confirming that modular, classification-driven content structuring improves AI-readiness.
• Shorter topics have the greatest potential for high distinctiveness, but short topics containing tables or reference content can also show low MSD.
• Embedding model comparison shows that all-mpnet-base-v2 (384 tokens) [13] produces larger similarity amplitudes than higher-token models, making it more suitable for ensemble analysis despite its smaller token limit.
• Truncation tests confirm that MSD distributions remain structurally stable across token lengths below 384. In order to overcome token dependency and limitations, short topics yield the most reliable vectorization results.
• RAG pre-tests validate the metric: topics with high MSD yield steep retrieval score curves, while low MSD topics produce flat distributions with ambiguous context selection.

Investigating the particularly challenging case of variant-dense engine manufacturing documentation [6], the following was found:

• Decreasing ensemble sizing reduces the similarity width, demonstrating that metadata-based retrieval space reduction is highly effective.
• Three semantic levels were used: Level 1 (content + title), Level 2 (XML structural tags), Level 3 (intrinsic metadata scoring function)—only Level 3 produces a significant improvement in MSD and SimWidth.
• Semantic Level 3 applies a simple and discrete metadata scoring function combined with cosine similarity, effectively pushing mismatching topics toward zero similarity while maintaining high scores for matching topics.

4.3. Summary

In summary, the following conclusions can be drawn from this initial approach based on similarity-based ensemble metrics.

As a complement to query-and-answer-based evaluation, the primary metric of Similarity Difference and the secondary threshold-dependent metric of similarity width fundamentally characterize the behavior of content in RAG implementations. RAG pre-tests based on these metrics confirm the observed behavior.

The introduction of information architectures with a clear topic concept, concise metadata usage, and limited module sizes has a demonstrably positive effect on response quality and contextual precision in RAG processes. Extrinsic and variant-based filtering should additionally be applied in AI-based delivery implementations. Such filtering is more straightforward to implement with topic-based chunks than with PDF extraction and text chunking approaches. Although XML structuring as a semantic level does not directly improve ensemble metrics—as empirically confirmed in the engine manufacturing study—structured content remains an integral part of an optimized information architecture and is expected to contribute indirectly to the linguistic differentiation and distinctiveness of content units. In general, the comparison of different content architectures, chunking strategies, or product domains using the compressed ensemble representation enables systematic—rather than selective and potentially biased—investigation and optimization of delivery behavior.

On the academic side, these investigations integrate well into advanced student research projects and demonstrate that data science methods can be effectively applied within the field of technical communication.