ActivityRDI: A Centralized Solution Framework for Activity Retrieval and Detection Intelligence Based on Knowledge Graphs, Large Language Models, and Imbalanced Learning

Abstract

We propose a centralized Activity Retrieval and Detection Intelligence (ActivityRDI) solution framework, demonstrate its application performance in network threat detection in detail, and show its generalization to other domains. Network threat detection is challenging owing to the complex nature of attack activities and the limited historically revealed threat data from which to learn. To help enhance the existing methods (e.g., analytics, machine learning, and artificial intelligence) to detect the network threats, we propose a multi-agent AI solution for agile threat detection. In this solution, a knowledge graph is used to analyze changes in user activity patterns and calculate the risk of unknown threats. Then, an imbalanced learning model is used to prune and weight the knowledge graph and to calculate the risk of known threats. Finally, a large language model (LLM) is used to retrieve and interpret the risk associated with user activities from the knowledge graph and the imbalanced learning model. The preliminary results show that the solution improves the threat capture rate by 3–4% and adds natural language interpretations of the risk predictions based on user activities with 95% accuracy. Furthermore, a demonstration application has been built to show how the proposed solution framework can be deployed and used. The generalizability of the proposed solution in other domains is also shown through an application to customer engagement, with 97% accuracy.

1. Introduction

Activity-based intelligence (ABI) discovers insights and patterns in activities associated with entities of interest through intelligent algorithmic analysis [1,2,3,4]. Thanks to technological advancement, activities in various systems are now widely captured. This wealth of information brings opportunities and challenges in activity analysis across different domains. An important domain is network threat detection, which is challenging owing to the complex nature of attack activities and the limited historically revealed threat data from which to learn despite the abundance of activity data in general.

Network threats have brought about significant financial losses and public safety issues in recent years. The total reported loss from cybercrimes in the US was more than $12.5 billion in 2023, according to the FBI’s Internet Crime Complaint Center (IC3) report [5]. In addition, public safety systems face increasing disruption in emergency communication systems and operations due to malicious attacks [6]. These are caused by new and increasingly complicated network attack activities that are not detected in time [7]. This presents a significant need for agile threat detection, which aims to identify and respond to evolving threats rapidly and proactively [8].

Analytical, machine learning (ML) and artificial intelligence (AI) methods have been widely used by researchers and practitioners to discover the patterns of known threats and detect unusual signals of unknown threats from the activities of users. Traditional ML/AI models typically need to learn from extensive historical data to guarantee good model performance. However, there are very limited historical data on known threats that have been observed but are not detected every time they occur. Furthermore, there are no data on unknown threats that have never been observed before. These two types of network threats are challenging for traditional ML/AI models to predict accurately.

Compared to other ML/AI models alone, the addition of knowledge graphs increases efficiency in analyzing user activities and their relationships to discover abnormalities. However, it has three challenges. The first is to prune and weight the information properly in the graph to filter out weak or redundant information about network threats. The second is to include large texts as part of graphs and graph analysis. The third is to unravel, diagnose, and interpret the complex activities and relationships of the users in the graph.

To overcome the challenges above, we propose Activity Retrieval and Detection Intelligence (ActivityRDI), a centralized and principled framework for activity-based intelligence that integrates dynamic knowledge graphs, imbalanced learning, and large language models. Unlike existing approaches, which apply these components independently, ActivityRDI tightly couples them in a closed-loop architecture, where learning outcomes actively shape activity representations and support real-time retrieval and interpretation. The framework is designed to address both known and unknown threats while producing human-interpretable explanations suitable for analyst decision-making.

In this framework, a knowledge graph is used to analyze changes in the user activity pattern and calculate the risk of unknown threats, an imbalanced learning model is used to prune and weight the knowledge graph and to calculate the risk of known threats, and an LLM is used to retrieve and interpret the user activities from the knowledge graph and the imbalanced learning model.

Using LLMs empowered by knowledge graphs enhanced with imbalanced learning as a set of AI agents, an adaptive and real-time monitoring framework can be implemented to achieve fast and early detection of malicious behaviors. This approach integrates the strengths of LLMs in contextual reasoning with the structured relationship modeling capabilities of knowledge graphs to monitor, predict, and explain potential threats as they unfold. The synergy between these components ensures both depth and immediacy in threat detection, making the system highly effective in dynamic environments.

Activity-based intelligence emphasizes understanding intent and risk through the analysis of observed activities rather than isolated events. ActivityRDI directly operationalizes this principle by representing activities as evolving relational structures, detecting deviations in activity patterns over time, and translating these deviations into actionable intelligence. In our framework, activities are captured and structured in a dynamic knowledge graph, changes in activity patterns are assessed through graph evolution and similarity measures, and risk hypotheses are quantified using imbalanced learning. Large language models then enable retrieval, synthesis, and interpretation of activity-level intelligence, supporting effective human-in-the-loop analysis. We demonstrate the effectiveness of ActivityRDI in network threat detection and further show that the framework naturally generalizes to other activity-centered domains.

2. Related Work

2.1. Knowledge Graphs

A knowledge graph is a data structure that encodes entities as nodes and relations as edges, often with rich attributes for both. By capturing complex multi-relational data, knowledge graphs facilitate reasoning about connections [9]. In addition, knowledge graphs are extremely efficient in representing sparse big data on the basis of their relationships and discovering abnormal patterns [10,11].

Different algorithms (e.g., similarity, centrality, community detection, pathfinding, shortest path, and link prediction) can be selected and run flexibly on graph data structures. Knowledge graphs have become popular in many domains [12,13,14,15], including cybersecurity, to integrate heterogeneous logs, documents and threat intelligence [16,17,18,19,20,21]. In a security context, a knowledge graph might include nodes for entities (e.g., users, hosts, processes, and files) and edges for actions (e.g., logons and file accesses), enabling graph algorithms to detect suspicious patterns.

Despite all these advantages of knowledge graphs, weak or redundant information in the graph can create challenges. Furthermore, it is not an efficient practice to include extensive text data on either nodes or edges of a graph, for example, actual article text that a user reads. In addition, it requires a great deal of expert knowledge and experience to interpret the information in the graph and diagnose problems accordingly. The following are three practical implementation challenges:

• First, how can the activity nodes and edges in a large graph be pruned and weighted properly?
• Second, how can large amounts of content be incorporated into the activity graph analysis?
• Third, how can information from both a graph and an imbalanced learning model be retrieved and interpreted efficiently?

The above challenges can be overcome by combining a knowledge graph with imbalanced learning and a large language model.

2.2. Imbalanced Learning

In cybersecurity, malicious events are typically rare compared to benign events; this pattern is referred to as imbalanced data, where a dataset has far fewer observations in the minority class (e.g., malicious event, threat) than in the majority class (e.g., benign event, non-threat) [22]. Imbalanced learning is the process of learning patterns from imbalanced data.

Imbalanced learning aims to eliminate the bias in the learning processes of traditional ML/AI models on imbalanced data. Traditional models maximize overall accuracy, while imbalanced learning pays more attention to the accuracy of the minority class and maintains overall accuracy at a reasonable range. To adapt ML/AI models to imbalanced data, imbalanced learning techniques such as sampling, weighting, and thresholding are commonly used [23].

With extremely imbalanced data (for example, a dataset whose minority class constitutes less than 1% of the sample), the weighting approach, also called cost-sensitive or weighted classification, generally works better, especially when combined with thresholding approaches [24]. The main reason is that the number of observations in the minority class is too small to conduct the sampling in a representative manner. In a weighting approach, the observations in the minority class are given higher penalty weights in the loss function, indicating greater losses if they are misclassified than if observations in the majority class are misclassified. For example, in logistic regression, one can use a weighted log-likelihood loss that assigns an increased penalty weight of greater than one to threat observations and a weight of one to benign observations [25], where the penalty weight values are determined on the basis of heuristic rules. Another approach is algorithmic learning of both penalty weights and model coefficients from the training data using a custom log-likelihood function [24], as shown in Equation (1), where i is the training sample index, $y_i$ is the dependent variable value of the training sample i, $x_i$ is the independent variable vector value of the training sample i, and $\beta$ is the coefficient vector of independent variables. The mathematical derivation, implementation, and experiments of this custom log-likelihood function can be found in [26].

$$\begin{array}{cc}\hfill \underset{\beta ,\lambda }{min}& -\sum _{i=1}^m[{\lambda }_i{y}_{i}log\left(\pi \left({\beta }^T{x}_i\right)\right)+(1-y_i)log(1-\pi \left({\beta }^T{x}_i\right))]\hfill \end{array}$$

(1)

In our problem, imbalanced learning techniques are used to prune and weight the nodes and edges of the graph on the basis of their relationships to the network threat. Generally speaking, a graph is pruned and weighed on the basis of the importance of the information represented on the nodes and edges in the graph. This typically depends on the problems that are being solved and the algorithms that are used. Wu used a graph hierarchy inference method based on the Agony model to eliminate noisy nodes and edges in the graph [27]. Chong derived the graph weights from the graph adjacency structure [28]. Jarnac used bootstrapping via zero-shot analogical pruning to select relevant nodes or edges within the graph [29]. Given that our objective is to prune and weight the nodes and edges of the graph on the basis of their relationships to the network threat, a supervised learning approach is more suitable for our problem. Furthermore, because the historical data consist of less than 1% threat observations and more than 99% non-threat observations, the imbalanced learning techniques are specifically used in our solution.

2.3. Large Language Models (LLMs)

Large language models are neural networks that are trained on massive text corpora and can understand and generate natural language [30,31,32]. Existing LLMs include ChatGPT, LLaMA, Gemini, Claude, etc. [33]. LLMs have been used for applications including summarizing (e.g., summarizing user reviews for brevity), inferring (e.g., sentiment classification, topic extraction), transforming text (e.g., translation, spelling, and grammar correction), expanding (e.g., automatically writing emails), and retrieval-augmented generation (RAG, e.g., reference on a knowledge base beyond the training data before response) [34]. However, the standard LLM retrieval process is based on similarity [35], where the similarity between the user’s question and the documents in the database is measured and the most similar documents are selected to answer the user’s question, as shown in Figure 1.

Recent work has explored multi-agent LLM systems where multiple models collaborate to solve tasks. These agents can pose natural-language queries to each other and to external data sources [36,37]. This enables dynamically intelligent interactions and collaborations among LLMs and other models and tools that are typically required to work together to solve problems [37,38]. For example, to solve mechanical problems, the multi-agent AI platform MechAgents was developed with a comprehensive intelligent ability to retrieve and integrate relevant knowledge, theory and data; construct and execute code; and analyze results using multiple numerical methods [37]. Another example is that multi-agent AI systems are used to enhance the decision support for smart city management, combining LLMs with existing urban information systems to process complex routing queries and generate contextually relevant responses, achieving 94–99% accuracy [39].

In our framework, LLM agents serve as query and reasoning engines: they translate user questions into graph queries, iteratively refine the queries, and interpret the results as human-readable explanations. For example, an agent might summarize a subgraph or explain the rationale behind a flagged anomaly.

2.4. Graph Anomaly Detection

Detecting anomalies in graphs that evolve over time is a well-studied problem [40]. We focus on measures such as the weighted Jaccard similarity between successive graph snapshots. Given two weighted graphs G and H in the same node set, we define their weighted similarity in Equation (2), whose solutions are in the range $[0,1]$. $J=1$ if the graphs are identical, and smaller values indicate a structural change. If at time t we have graph $G_t$ and at $t+1$ we add some edges or weights to obtain $G_{t+1}$, then $J(G_t,G_{t+1})$ quantifies how much the graph changed. Intuitively, if few edges change, J stays close to unity, but a surge of new edges (an anomaly) will markedly decrease J.

$$\begin{array}{c}\hfill J(G,H)={\displaystyle \frac{{\sum }_{e}min(w_G\left(e\right),w_H\left(e\right))}{{\sum }_{e}max(w_G\left(e\right),w_H\left(e\right))}}\end{array}$$

(2)

In summary, our work integrates a graph backbone, imbalanced learning, and cooperative LLM reasoning into one framework; this undertaking is the first of its kind, despite that the individual metrics have been examined separately in many studies and for many applications. Knowledge-graph-based methods have been applied to security analytics and threat intelligence. Graph-based semi-supervised learning and bootstrap methods have been used to handle noisy security data. Imbalanced learning approaches have been adopted for rare-event detection and for activity log anomaly detection. In parallel, transformer-based models have been explored in cybersecurity. Recent work on multi-agent LLMs highlights that cooperative LLM systems can solve complex tasks through natural-language dialog, but their use in security has been limited.

3. Proposed Methodology

ActivityRDI has three core components: a knowledge graph, imbalanced learning, and a large language model. These components serve distinct and complementary roles. Knowledge graphs provide a natural representation for heterogeneous activities and their temporal relationships, but on their own, they can suffer from noise and redundancy. Imbalanced learning is therefore used not only for threat classification but also to prune and weight the knowledge graph, ensuring that activity representations emphasize threat-relevant signals under extreme class imbalance. Graph similarity measures enable the detection of anomalous behavioral shifts, allowing the system to identify potential unknown threats that are not captured by supervised models. Large language models are deliberately positioned outside the prediction loop and instead act as reasoning, retrieval, and explanation agents, translating structured activity intelligence into forms suitable for analyst understanding without requiring task-specific fine-tuning.

From an implementation perspective, this framework consists of three cooperating LLM agents: two collaborators and one supervisor, as shown in Figure 2. Collaborator 1 maintains a dynamic knowledge graph. At each time step t, it adds new events $\Delta E_t$ as edges to form $G_{t+1}=G_t\oplus \Delta E_t$, ensuring efficient online updates. Collaborator 2 trains and applies the weighted classifier on features derived from graph entities (e.g., node degrees, subgraph patterns) to estimate the likelihoods of threats. The two collaborators interact: if Collaborator 2 assigns high risk to certain events or nodes, Collaborator 1 increases their edge weights or marks them as unusual. In contrast, edges with low risk may be pruned to focus the graph on likely threats.

Periodically, Collaborator 1 also computes graph similarity by measuring the weighted Jaccard similarity between the current user graph $G_t$ and a reference graph (e.g., $G_{t-\tau }$ or a baseline). A drop in this similarity score signals an anomalous shift in user behavior, even if the classifier did not flag it. These signals are stored for reporting.

The supervisor agent handles user interaction. Upon receiving a user query (e.g., ‘What recent user behaviors look suspicious?’), the supervisor LLM generates structured queries for the graph database (using standard query languages or prompt-based retrieval). It may ask Collaborator 1 to list subgraphs around suspicious nodes or ask Collaborator 2 for classification probabilities. It then synthesizes these into a human-readable interpretation (e.g., ‘User X’s recent file access pattern is unusual given their history’ or ‘A new device connection to server Y matches no known normal behavior’). The supervisor thus bridges the gap between automated graph analytics and analyst understanding.

Figure 3 illustrates how agents exchange information through natural language and graph queries. Each agent can translate between text and data operations: graph queries, classifier invocations, and explanation generation. Through iterative prompting, agents refine their analysis: for example, the supervisor may refine a question such as ‘Why is node X flagged?’ and receive successive clarifications from the collaborators before answering.

Compared to the typical query process of an LLM as shown in Figure 1, the following additional functionalities are added to our multi-agent LLM query process:

• Multiple types of knowledge bases are used, including a user activity knowledge graph and documents. This ensures more comprehensive information to be considered.
• Interpretations are drawn from analytic models including graph similarity and imbalanced learning. This avoids the fine-tuning of an LLM for specific purposes, thus saving money and improving efficiency.

Compared to existing solutions, our proposed solution improves the detection of network threats by introducing the novel features listed in

Table 1. Summary of Novelties of Proposed Methodology.

Novelty	Existing Solutions	Proposed Solution
Detect the known threats and unknown threats at the same time	The detection of known threats and the detection of unknown threats are studied separately in existing solutions [41,42,43,44].	Our solution integrates the detection of known threats and unknown threats through a knowledge graph, imbalanced learning, and an LLM agent simultaneously.
Handle the extreme imbalance of known threat detection	The penalty weights are determined heuristically in existing solutions [45,46,47,48].	The penalty weights are determined algorithmically and learned from data in our solution based on our previous research.
Interpret unknown threats that are detected	The interpretation of unknown threats is rarely discussed in existing solutions [49,50].	Our solution detect activity changes through activity graphs and interpret changes using an LLM agent.
Prune and weight a large activity graph in a practically efficient and effective way	The pruning and weighting are performed heuristically in existing solutions [51,52].	Our solution algorithmically determines the weights of nodes according to their corresponding variable importance from the imbalanced learning model.

.

The representative works summarized above constitute the current state of the art in knowledge-graph-based threat detection, imbalanced learning, and LLM-driven cyberintelligence. Table 1 provides a structured comparison between these existing approaches and the proposed ActivityRDI framework along key methodological dimensions.

4. Experiments and Results

4.1. Data

The evaluation of the proposed solution in this paper is performed on the CERT Insider Threat Test Dataset [53]. This public dataset simulates enterprise user activities (logons, file accesses, emails, etc.) for a set of users and devices, with labeled insider threats at the logon level. Although the data are synthetic, the authors of the data set use a well-rounded approach to achieve realism [54], including detailed activity logs and ground-truth threat labels at the logon level, as listed in

Table 2. CERT Insider Threat Test Dataset files and fields.

File Name	Field Names
logon	id, datetime, user, pc, activity
device	id, datetime, user, pc, content, activity
email	id, datetime, user, pc, to, cc, from, activity, size, att, content
http	id, datetime, user, pc, url, content
file access	id, datetime, user, pc, url, activity, to, from, content
psychometric	name, email, role, project, dept, team, sup
answers (label)	scenario, start_datetime, end_datetime

; this realism makes it suitable and widely used for the research, development, and testing of many threat detection frameworks. All experiments in this paper use this dataset.

4.2. Knowledge Graph Creation

A graph G uses the nodes and edges to represent relational information about users, computing devices, and activities, as shown in Figure 4.

• Nodes V: A node represents a user, user role, device, activity type (i.e., logon, email, file access, removable connect, removable disconnect, web visit, logoff) or activity time.
• Edges E: The edges connect the user, the user role, the device, the activity type, and the activity time, indicating which user performed what activity on the device at what time.

4.3. Graph Pruning and Graph Weighting Using Imbalanced Learning Techniques

To reduce redundant and insignificant information in the knowledge graph and improve the algorithm’s efficiency, we prune and weight the graph using imbalanced learning techniques, which evaluate how the information represented by the nodes and edges in the graph are related to the threats in the historical data. First, the numerical representations of the user activities in the graph and the historical threats are created as independent variables (i.e., features) and a dependent variable, respectively. Then, their relationships are examined through information value, and variable clustering techniques filter out the independent variables with weak predictive power or redundant information. The nodes and edges representing weak or redundant information are pruned from the graph. Lastly, a predictive model is trained with a customized imbalanced learning technique to predict whether a logon session is a threat. The predicted value from this predictive model is used to weight the nodes in the graph.

4.3.1. Feature Creation

In the predictive model, the dependent variable is a binary variable with 1 indicating a threat logon and 0 indicating a normal logon. The independent variables are 56 variables (that is, features) representing the current and past activities of the users, such as the number of executable files running in the current session and the number of executable files running in the past sessions.

4.3.2. Feature Selection

The relationships between these independent variables and the dependent variable are then examined through the information value, and the interrelationships among these independent variables are examined through the clustering of variables. Sixteen independent variables are selected to be used in modeling.

4.3.3. Imbalanced Learning

The historical data are split into a training set (70%) to train the model and a validation set (30%) to evaluate the model performance later in a stratified manner, ensuring that the percentage of threats in the training and validation data is the same as in the historical data. In historical data, the percentage of threats detected is 0.34%. To mitigate data bias, the weight of each training sample ${\lambda }_i$ is first learned through the custom log-likelihood in Equation (1) from the training data; the code for this implementation can be found in [55]. The learned sample weights ${\lambda }_i$ are then applied to the machine learning model training process (e.g., a gradient boosting model). Based on our previous research results in [24,26], we have chosen to build a gradient boosting model trained without learnable weights and with learnable weights in this experiment. Their performance is evaluated on the validation set using the gain and the area under the precision–recall curve, as shown in

Table 3. Model performance.

Performance Metric	Model 1	Model 2
% captured true threats at top 3% predicted risky logons among all true threats (gain)	56%	60%
% captured true threats at top 30% predicted risky logons among all true threats (gain)	95%	98%
Area under precision–recall curve	0.186	0.204

.

The models are as follows:

• Model 1: gradient boosting model trained without learnable weights.
• Model 2: gradient boosting model trained with learnable weights from Equation (1).

Compared to Model 1, Model 2 trained with learnable weights can capture 4% more true threats in the top 3% of predicted risky logons and 3% more true threats in the top 30% of predicted risky logons. The overall improvement is approximately 2% under different probability cut-off points used to convert the predicted threat probability into binary values. Given the better performance of Model 2, the variables and their importance in Model 2 are used to prune and weight the corresponding nodes of the activity graph built on the graph schema in Figure 4. For example, the variable “removable device connection” shows higher importance, and so the node “Removable Connect” in the graph is given a greater weight based on the variable importance value.

Although the absolute performance improvements of Model 2 appear modest, they are significant in the context of insider threat detection, where malicious events account for less than one percent of observed activity. Even a small increase in early threat capture can substantially reduce missed detections, lower analyst workload, and enable earlier intervention. Therefore, the observed 3–4% improvement represents a significant operational gain in real-world detection environments characterized by extreme class imbalance. This can potentially prevent the loss of $0.2 billion in 2021, $0.3 billion in 2022, and $0.4 billion in 2023 [5].

4.3.4. Graph Similarity

To measure the change in user activity over time, we first build the current activity graph and the previous activity graph. Then, the weighted similarity (e.g., weighted Jaccard similarity) between these two graphs is computed.

To show the result, take the user CSC0217 as an example in Figure 5. Its current activity graph shows that it logs onto the PC5866 device in the afternoon and connects to a removable device, and its previous activity graph shows that it logs onto the PC3742, PC6377, and PC2288 devices in the morning and visits some websites. The Jaccard similarity score between these two graphs is very small, resulting in a high activity change score, as expected. The activity change score, combined with the explanation of activity changes from the LLM retriever and interpreter in Section 4.4.2, identifies the risk of unknown threats.

4.4. Graph Retrieval and Interpretation Using Large Language Models

To better understand why the user’s activities are detected as known threats or unknown threats in Section 4.3.3 and Section 4.3.4, we extend the knowledge graph to incorporate the semantic contents of the user’s email, files, and web visits, and we then integrate the knowledge graph with an LLM retriever agent and interpret the user’s activities in a structured and explainable manner. While Section 4.3.3 and Section 4.3.4 quantify risk through supervised imbalanced learning and graph similarity measures, this section focuses on transforming those quantitative outputs into human-understandable activity intelligence.

4.4.1. Graph Schema Creation Extended

In traditional graph representations, nodes and edges efficiently encode relational information such as users, devices, timestamps, and activity types. However, the textual content associated with emails, files, and web visits is long and semantically rich and therefore cannot be directly incorporated into the graph without affecting storage efficiency and query performance.

To address this limitation, we adopt a hybrid design in which the textual contents are embedded into dense vector representations and stored in a vector database, while the structured relational metadata remain in the graph database, as shown in Figure 6. Each content object is associated with its embedding through a unique identifier, allowing the knowledge graph to reference the semantic information without storing the raw text directly. This design enables efficient integration of large textual information into the activity graph while preserving scalability and query performance.

4.4.2. Large Language Model as Retriever and Interpreter

Once the knowledge graph is extended with content embeddings, the LLM acts as a retriever and interpreter that bridges natural language queries and structured graph operations. When a user submits a query, such as requesting a comparison of activity patterns between two time periods, the LLM first translates the natural language question into a graph query. This translation maps the query into the corresponding graph database language to retrieve the relevant subgraphs, including user nodes, device connections, activity types, timestamps, and weighted risk values derived from the imbalanced learning model. If the query involves semantic understanding of content, the LLM also retrieves the relevant embeddings from the vector database on the basis of a similarity search and incorporates the associated textual summaries into its reasoning process.

The retrieved graph data include both the predicted likelihood of being a known threat, obtained from the imbalanced learning model in Section 4.3.3, and the activity change score, derived from the weighted Jaccard similarity in Section 4.3.4. These quantitative signals continue to be generated by their respective analytical modules to ensure reproducibility and consistency. The LLM does not replace these models; instead, it interprets their outputs by synthesizing relational structure, historical comparisons, and semantic content. For example, if a user suddenly connects to a removable device after historically performing only web browsing activities and the supervised model assigns a high threat probability to removable connections the LLM explains that the elevated risk arises from both structural deviation and historically significant features. In this way, the explanation is grounded in observable graph evolution and model-derived importance measures rather than speculative inference.

The interaction among the three agents described in Section 3 becomes particularly important in this stage. The collaborator responsible for maintaining the dynamic knowledge graph provides updated structural information, the collaborator responsible for imbalanced learning supplies threat probabilities and feature importance values, and the supervisor LLM orchestrates the retrieval process and generates coherent interpretations. Through iterative prompting, the supervisor may refine its queries to gather additional context, such as requesting a breakdown of recent device connections or identifying which activity types contributed most to the threat score. This iterative reasoning process allows the system to move beyond static question answering and toward interactive analytical support.

In this experiment, an application demonstration is built to ask about a user’s activity changes in the current time period compared with a historical time period or reference time period and return the answers about the user’s activity summaries, changes, and risk interpretations, as shown in Figure 7.

For example, as shown in Figure 8, we ask about the user Lisa’s activity change in January 2024 compared to December 2023. In the generated answer, we learn that Lisa’s activity changes by shifting from a mix of web visits and logon/logoffs in December 2023 to solely logon/logoffs with removable connection and disconnection on different devices, which explains why its likelihood of unknown threat is 80% and its likelihood of known threat is 70%.

To evaluate the accuracy of the LLM retriever and interpreter, we generated 100 questions concerning user activities, temporal changes, and associated risk levels. Ground-truth answers were constructed directly from the knowledge graph and analytical outputs. The LLM-generated answers were then compared with the ground truth by a human evaluator. An answer was considered correct if it accurately reflected the graph structure, aligned with the risk probabilities produced by the imbalanced learning and similarity modules, and provided logically consistent interpretation of the activity change. The results show that 95% of the responses were consistent with the ground-truth answers. This metric reflects interpretative accuracy rather than predictive accuracy and demonstrates that the LLM component effectively translates structured activity intelligence into reliable explanations.

Regarding the implementation, the open-source graph database NebulaGraph is used to construct and store the user activity graph, the Python (Python 3.13.5) package llama-index is used to index and interface with the graph and vector database, the Text-embedding-3-large model is used to generate content embeddings, and the GPT-4o mini model is used to perform retrieval and interpretation. The average response time of approximately four seconds per query indicates that the retrieval and interpretation layer can operate nearly in real time without interfering with the prediction pipeline. Additionally, to build the content nodes in the graph, we use online public documents instead of the content text data in the Insider Threat Test Dataset, because the latter data appear to be a mix of random words and phrases that do not form coherent sentences or convey a clear message, making it difficult to interpret any specific meaning.

5. Applications in Other Domains

The applicability of ActivityRDI beyond cybersecurity follows directly from its activity-centric abstraction rather than from domain-specific assumptions. Any domain characterized by heterogeneous entities, temporally evolving activities, rare critical events, and a need for human-interpretable intelligence can benefit from this framework.

To illustrate this generality, we present an application in customer engagement to demonstrate that ActivityRDI supports accurate, scalable, and timely retrieval and interpretation of activity intelligence across multiple systems in a complex real-world data environment.

Customer activity data, including digital activities, in-person event activities, and customer identifier hierarchy at different levels, are captured and stored in 10+ systems. In our solution implementation, we have developed a centralized customer engagement knowledge graph that connects all activities at different identifier levels.

When we added LLM-based natural language query functionality to this customer engagement graph, the preliminary results show a 97% accuracy on the retrieved information in 4 s on average among 700+ tested questions. Accurate, agile, and centralized customer activity retrieval enables faster business actions and better business strategies to convince customers and close deals.

6. Conclusions

Our centralized Activity Retrieval and Detection Intelligence (ActivityRDI) framework enhances existing practices of using analytics, machine learning, and artificial intelligence for network threat detection by integrating knowledge graph modeling, imbalanced learning, and large language model reasoning into a unified architecture. Rather than treating detection, anomaly analysis, and interpretation as separate processes, the proposed framework connects them through a closed-loop design in which structured activity representation, risk quantification, and natural language explanation mutually reinforce one another.

The experimental results demonstrate that incorporating learnable weights in the imbalanced learning model improves the capture rate of true threats by approximately 3–4% in the highest-risk segments. Although the absolute numerical improvement may appear modest, its operational significance is substantial given that insider threats account for only 0.34% of the observed logon sessions in the dataset. In such highly imbalanced environments, even a small improvement in ranking performance at the top predicted risk levels corresponds to a meaningful increase in early interception of malicious behavior. Early identification reduces the probability of missed detections and lowers the downstream cost associated with delayed intervention. Therefore, the reported improvement reflects not merely a statistical gain but an enhancement in prioritization efficiency and risk mitigation capability.

Beyond the supervised detection of known threats, the graph similarity mechanism enables the identification of unknown threats through structural changes in user behavior. By measuring weighted Jaccard similarity between current and historical activity graphs, the framework detects behavioral deviations that may not yet be represented in labeled data. The integration of supervised risk probabilities and unsupervised structural deviation scores provides a dual-layer detection capability. This simultaneous handling of known and unknown threats distinguishes the proposed framework from existing solutions, which typically focus on one dimension independently. The combination ensures that the system remains effective both for historically observed attack patterns and for emerging or previously unseen behaviors.

Equally important is the interpretability achieved through the LLM-based retrieval and explanation layer. The evaluation shows that 95% of the generated interpretations are consistent with the ground-truth graph data and analytical outputs. This high interpretative accuracy indicates that the LLM successfully translates quantitative model outputs and structural graph information into coherent explanations grounded in evidence. In operational environments, accurate interpretation is critical because actionable intelligence depends not only on identifying high-risk activities but also on understanding why they are considered risky. By separating predictive computation from explanation generation, the framework maintains analytical rigor while enabling transparent and human-centered reasoning.

The generalization results further demonstrate that the framework’s effectiveness is not limited to cybersecurity. In the customer engagement application, the centralized knowledge graph combined with LLM-based retrieval achieved 97% accuracy in information retrieval across heterogeneous data sources. This cross-domain performance suggests that ActivityRDI is fundamentally activity-centric rather than domain-specific. Any environment characterized by heterogeneous entities, temporally evolving interactions, rare critical events, and a need for interpretable intelligence can potentially benefit from this design. The underlying abstraction, which captures activities as evolving relational structures, quantifying risk through supervised and structural signals and interpreting results through language-based reasoning, remains valid across application domains.

7. Limitations and Future Work

As with any activity-driven framework, the effectiveness of ActivityRDI depends on the quality and coverage of the observed activity data. In environments where activity logs are sparse or highly noisy, graph similarity measures may be less stable, and LLM-based interpretations may require additional validation. These limitations suggest directions for future work on robustness and adaptive calibration, rather than fundamental constraints of the proposed framework.

Furthermore, the CERT Insider Threat Test Dataset used in the paper consists of synthetic data, which may not capture all characteristics of realistic, real-world data owing to privacy regulations and the subtlety of malicious actions by authorized users. It is a well-known challenge to collect suitable real-world threat data of good quality for the research and development of threat detection [54]. To show the performance of the proposed solution on real-world data, we have extended the proposed solution to applications in other domains where the data are easier to capture, such as customer engagement, as illustrated in Section 5 (Applications in Other Domains). However, we cannot disclose that dataset in this paper because of business confidentiality. In the future, we would like to apply the proposed solution to additional real-world data.