DF-TransVAE: A Deep Fusion Network for Binary Classification-Based Anomaly Detection in Internet User Behavior

Abstract

User behavior anomaly detection plays a vital role in network security for identifying malicious access and abnormal activities in high-dimensional internet user behavior data. Although Transformer architectures have been widely adopted in anomaly detection tasks, and their integration with Variational Autoencoders (VAEs) has often been used to further improve detection accuracy, existing integration methods have failed to effectively balance global feature dependency modeling and generative data distribution learning. This results in limited capability in identifying complex anomalous patterns. To address this issue, this paper proposes DF-TransVAE, a novel deeply integrated framework that advances the integration of a Transformer and a VAE for supervised anomaly detection. The framework first fuses global contextual representations from the Transformer encoder with original input features, then maps the fused representation into the latent space via the VAE encoder. A cross-attention mechanism is introduced as the core of deep integration, enabling dynamic, bidirectional interaction between the fused features and latent variables to enhance information fusion. Lastly, a fully connected classifier equipped with residual connections outputs anomaly probabilities for supervised binary classification. Experimental results on two public datasets demonstrate that the proposed framework achieves better performance than existing deep learning methods in terms of accuracy, precision, recall, and F1-score, particularly in detecting complex anomalous patterns. Our results indicate that the deep integration mechanism we propose effectively addresses the limitations of conventional Transformer–VAE combinations.

1. Introduction

Internet user behavior data has expanded dramatically alongside the evolution of information technology and internet-based applications. This data captures diverse, complex interactions between users and systems, while also harboring hidden security threats and anomalous behaviors. However, the high-dimensional, temporal, nonlinear, and dynamic nature of such data poses considerable challenges to the accuracy and generalization of detection models [1]. For this reason, achieving efficient and precise identification of anomalous patterns in this data remains a major research focus and challenge, especially in scenarios where labeled normal/abnormal samples are available for supervised classification.

In this work, the research scope of internet user behavior data is defined as the high-dimensional sequential interaction data generated by users during their access to network services, including user access sequences, session feature vectors, operation records, service request frequency, access duration, and behavioral interaction patterns. Distinct from traditional network traffic (focusing on low-level transmission features such as data packets, ports and protocols) and system/industrial time series data (presenting relatively stable and regular temporal patterns), internet user behavior data has unique characteristics for malicious anomaly detection: it reflects high-level semantic interaction patterns between users and systems, exhibits strong individual differences and real-time dynamics, and its malicious anomalies are usually hidden in normal interaction sequences and manifested as subtle out-of-distribution behavioral deviations rather than obvious structural anomalies. These unique characteristics further highlight the necessity of jointly modeling global temporal contextual dependencies and latent data probability distributions for anomaly detection in such data.

Anomaly detection methods for internet user behavior can be broadly categorized into non-deep learning-based approaches and deep learning-based approaches. In recent years, deep learning-based methods have become the predominant techniques in this field, owing to their strong feature extraction capabilities and ability to model complex nonlinear relationships. Among these methods, the Transformer model provides an effective solution with its global self-attention mechanism [2], which excels in capturing dependencies between any two positions in a sequence and overcomes the limitations of recurrent neural networks (RNNs) in modeling long-range interactions.

To further promote the development of anomaly detection for sequential user behavior data, researchers have designed a variety of improved Transformer-based models. Representative studies have focused on multi-dimensional dependency modeling, adversarial training strategies, multi-scale feature extraction, and specialized loss design, as well as optimized encoder–decoder and memory-augmented structures [3,4,5,6,7,8,9,10,11,12]. These approaches have achieved remarkable progress in sequence representation and anomaly detection performance.

These methods are mainly applied in internet user behavior analysis, network security monitoring, and time series anomaly detection, and considerable improvements in detection accuracy, efficiency, and robustness have been widely demonstrated in related research.

However, as a deterministic model, a inherent limitation is revealed by the Transformer architecture when directly applied to anomaly detection tasks: while contextual dependencies within sequences are effectively captured by it, the capacity to explicitly model the underlying uncertainty and probability distribution of the data is inherently lacking [2]. This shortcoming is particularly pronounced in anomaly detection, where compact clusters in the latent space are typically formed by normal samples, while out-of-distribution (OOD) outliers are manifested by anomalies. For this reason, the sensitivity and robustness of its anomaly discrimination are restricted by the Transformer’s inability to quantify this distributional divergence.

To mitigate this limitation, the integration of Transformers with probabilistic generative models, such as Variational Autoencoders (VAEs), has been explored by researchers [13,14]. Specifically, You et al. [13] proposed a hybrid VAE–Transformer model for mechanical fault detection, in which a CBAM-VAE and a Transformer are used separately for data imbalance processing and fault classification. Song et al. [14] introduced a sequential VAE–Transformer–VAE architecture for time series anomaly detection, where the VAEs are responsible for dimensionality reduction and data reconstruction, and Transformers are adopted to capture contextual information. In these two mainstream hybrid approaches, dimensionality reduction and latent distribution modeling are typically performed by VAEs, while global contextual dependencies are captured by Transformers, and anomalies are identified through reconstruction error analysis.

Despite the effectiveness of these hybrid approaches demonstrated in [13,14], persistent limitations in their fusion paradigm have been identified through analysis. First and foremost, Transformers and VAEs are often loosely cascaded rather than deeply integrated, resulting in the failure to achieve synergistic alignment between global contextual representations and local input features. Secondly, an explicit interaction mechanism between the latent space and sequential context is lacking, which restricts the model’s discriminative capability for complex anomalous patterns. As a result, the formation of a unified and compact representation of normal behavioral patterns is challenging for these models, which leads to suboptimal performance in the distinction of highly complex anomalies. Based on this analysis, it is argued that most existing hybrid Transformer–VAE models (including [13,14]) remain limited to simple stacking or loose coupling. An ideal anomaly detection model for internet user behavior, however, must not only have both contextual modeling and distributional modeling capabilities integrated but also address a more core challenge: how deep interaction between global dependency representations and latent probability distributions can be fostered. This interaction is essential for the formation of a unified discriminative criterion—one that produces a more compact representation of normal behavior while sensitivity to anomalies is simultaneously enhanced.

To address these limitations, the DF-TransVAE hybrid model is innovatively proposed, representing a novel pathway for deep fusion between Transformers and VAEs. Rather than simply combining the two components as in [13,14], its core procedure is designed as follows: First, the global contextual representations extracted by the Transformer encoder are concatenated with the original input features to construct a more informative fused representation, laying the foundation for subsequent deep processing. Subsequently, a carefully designed cross-attention mechanism serves as the core of a closed-loop collaborative learning architecture. In this framework, the optimization of the VAE’s latent distribution is guided by the fused features, while the distribution information learned by the VAE is fed back through the same mechanism to refine the Transformer’s contextual understanding. This progressive design, advancing from feature concatenation to deep interaction, enables the model to form a unified discriminative criterion that captures a more compact representation of normal behavior while heightened sensitivity to anomalies is demonstrated.

The primary aim of this work is to develop a deeply integrated Transformer–VAE framework (DF-TransVAE) that not only improves the accuracy and robustness of internet user behavior anomaly detection but also addresses the inherent limitations of existing hybrid models (e.g., [13,14]) by fostering deep interaction between global contextual representations and latent probability distributions, thereby forming a more compact and discriminative representation of normal behavior while enhancing sensitivity to anomalies. The main contributions of this work are as follows:

• A novel, deeply integrated DF-TransVAE framework is proposed, establishing a new paradigm for the fusion of Transformers and VAEs in distribution-based anomaly detection and overcoming the limitations of individual models.
• An inter-feature cross-attention mechanism is presented to enable deep bidirectional interaction between contextual features and latent variables, which promotes the development of representation learning for sequential anomaly detection.
• Residual connections are introduced to alleviate gradient degradation and improve training stability, providing an effective optimization strategy for deep anomaly detection models.

This paper is arranged as follows: Related research is explored in Section 2, primarily describing the algorithms related to this article. The methodology is described in detail in Section 3. The details and conclusions of the experiments and comparisons with other models are detailed in Section 4. Section 5 summarizes contributions and possible improvements.

2. Related Work

The inherent challenges of detecting anomalies in internet user behavior arise from the complex characteristics of the data, such as its high dimensionality, nonlinearity, and dynamic temporal evolution. In response, the integration of Transformers—renowned for capturing long-range contextual dependencies—and Variational Autoencoders (VAEs)—known for learning latent data distributions—has been extensively explored, leading to the development of hybrid deep learning models. For example, a hybrid model for bolt loosening detection was proposed by You et al. [13], where CBAM-VAE is employed to address data imbalance and a Transformer is used for fault classification. A deep learning model combining a base VAE with a Transformer for anomaly detection was proposed by Song et al. [14]. In this approach, dimensionality reduction and embedding on time series data are first performed by a VAE encoder. Contextual representations are then generated by a Transformer, followed by data reconstruction by a VAE decoder. Anomalies are identified through reconstruction error analysis. In the ST-MTS-AD model proposed by Huo et al. [15], the Transformer encoder is used as the inference network of the VAE, while a multilayer perceptron is used as the generative network. In the work by Wang et al. [16], complex interdependencies among features are captured by a Transformer encoder, while image data and structured features from the encoder output are fused and then encoded into a latent space by a VAE. Disease classification is subsequently performed via a fully connected (FC) layer. The VLT-Anomaly framework was proposed by Zhang et al. [17], where a modified β-VAE is integrated with a Transformer and a BiLSTM for anomaly detection. Time series windows are encoded into low-dimensional embeddings by the β-VAE, while long-term dependencies are extracted by the Transformer and short-term dependencies by the BiLSTM. The processed features are subsequently fed into the VAE decoder to reconstruct the original data, thereby enabling anomaly detection via reconstruction error analysis. The LR-SemiVAE and RT-SemiVAE frameworks were proposed by Chen et al. [18]. Initially, dimensionality reduction and reconstruction of sequences are performed by a VAE, and anomalies are identified through reconstruction probability scores. Subsequently, LSTM modules are incorporated into the VAE’s encoder–decoder to capture short-term temporal dependencies, while label prediction is performed by a classifier. To further address the limitations of LSTM in modeling long sequences, the RT-SemiVAE variant has been proposed, where long-range dependencies are learned by a Transformer and local dynamics are preserved by retained LSTM components. A spatio-temporal anomaly detection model named CAE-T was proposed by Shang et al. [19], where a deep Convolutional Autoencoder (CAE) and a Transformer are integrated in parallel. The CAE is used for spatial feature compression and reconstruction of high-dimensional time series data, while the Transformer receives the convolutional latent representations to model long-term temporal dependencies and generate context-aware temporal embeddings. These joint representations are subsequently fed into a Support Vector Data Description (SVDD) module, and anomalies are ultimately identified by integrating three anomaly scores: reconstruction error, representation distance, and prediction error. A novel model named T-VAE was proposed by Li et al. [20] for multivariate time series anomaly detection. The architecture consists of a representation network, where information is extracted via the self-attention mechanisms, and a memory network, in which the data distribution is learned by a Variational Autoencoder (VAE). For patient survival analysis, the TransVCOX model was proposed by Li et al. [21], where feature enhancement and dimensionality reduction are performed by a pre-trained VAE, and inter-modal dependencies are subsequently captured by a Transformer encoder. Meanwhile, MT-RVAE was proposed by Wang et al. [22], in which sequence dependencies are captured by a Transformer with enhanced positional encoding, the KL divergence vanishing issue is mitigated by a residual-augmented autoencoder, and anomalies are ultimately identified via reconstruction errors and dynamic thresholding. The LSTransGAN-AE model was proposed by Jiang et al. [23], for which an LSTM-TS module was designed by integrating a spatio-temporal attention mechanism with a Long Short-Term Memory (LSTM) network for feature extraction. A TransGAN-AE module was also designed, combining Generative Adversarial Networks (GANs) with Transformer and autoencoder architectures for anomaly detection. In another work, a Transformer–VAE-based detection method was proposed by Hou et al. [2] to identify anomalous ship trajectories. In this approach, a Transformer encoder is employed as the encoder of a Variational Autoencoder (VAE) to map input sequential data into latent space representations, after which the VAE decoder is used to reconstruct the data. The loss function only uses reconstruction loss and KL divergence, with anomaly scores derived from the reconstruction loss.

The aforementioned research has been found to provide valuable insights for the integration of a Transformer with a VAE; however, notable limitations have been identified in their integration strategies—specifically regarding the separate processing of feature extraction tasks, the insufficient depth of inter-layer information interaction, and the lack of targeted structural optimization for the classifier module.

To address these limitations, the DF-TransVAE model is proposed in this study. Detailed descriptions of its methodological design (including feature extraction, fusion strategies, and classifier optimization) are included in the Methodology section for clarity and compliance with the manuscript structure.

3. Methodology

In this section, the process of detecting anomalies in user behavior is initially outlined. Following this, the DF-TransVAE framework proposed and the loss function used are described in detail. Figure 1 shows a diagram of the DF-TransVAE structure.

3.1. Anomaly Detection Process

The symbols used in the task are defined here. The feature vector before data preprocessing is defined as follows:

$$\mathrm{X}={\left[{\mathrm{x}}_{\mathrm{i}}\right]}_{\mathrm{i}\in \mathrm{I}}\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{E}}$$

(1)

where N denotes the number of samples and E denotes original feature size.

The feature vectors after data preprocessing are defined as follows:

$${\mathrm{X}}^{\mathrm{p}}={\left[{\mathrm{x}}_{\mathrm{i}}^{\mathrm{P}}\right]}_{\mathrm{i}\in \mathrm{I}}\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{F}}$$

(2)

where F denotes the preprocessed feature dimension.

The output feature vector of the Transformer encoder is defined as follows:

$${\mathrm{X}}^{\mathrm{n}}={\left[{\mathrm{x}}_{\mathrm{i}}^{\mathrm{n}}\right]}_{\mathrm{i}\in \mathrm{I}}\in {\mathbb{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}}$$

(3)

where ${\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}$ denotes the feature dimension of the Transformer output.

Concatenating ${\mathrm{X}}^{\mathrm{p}}$ and ${\mathrm{X}}^{\mathrm{n}}$ in the feature dimension results in the following:

$${\mathrm{X}}^{\mathrm{s}}=[{\mathrm{X}}^{\mathrm{p}}:{\mathrm{X}}^{\mathrm{n}}]={\left[{\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}}\right]}_{\mathrm{i}\in \mathrm{I}}\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{S}}, \mathrm{S}=\mathrm{F}+{\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}$$

(4)

Inputting ${\mathrm{X}}^{\mathrm{s}}$ into the VAE encoder obtains latent space variables as follows:

$$\mathrm{z}=\mathrm{V}\mathrm{A}\mathrm{E}\left({\mathrm{X}}^{\mathrm{s}}\right)\in {\mathbb{R}}^{\mathrm{N}\times {\mathrm{d}}_{\mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{l}}}$$

(5)

where Latent_dim is the latent space dimension.

The Transformer output ${\mathrm{X}}^{\mathrm{n}}$ is initially projected via a linear transformation to [N, latent_dim]. Feature interaction between $\mathrm{z}$ and ${\mathrm{X}}^{\mathrm{n}}$ is subsequently conducted using a cross-attention module, producing the final output ${\mathrm{h}}_0$:

$${\mathrm{h}}_0=\mathrm{C}\mathrm{r}\mathrm{o}\mathrm{s}\mathrm{s}\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}({\mathrm{X}}^{\mathrm{n}},\mathrm{z})\in {\mathbb{R}}^{\mathrm{N}\times \mathrm{L}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\_\mathrm{d}\mathrm{i}\mathrm{m}}$$

(6)

The interacted features ${\mathrm{h}}_0$ are fed into a fully connected classifier with residual connections to obtain the final classification according to the probability values:

$$\mathrm{P}=\mathrm{C}\mathrm{l}\mathrm{a}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{r}\left({\mathrm{h}}_0\right)$$

(7)

This paper aims to enhance the VAE encoder’s ability to learn robust feature transformations and mappings by concatenating ${\mathrm{X}}^{\mathrm{p}}$ and ${\mathrm{X}}^{\mathrm{n}}$ to form a fused feature. This provides the VAE encoder with more-complex input patterns, enabling it to generate more-representative latent space variables $\mathrm{z}$. Furthermore, the paper aims to boost model performance by promoting interaction between ${\mathrm{X}}^{\mathrm{n}}$ and $\mathrm{z}$. This interaction enables the integration of features from different levels, letting the model use both global structural and semantic information from the latent space.

3.2. DF-TransVAE Framework

To address the aforementioned limitations of existing Transformer–VAE integration strategies, the core design details of the DF-TransVAE model are elaborated as follows, focusing on feature extraction, deep fusion, and classifier optimization:

• A parallelized dual-stream feature extraction mechanism is designed and adopted. To overcome the drawbacks of the sequential cascaded structures used in existing models (where the Transformer and VAE are combined sequentially), two parallel and independent feature extraction branches are constructed. Through the implementation of a dual-stream architecture where the Transformer and VAE encoder operate simultaneously and independently, the integrity of global dependency characteristics (extracted by the Transformer) and latent distribution properties (captured by the VAE encoder) is maintained via source-level separation. This design avoids information loss caused by sequential processing and ensures the completeness of both types of core features.
• A cross-attention-based feature deep fusion strategy is introduced and implemented. After preliminary global contextual features (from the Transformer) and latent variables (from the VAE encoder) are acquired—elements that are usually directly transferred or merely aggregated in existing approaches—a cross-attention mechanism is employed to facilitate in-depth, bidirectional interplay between temporal contextual features and latent distribution variables. This mechanism dynamically calculates the correlation weights between the two types of features, thereby adaptively enhancing the latent patterns most relevant to the current context. For this reason, a fusion depth significantly exceeding that achieved by the simple concatenation or unidirectional transmission used in existing methods is realized, which effectively improves the discriminative ability of the fused features.
• The classifier architecture is enhanced via the embedding of residual connections. In the final classification decision stage, standard feedforward networks (commonly adopted in existing studies) are not used; instead, a residual connection structure is embedded into the fully connected layer classifier for the first time. This structural optimization effectively mitigates gradient degradation in deep networks during the training process, enhances training stability, and further improves the overall classification performance and robustness of the model.

These four components of the DF-TransVAE framework work together to realize the anomaly detection task. The input data is first provided to the Transformer encoder, which extracts sequence features by means of a multilayer structure to capture data characteristics. Then, the features and output from the Transformer encoder are spliced. That is, they are placed behind as a new column for developing a combined feature. They are then supplied to the VAE encoder. The encoder uses three fully connected layers to extract the local features of the data to obtain latent spatial variables. Next, the cross-attention module is interacted with to enhance feature fusion. Lastly, the interacted features are input into a fully connected classifier with residual connectivity, which outputs the anomaly probability and realizes anomaly detection and classification.

3.2.1. Transformer

This model was first proposed in 2017 [24]. However, in the initial period, the model was not used for anomaly detection tasks but was applied to natural language processing tasks [25]. Because of the excellent results achieved in that task, this model has been gradually applied to the anomaly detection task of sequences by researchers. The Transformer model is different from previous RNNs and CNNs in that it uses a self-attentive mechanism to capture the dependencies of the data and thus has a great advantage in processing sequences.

The first proposed Transformer is composed of two parts; that is, it contains both encoder and decoder parts. However, in this model, the decoder part is not used and only the encoder part is used to extract features, which makes the model simpler. The Transformer encoder is built by stacking multiple identical layers, with each layer sharing parameters. The Transformer encoder structure is shown in Figure 2.

In this model, a Transformer neural encoder and not a decoder is applied for feature extraction. The Transformer encoder effectively captures long-range contextual dependencies, thereby extracting informative feature representations. Unlike traditional recurrent neural networks (RNNs), the Transformer encoder does not depend on sequential computation order and can process the entire sequence in parallel. This feature allows the encoder to fully leverage the parallel computing capabilities of modern hardware like GPUs and TPUs, thereby greatly improving training and inference efficiency.

3.2.2. VAE

The VAE used here is different from previous models in that instead of mapping the inputs directly to a fixed vector, it maps inputs to a probability distribution, making it able to capture the complex distributional properties of the data. The VAE has two components: the encoder and the decoder. This study uses the encoder part of the VAE without using the decoder part, which makes the model simple.

The VAE’s encoder focuses on learning the latent representation of the input data with the goal of making it closer to the prior distribution. The KL dispersion term in its objective function is used to measure the gap between the encoder-generated posterior distribution and the prior distribution, as a way to drive the latent spatial distribution closer to the pre-determined prior distribution. KL divergence is expressed as follows:

$$\mathrm{K}\mathrm{L}({\mathrm{q}}_{\mathrm{\phi }}(\mathrm{z}|{\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}})||\mathrm{p}(\mathrm{z}))={\displaystyle \frac{1}{2}}{\sum }_{\mathrm{i}=1}^{\mathrm{d}}[{\mathsf{\sigma }}_{\mathrm{\phi }}^2+{\mathsf{\mu }}_{\mathrm{\phi }}^2-\mathrm{l}\mathrm{o}\mathrm{g}({\mathsf{\sigma }}_{\mathrm{\phi }}^2)-1]$$

(8)

where $q_{\mathrm{\phi }}\left(z\right|{\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}})$ is the latent distribution of the encoder output, $\mathrm{p}\left(\mathrm{z}\right)$ is the prior distribution, and d is the dimension of the latent variable.

To enable back propagation and optimization, the VAE introduces the reparameterization trick to separate randomness from the network parameters. Specifically, it is transformed linearly to obtain the latent variable through the following transformation:

$$\mathrm{z}={\mathsf{\mu }}_{\mathrm{\phi }}+\mathsf{\epsilon }·{\mathsf{\sigma }}_{\mathrm{\phi }}$$

(9)

where ${\mathsf{\mu }}_{\mathrm{\phi }}$ and ${\mathsf{\sigma }}_{\mathrm{\phi }}$ are the mean and standard deviation of the encoder output, respectively.

To train the VAE encoder, the following two steps are employed, First, the encoder transforms the input data ${\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}}$ into an approximate posterior distribution $q_{\mathrm{\phi }}\left(z\right|{\mathrm{x}}_{\mathrm{i}}^{\mathrm{s}})$ of the latent variable $\mathrm{z}$, from which samples of the latent variable $\mathrm{z}$ are drawn. Second, by minimizing the KL divergence, the model is encouraged to generate a latent variable distribution that closely aligns with the prior distribution. This helps ensure that the learned latent variables are evenly distributed across the latent space rather than being confined to a specific region, thereby preventing the learning of noise from the training data or overfitting to specific patterns in the training set, which could degrade performance on unseen data and lead to overfitting.

In the proposed DF-TransVAE model, the input to the VAE encoder is not a single-source feature but a fused feature formed by concatenating contextual features from the Transformer output with original input features along the feature axis. This central design, rooted in the theory of multi-perspective information fusion, is designed to synergistically utilize features across different abstraction levels for enhancing latent space modeling. The theoretical basis is primarily founded upon the following two aspects:

First, the design is guided by the principle of informational complementarity. The most complete and fine-grained information is preserved in Feature ${\mathrm{X}}^{\mathrm{p}}$ (the original input), which functions as the core building block of user behavior patterns; however, it is limited when it comes to explicitly representing complex global dependencies among features. Such contextual information is provided precisely by Feature ${\mathrm{X}}^{\mathrm{n}}$ (the encoded representations), which is extracted through the self-attention mechanism of the Transformer. Were only Feature B to be utilized, the model might be rendered over-dependent on abstract associations, with the risk of overlooking subtle signals or isolated patterns in raw data that are essential for anomaly detection. By concatenating these two feature types, a more informationally comprehensive input is fed into the VAE encoder. This approach ensures, from the outset, that the latent variables $\mathrm{z}$ can learn a data distribution encompassing both fine details and global context simultaneously, thereby endowing the model with enhanced discriminative capability.

Secondly, the architecture is conceptually derived from the principles of residual learning. A concatenation operation ${\mathrm{X}}^{\mathrm{s}}=[{\mathrm{X}}^{\mathrm{p}}:{\mathrm{X}}^{\mathrm{n}}]$ is established to create a direct “shortcut” from the raw input to the VAE encoder, with this design rationale being structurally comparable to that of residual blocks $\mathrm{y}=\mathrm{F}(\mathrm{x},\mathrm{w})+\mathrm{x}$. This configuration not only facilitates the establishment of a more direct gradient flow path during backpropagation—thereby partially mitigating training challenges—but more importantly, it equips the model with the capacity to dynamically balance information sources. During the training process, the VAE encoder is empowered to autonomously determine, when inferring the latent distribution, whether greater emphasis should be placed on the basic raw signals ${\mathrm{X}}^{\mathrm{p}}$ or higher trust should be assigned to the processed contextual information ${\mathrm{X}}^{\mathrm{n}}$. For this reason, a notable improvement in the model’s adaptability and robustness to diverse anomalous patterns is realized. The detailed structure of this VAE encoder is illustrated in Figure 3.

3.2.3. Interactive Module

This module is implemented using cross-attention, which allows the interaction of information from two different feature spaces in the model [26]. Specifically, it dynamically fuses information in two feature spaces by calculating the similarity between query vectors Q in one feature space and key vectors K and value vectors V in another feature space. This paper validated the effectiveness of this mechanism in ablation experiments. Better detection performance can be achieved when the latent variable $\mathrm{z}$ is used as the query and the Transformer output ${\mathrm{X}}^{\mathrm{n}}$ is adopted as the Key/Value. Therefore, this cross-attention direction is employed in the final model. The network architecture of the cross-attention mechanism is shown in Figure 4.

This mechanism allows for effective fusion of features from the Transformer and VAE encoders, enhancing the model’s capacity to detect anomalies. It enables dynamic information exchange between the two feature spaces and adaptively adjusts feature fusion based on input data. This capability supports the learning of complex feature relationships.

3.2.4. Residual Connections

Residual connections are a core component of deep residual networks (ResNet), introduced by He et al. [27] in 2016. The main idea is to solve the problem of vanishing and exploding gradients by skipping some layers, thereby facilitating the training of deeper models.

Traditional deep neural networks often encounter gradient vanishing or exploding problems as the number of layers increases, making model training challenging. ResNet mitigates these issues by incorporating residual blocks, which enable the network to learn residuals between inputs and outputs rather than directly learning the target output. This design allows inputs to “skip” certain layers and connect to deeper outputs through skip connections, preserving input information and alleviating gradient problems in deep networks.

3.2.5. Residual Connection—Fully Connected Layer Classifier

The fully connected layer classifier uses layer-by-layer feature transformation and nonlinear activation through a multilayer neural network to map input high-dimensional feature vectors to a specified category space, thereby enabling the prediction of sample classes, as illustrated in Figure 5.

Through the processing conducted by each fully connected layer, the initial data is progressively transformed. The final output ${\mathrm{h}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$, a real-valued scalar, is projected onto the [0, 1] interval via the sigmoid activation function to represent the probability of the sample belonging to a specific category. The fully connected layer classifier can effectively utilize the features fused by the cross-attention mechanism to learn the mapping relationship from features to category labels, thus achieving anomaly detection classification.

3.3. Loss Function

The DF-TransVAE uses Binary Cross-Entropy (BCE) loss and Kullback-Leibler (KL) divergence as its loss functions, with their weights optimized as hyperparameters. Among them, ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{B}\mathrm{C}\mathrm{E}}$ directly optimizes the model’s classification performance, allowing it to more accurately predict anomalous samples. ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{K}\mathrm{L}}$ ensures the latent space distribution aligns with the prior distribution, preventing overfitting and improving robustness and generalization. This paper aims to simultaneously optimize ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{B}\mathrm{C}\mathrm{E}}$ and ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{K}\mathrm{L}}$ so that the model can not only accurately classify anomalous samples but also learn meaningful latent representations, thereby achieving better performance in anomaly detection tasks.

For a batch of samples, this is typically computed by summing the BCE loss over all samples and then dividing by the number of samples.

$${Loss}_{BCE}=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\left[y_{i}log\right(p_i)+(1-y_i\left)log\right(1-p_i\left)\right]$$

(10)

The specific formula for KL divergence is

$${Loss}_{KL}={\displaystyle \frac{1}{2}}{\sum }_{i=1}^d({\mathsf{\mu }}_i^2+{\mathsf{\sigma }}_i^2-\mathrm{l}\mathrm{o}\mathrm{g}\left({\mathsf{\sigma }}_i^2\right)-1)$$

(11)

The total loss function after combining the divergence of ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{B}\mathrm{C}\mathrm{E}}$ and ${\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}_{\mathrm{K}\mathrm{L}}$ is

$${Loss}_{total}={Loss}_{BCE}+\mathsf{\beta }{Loss}_{KL}$$

(12)

The parameter $\mathsf{\beta }$ is a hyperparameter used to balance the weights between BCE and KL divergence. It determines how much the model emphasizes the classification task versus latent space regularization during optimization.

By minimizing the loss function value, the model iteratively optimizes its parameters, enhancing performance on the training data while improving generalization ability through regularization and validation set monitoring. Ultimately, the trained model performs well on unseen data, enabling accurate predictions on new data.

4. Experimental Results and Analysis

This section begins with a presentation of the datasets used in the experiments, covering their pre-processing, experimental environment, and evaluation metrics. Subsequently, it details the experimental setup, comparing the proposed DF-TransVAE with baseline methods and existing algorithms, and presents the conclusions. Lastly, both quantitative and qualitative analyses of the results are provided.

4.1. Dataset and Its Preprocessing

In order to test the performance of the model, two public datasets in the field of cyber security are used in this paper. The two public datasets are described below:

Cybersecurity Intrusion Detection Dataset (https://www.kaggle.com/datasets/dnkumars/cybersecurity-intrusion-detection-dataset (accessed on 5 April 2025)): The dataset is derived from Kaggle and focuses on network security intrusion detection. It covers two types of features: network-based features, including packet size (in bytes); communication and encryption protocols; and user-behavior-based features, including the number of login attempts, session duration (in seconds), the number of failed login attempts, abnormal login times, IP trustworthiness, and browser name. The dataset contains nine features (except the id column) and one label column, where the label “1” represents abnormal and “0” represents normal, with a total of 9537 samples. In this paper, these nine features are input into the model as variables. This dataset is hereinafter referred to as the Cybersecurity Dataset.

UGRansome dataset (https://www.kaggle.com/datasets/nkongolo/ugransome-dataset/data (accessed on 5 April 2025)): The dataset is derived from Kaggle, which is a versatile cybersecurity resource focused on analyzing ransomware and zero-day cyberattacks, in particular those attacks that exhibit cyclic smooth behavior. This dataset contains categorization using timestamps, flag types, etc. The dataset contains attack signatures and provides synthetic signatures for testing and modeling cybersecurity defenses. The dataset contains 13 features and one labeled column, where the label “A” stands for anomaly, “S” stands for attack signature, and “SS” stands for synthetic signature, for a total of 149,043 samples. In order to form a standard binary classification task, we unify the labels as follows: A is encoded as 0, while S and SS are encoded as 1. Thus, a binary classification task between basic anomaly and malicious attack signature is constructed. Both classes 0 and 1 are complete, so standard classification metrics such as accuracy, precision, recall, and F1-score can be normally calculated. The number of class 0 and class 1 classifications for the two datasets is shown in Figure 6. In this study, positive samples are defined as samples with the label 0, and negative samples are defined as samples with the label 1.

The datasets utilized in this study are composed of network security parameters, which serve as quantitative representations of internet user behavior. These parameters reflect core behavioral characteristics such as user access frequency, session patterns, request sequences, and operational rhythms, which can effectively capture malicious behaviors including automated bot access, online scams, and fraudulent activities. Therefore, the dataset is highly consistent with the research goal of malicious internet user behavior detection.

For the Cybersecurity Dataset and UGRansom dataset, this study partitions them into training, validation, and test sets according to an 8:1:1 ratio. This approach ensures that the model comprehensively learns data features during training, effectively tunes hyperparameters with the validation set, and accurately assesses generalization capability using the test set. In subsequent research phases, the training set is dedicated to feature and pattern learning. The validation set helps optimize hyperparameters and fine-tune the model. The test set acts as the ultimate evaluator, gauging model’s performance on new data and ensuring its reliability and effectiveness. The data preprocessing module is illustrated in Figure 7.

During the data preprocessing phase, handling missing values is essential as it directly affects the model’s training effectiveness and prediction performance. For the “communication protocol” feature in the Cybersecurity Dataset, there are 1966 missing values, which indicate the absence of encryption technology. Therefore, this paper fills these missing values with “0”. This enables the model to better understand the data, improving training and prediction accuracy.

Both the Cybersecurity Dataset and UGRansome dataset contain non-numeric data columns, which computers cannot directly recognize and process. Thus, this paper converts them into numeric data. Label encoding is employed to map each category to a unique integer, followed by further processing, ultimately serving as input to the model.

Min–Max normalization is applied to normalize the datasets. Same maximum and minimum values are then used to normalize the test set, scaling feature values within the range [0, 1]. The preprocessing of the two datasets is shown in

Table 1. Data preprocessing steps for two datasets, where Y indicates that the operation is performed and N indicates that it is not performed.

Datasets	Redraw	Fill	Coding	Normalization	Feature Engineering
Cybersecurity	Y	Y	Y	Y	N
UGRansome	Y	N	Y	Y	N

.

4.2. Experimental Environment

This paper developed DF-TransVAE in CUDA 12.2 and Torch 2.5.1 environments, using the following hardware: CPU: Intel (R) Core (TM) i5-1035G1 CPU@1.00GHz; GPU: NVIDIA GeForce MX350.

4.3. Evaluation Indicators

This study employed three widely used metrics to assess the proposed model’s predictive performance: ROC-AUC, ACC, and F1-score. ACC is the proportion of correctly predicted normal and anomalous samples among all test samples. An ACC closer to 1 indicates better model performance. F1-score is the harmonic mean of precision and recall. Precision is the proportion of actual anomalous samples among those predicted as anomalous, and recall is the proportion of correctly predicted anomalous samples among all actual anomalous samples. The ROC curve, or Receiver Operating Characteristic curve, helps assess classification performance by showing the relationship between TPR and FPR. Each point on the curve corresponds to a specific threshold: samples are classified as anomalous if their predicted score exceeds this threshold, and normal otherwise.

4.4. Experimental Setup and Comparative Model

The hyperparameter setup for the DF-TransVAE was determined using Bayesian hyperparameter optimization, which efficiently locates the optimal configuration compared to grid and random searches. In this paper, ACC was chosen as the validation metric, and the Bayesian optimization algorithm selected the hyperparameter set delivering the highest ACC as the optimal configuration for the model. A single search process was conducted on the Cybersecurity Dataset to obtain the best hyperparameter configuration for this dataset. The search ranges for all hyperparameters were set based on empirical knowledge. For example, d_mode was set to {16, 32, 64, 128, 256, 512, 1024}; n_head was set to {4, 6, 8, 10, 12}; num_layers was set to {4, 6, 8, 10, 12}; latent_dim was set to {32, 64,128, 256}; num_heads was set to {4, 6, 8, 10}; $\mathsf{\beta }$ was set to {0.01, 0.05, 0.1, 0.2, 0.3}; lr was set to {$5\times {10}^{-3}$, $2\times {10}^{-3}$, $1\times {10}^{-3}$, $5\times {10}^{-4}$, $2\times {10}^{-4}$, $1\times {10}^{-4}$}; and dropout rate was set to {0.1, 0.2, 0.3, 0.4, 0.5, 0.6}. The optimal hyperparameter configuration of the DF-TransVAE on the dataset is shown in

Table 2. Hyperparameter configuration for DF-TransVAE.

Hyperparameter	Numerical Value
d_model	256
nhead	8
num_layers	4
latent_dim	64
num_heads	4
β	0.1
lr	0.0001
threshold	0.5

.

To assess DF-TransVAE’s performance, experiments were conducted on two publicly available datasets. The baseline methods used in this study were CNN [28], DNN [28], and CNN-LSTM [28], while the latest research methods included Transformer-VAE [2], SS-DeepCAE [29], DLTML-RTADPM [30], and Deep-Belief-MobileNet 1D [31]. To guarantee a fair experimental comparison, Bayesian optimization was utilized to fine-tune the hyperparameters of the comparison algorithms. To reduce the influence of randomness on the experimental outcomes, each experiment was executed five times independently, with the mean value of these trials being documented. The results are shown in

Table 3. Anomaly detection results for all methods across two datasets, bolding indicates best, underlining indicates second best.

Model	Cybersecurity Dataset			UGRansome Dataset
	ROC-AUC%	ACC/%	F1-Score/%	ROC-AUC%	ACC/%	F1-Score/%
DNN [28]	87.86	86.57	82.66	99.87	98.45	98.92
CNN [28]	86.04	83	78.91	98.44	94.79	96.40
CNN-LSTM [28]	83.76	80.27	76.50	99.69	97.57	98.30
SS-DeepCAE [29]	87.50	86.36	82.09	99.79	98.30	98.81
DLTML-RTADPM [30]	79.20	76.29	70.65	82.36	86.54	90.65
Deep-Belief-MobileNet 1D [31]	83.01	81.32	76.88	99.93	98.71	99.09
Transformer-VAE [2]	62.98	61.70	62.25	74.87	69.34	76.71
DF-TransVAE	88.30	89.93	87.13	99.95	98.94	99.26

. Bolding indicates best, underlining indicates second best. All the code used in this study, including model implementation and experimental procedures, is publicly accessible at https://github.com/Fhh-1/DF-TransVAE (accessed on 10 February 2026 ) to ensure the reproducibility of the reported results.

According to Table 3, the proposed DF-TransVAE demonstrated superior anomaly detection performance on both datasets, ranking first in ROC-AUC, ACC, and F1-score metrics. On the Cybersecurity Dataset, the DF-TransVAE model is shown to achieve improvements in ROC-AUC of 9.1%, 5.29%, and 25.32% compared to the state-of-the-art DLTML-RTADPM, Deep Belief-MobileNet 1D, and Transformer–VAE models, respectively. Accuracy is enhanced by 9.66%, 13.64%, and 28.23% over the CNN-LSTM, DLTML-RTADPM, and Transformer–VAE models. The F1-score is improved by 10.63%, 16.48% and 24.88% compared to the CNN-LSTM model, DLTML-RTADPM model, and Transformer-VAE, respectively. On the UGRansome dataset, the DF-TransVAE improved the ROC-AUC metric by 17.59% and 25.08%, the ACC metric by 12.4% and 29.6%, and the F1-score by 8.61% and 22.55% compared to the DLTML-RTADPM and Transformer–VAE models.

The suboptimal performance of the Transformer–VAE model on both datasets can be primarily attributed to its training objective, which relies exclusively on reconstruction loss and KL divergence without direct optimization for classification performance. Although this design facilitates learning latent data representations, it may be insufficient for capturing discriminative features essential for classification tasks. Furthermore, model efficacy can be compromised when the true data distribution diverges from the Gaussian prior assumed by the VAE, potentially hindering effective learning of the latent structure. For this reason, for these specific datasets, Transformer–VAE may fail to learn adequately discriminative feature representations for class separation. In contrast, the DF-TransVAE model proposed in this work, which deeply integrates Transformer with VAE, is validated to deliver advanced anomaly detection capabilities on both datasets.

The learned feature space of the model is visualized for qualitative assessment of its classification capability. The Uniform Manifold Approximation and Projection (UMAP) technique is employed for visualization generation owing to its capacity to preserve both local and global data structures while retaining computational efficiency. Specifically, the feature vectors preceding the final classification layer, together with their predicted labels, are extracted and then projected into a two-dimensional space via UMAP. During the visualization process, UMAP parameters are set as follows: n_neighbors = 15, min_dist = 0.1, and a fixed random seed is applied to guarantee reproducibility. The resulting visualizations of the two datasets are displayed in Figure 8 and Figure 9, respectively. By comparing the compactness and separation of feature clusters generated by different methods, the relative advantages of each approach regarding classification performance can be intuitively evaluated.

Figure 8 presents the 2D UMAP clustering results of eight comparison methods on the Cybersecurity dataset, while Figure 9 shows the corresponding results on the UGRansome dataset. In both figures, green points represent positive samples, and red points represent negative samples. Each subplot corresponds to a specific model: (a) DNN, (b) CNN, (c) CNN-LSTM, (d) SS-DeepCAE, (e) DLTML-RTADPM, (f) Deep-Belief-MobileNet 1D, (g) Transformer–VAE, and (h) the proposed DF-TransVAE.

On the Cybersecurity dataset (Figure 8), DNN (a) achieves clear separation between normal and abnormal samples: green (normal) samples form a compact cluster in the lower-right region, while red (abnormal) samples are concentrated in the upper left with minimal overlap, indicating that even a basic DNN can capture distinct patterns for this dataset. CNN (b) shows severe class entanglement, with green and red samples intermingled across multiple scattered clusters, reflecting its limited ability to disentangle complex behavioral features. CNN-LSTM (c) does not improve separability; instead, normal and abnormal samples remain heavily mixed in intertwined clusters, suggesting that sequential modeling alone is insufficient to resolve feature ambiguity in this context. SS-DeepCAE (d) demonstrates strong separability: red (abnormal) samples form a long, continuous chain-like structure with minimal overlap, while green (normal) samples appear as distinct, small clusters in separate regions, indicating effective feature compression and separation. DLTML-RTADPM (e) achieves clear class separation, with green (normal) samples forming a large, continuous arc-shaped cluster and red (abnormal) samples forming a smaller, distinct arc, with almost no overlap between the two. Deep-Belief-MobileNet 1D (f) also shows strong separability: green (normal) samples form a large, continuous S-shaped cluster, while red (abnormal) samples form a smaller, distinct S-shaped cluster, with minimal overlap, indicating effective feature extraction and separation. Transformer–VAE (g) exhibits severe class entanglement, with green and red samples uniformly mixed into a single, dense cluster, reflecting its failure to learn discriminative representations that distinguish normal from abnormal behavior. In contrast, DF-TransVAE (h) demonstrates the most optimal clustering performance: green (normal) samples form a tight, well-defined cluster in the upper right, while red (abnormal) samples form a compact, distinct cluster in the lower left, with no overlap between the two, confirming the model’s ability to learn a highly discriminative feature manifold for anomaly detection.

On the UGRansome dataset (Figure 9), which contains more-complex ransomware and attack signatures, traditional models (DNN, CNN, CNN-LSTM) show increased class overlap, as the dataset’s higher dimensionality and attack diversity challenge their feature extraction capabilities. DLTML-RTADPM (e) maintains strong visual separability, with normal and abnormal samples forming distinct clusters, but its quantitative metrics are slightly lower than CNN/CNN-LSTM, which we attribute to a suboptimal mapping from the well-structured feature space to the final decision boundary. Transformer–VAE (g) again fails to separate classes, with samples from both categories uniformly mixed, confirming its limitations in capturing the complex distributional patterns of malicious behavior. Notably, DF-TransVAE (h) retains its superior clustering performance, with clear separation between normal and abnormal samples across both datasets, highlighting the robustness of its feature representation to varying dataset characteristics.

Existing hybrid models, such as Transformer–-VAE [2], rely on loose coupling of components, leading to insufficient feature interaction and poor clustering performance. In contrast, DF-TransVAE’s deep fusion of Transformer and VAE through cross-attention enables the learning of a more structured feature manifold, directly improving clustering quality. While DLTML-RTADPM achieves strong visual separability, its suboptimal quantitative performance highlights a notable trade-off: a well-structured feature space is necessary but not sufficient for high classification accuracy. DF-TransVAE addresses this by ensuring both high-quality feature representation and efficient decision boundary mapping, as evidenced by its leading quantitative metrics and superior clustering visualization.

The clustering results demonstrate that DF-TransVAE possesses strong generalizability in clustering tasks. It maintains consistent, high-quality clustering on both the Cybersecurity and UGRansome datasets, which differ in feature dimensions, attack types, and sample distributions. The model learns a feature space where normal samples form tight clusters and abnormal samples are distinctly separated, enhancing both the interpretability and robustness of anomaly detection. Moreover, the clustering results directly support the anomaly detection task by providing a clear decision boundary, reducing the risk of overfitting to local noise in the data. In summary, the clustering analysis confirms that DF-TransVAE successfully integrates the advantages of both high-quality feature representation and precise decision-making. By fostering deep interaction between global contextual representations and latent probability distributions, the model learns an inherently separable feature manifold, leading to comprehensively superior performance in both clustering and anomaly detection tasks.

The experimental results on both the Cybersecurity and UGRansome datasets consistently demonstrate the superior performance of the proposed DF-TransVAE model compared with existing methods. Quantitatively, DF-TransVAE achieves the highest accuracy, F1-score, and AUC among all compared models, which verifies its strong capability in distinguishing normal user behavior from malicious anomalies. Qualitatively, the UMAP clustering visualizations in Figure 8 and Figure 9 further illustrate that DF-TransVAE learns a highly discriminative feature manifold, in which normal samples form compact and clear clusters while abnormal samples are well separated with little overlap. Such performance is in sharp contrast to traditional models including DNN and CNN, which suffer from serious class entanglement, and even advanced hybrid models such as Transformer–VAE, which cannot learn effective discriminative representations for anomaly detection tasks. Compared with existing Transformer+VAE models [2], the superiority of DF-TransVAE is more prominent. Previous hybrid schemes only adopt loose coupling between the Transformer and VAE modules, resulting in insufficient feature interaction and limited detection performance. In contrast, DF-TransVAE employs a cross-attention-based deep fusion mechanism, which supports closed-loop collaborative learning between the contextual representations of the Transformer and the latent distribution of the VAE. Such deep integration enables the model to capture both global temporal dependencies and local distribution features, thus forming a more compact and discriminative feature space. Both quantitative metrics and qualitative clustering results confirm that this structure effectively overcomes the limitations of existing hybrid models and improves the performance of user behavior anomaly detection.

4.5. Ablation Experiment

This ablation study is designed to enable in-depth analysis of the contribution of each component within the DF-TransVAE framework. The primary objectives are to determine which component acts as the primary driver of performance enhancement and to clarify the contributions and potential redundancies of various feature fusion strategies. Specifically, disabling “concatenation” implies that the raw data is directly input into the VAE encoder; disabling “Transformer” indicates the Transformer encoder is not used; disabling “VAE” means the VAE encoder is excluded; disabling “interaction” implies that $\mathrm{z}$ and ${\mathrm{X}}^{\mathrm{n}}$ are not concatenated along the feature dimension; disabling the “classifier” indicates that the classifier module is replaced with a single, fully connected layer. Model_9 is designed to verify the direction of the cross-attention mechanism, in which the output of the Transformer is adopted as the query and the latent variable is utilized as the Key/Value.

The ablation study results for anomaly detection on the Cybersecurity Dataset are presented in

Table 4. Results of ablation experiments for abnormal detection. √ indicates that this module is used, bolding indicates best, underlining indicates second best.

Model	Component					Cybersecurity Dataset
	Splicing	Transformer	VAE	Cross Attention	Classifier	ROC- AUC/%	ACC /%	F1-Score /%
Model_1		√	√	√	√	87.33	85.52	82.58
Model_2	√	√	√		√	88.92	89.51	86.52
Model_3	√	√	√	√		88.11	86.99	83.24
Model_4		√	√		√	88.95	86.15	82.81
Model_5	√	√	√			87.77	89.09	86.21
Model_6		√	√	√		89.23	88.98	85.91
Model_7			√		√	85.89	82.06	78.76
Model_8		√			√	88.10	89.61	86.82
Model_9	√	√	√	√	√	88.27	89.49	86.61
DF-TransVAE	√	√	√	√	√	88.30	89.93	87.13

. The optimal detection results are marked in bold, while the next-best results are indicated with a strikethrough. All comparison methods were manually reproduced under identical conditions. One particularly notable finding is revealed through the analysis of the tabulated data: all models incorporating a Transformer encoder (Models_1–6, 8) consistently demonstrate notably better performance compared to the VAE-only baseline Model_7 (contains only VAE and classifiers). This observation is precisely consistent with our introductory discussion regarding the indispensable role of Transformers in capturing complex global dependencies within user behavior. For instance, Model_8 (No splicing)—where only the Transformer component is introduced—achieves a notable improvement of over seven percentage points in primary metrics relative to Model_7 (contains only VAE and classifiers), thereby offering strong evidence that the integration of strong contextual modeling capabilities serves as the core foundation for the model’s success.

Following the confirmation of the Transformer’s central importance, the role of feature fusion strategies is further examined. When Model_1 (No splicing) is compared against models incorporating concatenation (e.g., Models_2, 8, here, 2 indicates no cross-attention, while 8 denotes a model containing only a Transformer and a classifier.), consistent performance gains are observed across all evaluation metrics. This outcome validates our original design premise: the concatenation of raw features with abstract Transformer-derived features effectively enables the complementary integration of local details and global context, which represents a clearly beneficial enhancement strategy.

The complete DF-TransVAE model—incorporating the cross-attention mechanismreaches optimal performance among all ablated variants, particularly in terms of F1-score. This indicates that the cross-attention mechanism facilitates more refined and adaptive feature interactions compared to simple concatenation, thereby contributing to the elevation of the model’s performance ceiling. However, the closely matched performance between Model_4 (with cross-attention but without concatenation) and the complete model suggests partial information overlap between the representations extracted by the cross-attention mechanism and those inherent to the Transformer itself. For this reason, its contribution may be characterized as a marginal optimization that offers additional refinement to an already robust foundation.

In conclusion, a clear functional hierarchy is revealed by the ablation study: the Transformer is identified as the core driver of performance, tasked with capturing global patterns; feature concatenation is positioned as a key enhancement module, boosting robustness; and cross-attention functions as a high-level optimizer, facilitating the refinement of the model’s performance potential through deep feature interactions, thereby bringing the model’s performance closer to its theoretical upper limit.

For this reason, while not all modules are strictly indispensable, they are collectively integrated into a high-performance system featuring a collaborative hierarchical structure that evolves from foundational capabilities to enhanced robustness, and ultimately to refined optimization.

The direction setting of the cross-attention mechanism is further verified. In this verification, the output features of the Transformer encoder are adopted as the query vector, and the latent variables of the VAE encoder are utilized as the key and value vectors for Model_9. By contrast, the latent variable $\mathrm{z}$ of the VAE encoder is employed as the query vector, and the output $X^n$ of the Transformer encoder is used as the key and value vectors for the proposed DF-TransVAE. The comparative results demonstrate that superior performance is achieved by DF-TransVAE in terms of detection accuracy, F1-score, and ROC-AUC.

4.6. Hyperparameter Analysis

During the construction of DF-TransVAE, several hyperparameters require careful tuning, as their configurations can affect model performance. Analyzing these hyperparameters is essential to understanding their influence on DF-TransVAE’s effectiveness. Figure 10 illustrates the impact of varying values of d_mode on the anomaly detection performance of DF-TransVAE, where d_mode takes values from the set {16, 32, 64, 128, 256, 512, 1024}. In DF-TransVAE anomaly detection, the dimensionality of z within the Transformer encoder plays a central role in determining detection outcomes. When latent space dimensionality is excessively large, the concatenated feature vector—comprising the Transformer-extracted features and the original input—becomes high-dimensional. This increased dimensionality can make it challenging for the VAE encoder to effectively capture the underlying structure and distribution of data, ultimately degrading the quality of the learned representations. Conversely, when latent space dimensionality is too small, the Transformer encoder applies a higher degree of compression to the input sequence, potentially failing to capture complex relationships and diverse patterns within the sequence. As a result, the fused features passed to the VAE encoder may contain insufficient information, which can hinder the VAE’s ability to learn a comprehensive representation of data structure and distribution. This, in turn, limits the expressiveness and accuracy of the learned latent variables. According to the results shown in Figure 10, DF-TransVAE achieves the highest accuracy when d_mode is set to 256, while the ROC-AUC reaches its second-highest value under this configuration.

5. Conclusions

This paper offers a novel study on user behavior anomaly detection. It introduces a new deep learning framework called DF-TransVAE, which combines the Transformer’s sequence modeling with the VAE’s latent space modeling. By inputting fused features into the VAE, DF-TransVAE gains richer information and stronger representational power. Additionally, a cross-attention mechanism is included. This mechanism enhances DF-TransVAE’s ability to detect anomalies, allows dynamic information exchange between feature spaces, and adjusts feature fusion based on input data. It also captures complex inter-feature relationships, improving anomaly detection’s accuracy and robustness. The framework achieves deep feature extraction and efficient anomaly classification. Experimental results from two benchmark datasets show the proposed model’s effectiveness and robustness in user behavior anomaly detection.

Compared to other methods for user behavior anomaly detection, the proposed DF-TransVAE demonstrates significant advantages. By performing deep feature extraction and effectively integrating both the long-term dependencies and intrinsic structural characteristics of the data, DF-TransVAE achieves more-accurate anomaly detection. Furthermore, DF-TransVAE exhibits strong generalizability, making it applicable to a wide range of scenarios that require deep feature representation.

Specifically, its superior discriminative capability, interpretable feature representation, and robust performance make it well suited for integration into real-world cybersecurity scenarios. These include enterprise Security Information and Event Management (SIEM) systems for insider threat detection, cloud-based user behavior analytics platforms for multi-tenant security, and (with future efficiency optimizations) IoT edge devices for real-time anomaly monitoring. Such integration can fully leverage the model’s strengths to address practical security challenges.

Nevertheless, potential directions for enhancement remain in the DF-TransVAE model. Its performance is highly dependent on the quality and representativeness of the training data, as well as the coverage of anomalous cases. A decision boundary is established by the model through supervised learning within a joint feature space. If the normal samples in the training data fail to fully cover all potential behavioral patterns, the representations of some unseen normal cases may be positioned on the “anomalous” side of the learned decision boundary in the feature space, resulting in misclassification. This phenomenon is underscored to reflect the model’s inherent dependency on the training data distribution. For this reason, exploring methods to improve the model’s open-set recognition capabilities and out-of-distribution detection robustness in practical scenarios with high annotation costs or class imbalance deserves further in-depth investigation. In addition, while the proposed DF-TransVAE demonstrates superior performance in anomaly detection, future work will also focus on optimizing its timing characteristics and cost-effectiveness for practical deployment, including profiling the model’s inference latency and computational overhead on diverse hardware platforms (e.g., edge devices, cloud servers), exploring model compression techniques (e.g., quantization, pruning) to reduce memory usage and improve efficiency without significant performance degradation, and conducting a comprehensive cost–benefit analysis to compare DF-TransVAE with lightweight anomaly detection models to ensure its viability for real-time cybersecurity applications.