Dual-Branch Network for Video Anomaly Detection Based on Feature Fusion

Abstract

Anomaly detection is a critical task in video surveillance, with significant applications in the management and prevention of criminal activities. Traditional convolutional neural networks often struggle with motion modeling and multi-scale feature fusion due to their localized field of view. To address these limitations, this work proposes a Dual-Branch Interactive Feature Fusion Network (DBIFF-Net). DBIFF-Net integrates a CNN branch and a swin transformer branch to extract multi-scale features. To optimize these features for efficient fusion, an interactive fusion module is introduced to efficiently fuse these multi-scale features through skip connections. Then, the temporal shift module is employed to exploit dependencies between video frames, thereby improving the identification of anomalous events. Finally, the channel attention is utilized for decoder to better assist in restoring complex object features in the video. System performance is evaluated on three standard benchmark datasets. DBIFF-Net achieves the area under the receiver operating characteristic (AUC) of 97.7%, 84.5%, and 73.8% on the UCSD ped2, CUHK Avenue, and ShanghaiTech Campus dataset, respectively. Extensive experiments demonstrate that DBIFF-Net outperforms most state-of-the-art methods, validating the effectiveness of our method.

1. Introduction

With the widespread adoption of video surveillance systems in public security, intelligent transportation, and urban management, video anomaly detection (VAD) has emerged as a critical task. It enables early warning, rapid response, and post-event analysis of criminal activities [1,2,3,4]. Compared to conventional surveillance, which heavily relies on manual monitoring, deep learning-based VAD algorithms can automatically identify anomalies. This technology effectively reduces the burden on operators while enhancing overall monitoring efficiency [5,6].

Due to the low incidence and diversity of anomalies, it is challenging to comprehensively label all anomalous patterns in real-world surveillance videos [7,8]. Therefore, the VAD is frequently addressed as a one-class classification (OCC) problem [9,10]. In this problem, models are trained solely on normal video data, and they identified anomalies during inference [11]. Furthermore, video data possess intrinsic temporal dependencies that are absent in static images [12,13]. This necessitates models that can simultaneously capture both the spatial details within individual frames and the dynamic changes across a sequence, thereby increasing the complexity of the VAD task. Early approaches to VAD relied on traditional machine learning techniques, which model normal events through hand-crafted features and shallow models. For example, Chuang et al. [14] utilized spatio-temporal ratio histograms to represent video features, with anomalies detected via a finite-state machine (FSM) model. Mo et al. [15] propose a joint sparsity model with kernelization for multi-object anomaly detection, employing a greedy pursuit algorithm and linear sparsity to enhance class separability. Other methods, including the support vector machine (SVM) [16], the random forest (RF) [17], and the probabilistic-based model [18], have also achieved notable success in VAD. However, these traditional methods are significantly limited because they depend on manual feature engineering, which is both inefficient and difficult to capture the full complexity of data.

In recent years, deep learning technology has developed rapidly, effectively addressing the limitations of traditional machine learning. A prevalent deep learning approach for OCC is the use of reconstruction-based Auto-Encoders (AEs) [19,20]. These methods are generally trained by minimizing reconstruction error on normal video data and subsequently use reconstruction error as a measure of anomalies. However, these methods continue to face several key challenges. Several recent studies have sought to address this issue from different perspectives. For instance, Liu et al. [21] found that while existing VAD methods leverage multi-modal inputs to maintain robustness in complex scenes, they suffer from high computational costs, substantial storage demands, and strong multi-modal dependencies. To address this, they proposed a semi-supervised knowledge distillation framework that preserves the advantages of multi-modal learning during training while requiring only single-modal data for inference. Jiang et al. [22] pointed out that existing methods for VAD often overlook the properties of motion features, leading to a failure to effectively integrate appearance and motion cues. They introduced a multi-scale differential perception network to enhance motion representation in the appearance branch and eliminate redundant information through fine-grained deep feature processing. Qiu et al. [23] argued that the blurred boundary between normal and abnormal video data leads to low detection accuracy. They introduced a convolution-enhanced self-attentive video auto-encoder based on U-Net, coupled with a dual-scale clustering module, that enhances feature representation, amplifies the normal-abnormal distinction, and provides robust anomaly scoring against temporal leakage. In addition, some studies have introduced the two-stream networks to enhance models’ ability to perceive anomalies in both spatial and temporal stream [24]. However, the introduction of optical flow requires its explicit extraction, which complicates the training process and poses a non-trivial challenge in modeling inter-stream interactions. Meanwhile, other works have explored hybrid architectures that combine local convolutional features with global attention mechanisms to better characterize the spatio-temporal dynamics of abnormal behaviors [25]. Despite their advancements, these methods still struggle to learn fine-grained, localized appearance anomalies. These fine-grained abnormal features are not visually prominent, but they are widespread in real-world surveillance scenarios and are of significant value for detecting anomalies.

To address these limitations, this study proposes the Dual-Branch Interactive Feature Fusion Network (DBIFF-Net), a novel unsupervised framework for video anomaly detection. By leveraging the advantages of the convolutional neural network (CNN) and the swin transformer [26], DBIFF-Net achieves precise prediction for anomalies. Its CNN branch captures detailed textures and structures at multiple spatial resolutions. Its swin transformer branch focuses on modeling human movements, localization, and global information through window-based self-attention. Subsequently, DBIFF-Net also integrates three components to enhance its performance: interactive fusion module, a temporal shift module (TSM) and the channel attention. The interactive fusion module employs an interactive attention mechanism to aggregate multi-scale features across different receptive fields from both branches. TSM [27] is incorporated to model temporal dependencies by shifting feature channels across consecutive frames. The decoder incorporates channel attention gates at each upsampling stage to better exploit the channel relationship of the features for image features restoration. The main contributions are summarized as follows:

• This work presents DBIFF-Net, a new video anomaly detection framework that tightly couples a CNN-based local encoder with a swin transformer–based global encoder. Unlike prior dual-branch architectures that simply concatenate or sum multi-scale features, DBIFF-Net is designed to jointly capture fine-grained spatial cues and long-range temporal dependencies through interactive fusion, leading to more discriminative multi-scale representations.
• This work proposes an interactive fusion module to facilitate fusion of multi-scale features between local and global encoders. The module performs dynamic, cross-scale interaction, allowing features from different semantic levels and branches to reinforce each other and significantly enhance representation quality.
• This work proposes an attention decoder module that employs the channel attention to explicitly models inter-channel dependencies during upsampling, thereby improving the recovery of complex object features.
• Comprehensive experiments show that DBIFF-Net surpasses most methods across three benchmark datasets, and extensive ablation studies validate the necessity and effectiveness of each module, confirming the soundness of the overall architectural design.

2. Related Work

Over the past years, a significant amount of research has been dedicated to addressing the VAD problems. This section will focus on three categories of models: AE-based models, dual-stream models, and multi-scale features models.

Due to the scarcity and diversity of anomalies in real-world scenarios, unsupervised reconstruction-based models have been widely applied in VAD [28,29]. Among these, many studies leverage the AE as the core of their detection framework. These frameworks aim to reconstruct future frames or masked frames using normal training data. Anomalies are then detected during inference by identifying frames with high reconstruction errors. For instance, a few works utilize convolutional AEs and their variants to reconstruct frames. The Mem-AE method [30] adopted a memory module to enhance memorization of normal patterns, which helps to reduce the improper reconstruction of anomalies and thus improves detection performance. The ML-MemAE-SC method [31] focused on predictive reconstruction, where the model learns to predict the next frame from a sequence of preceding frames. Furthermore, the ST-MemAE method [32] improved the memory structure by preserving spatio-temporal correlation features, leading to stable and accurate anomaly detection results on major benchmark datasets.

To address the limitations of autoencoders in jointly capturing local details and global context, visual transformers have emerged as an effective solution. For instance, Yuan et al. [33] presented trans-anomaly, a framework that combines a U-Net with a video vision transformer. The framework is designed to capture richer temporal information and broader global contexts, leading to improved anomaly detection performance. Lee et al. [34] proposed AnoViT, a vision transformer-based encoder-decoder model. The model utilized an attention operation to learn global relationship between patches, thereby overcoming the local-feature limitation of traditional convolutional encoder-decoders. Beyond visual transformers, several recent studies have employed dual-stream encoder structures to model the relationship between appearance and motion information in recent years. For example, Cai et al. in [35] proposed AMMC-Net, a dual-stream network that uses a memory mechanism to fuse information from two modalities. Thakare et al. [36] proposed an end-to-end multi-stream architecture for anomaly detection. This architecture employed two parallel spatio-temporal deep feature extractors and a fuzzy aggregation method to fuse the anomaly scores. While these architectures enhance the expressive power of motion features by using parallel networks for optical flow and RGB features, they often incur significant computational costs and data processing overheads. To mitigate the issue of cost, recent studies have proposed using deep transfer learning (DTL) and deep domain adaptation (DDA) to alleviate this problem and have achieved a series of results [37].

The effective fusion of multi-scale features is crucial for anomaly detection, as it enables models to capture both fine-grained disturbances and overall pattern deviations. Common fusion strategies include element-wise summation, concatenation, and attention-based weighting mechanisms. For instance, the ASTNet method [38] employed the residual attention fusion and channel classification gates to aggregate features from temporal and spatial branches. Another class of methods employed feature alignment loss, which explicitly constrains the consistency of the embedding representations generated by different branches (e.g., spatial and temporal branches) within the feature space. By aligning these representations, such methods facilitate more effective collaborative fusion of multimodal or multi-stream information. The CMTSTFN method [39] attempted to guide networks to simultaneously focus on large-scale scene changes and small-scale target details by aligning global and local feature representations. This process enhanced the robustness and accuracy of its anomaly detection. However, most existing fusion methods are still limited by their reliance on single-resolution or shallow operations.

Different from existing methods, this work proposes a dual-branch architecture that applies swin transformer blocks and CNN blocks to the encoder and research multi-scale feature fusion. We believe this dual-branch mechanism facilitates stronger performance in video anomaly detection through multi-scale feature fusion.

3. Methods

The overall architecture of the DBIFF-NET model is shown in the Figure 1. The model consists of two main components: a dual-branch encoder and a decoder. The dual-branch encoder extracts features from the input image and generates global and local feature map sets. To effectively integrate information across different receptive fields, an interactive fusion module combines shallow-level features with deep-level ones. This fusion is crucial because features from convolutional layers at different depths exhibit significant differences. Deeper network layers possess larger receptive fields and extract more abstract features, resulting in the loss of image details. By combining features from different depths, the model can capture more information. Finally, the decoder processes the three groups of fused features through an attention mechanism to predict the corresponding future frame. Due to the scarcity of datasets containing anomalous events, the model adopts an unsupervised approach to detect abnormal activities in videos.

3.1. Dual-Branch Encoder

This encoder comprises two parallel networks: a CNN local branch and a swin transformer global branch. The input of the dual branch encoder is a video frame sequence $I=\left\{I_t\right|I_t\in {\mathbb{R}}^{C\times H\times W}\}$, where t is the length of video frame, and C, H, and W denote the number of channels, height, and width of the video frames, respectively. Following prior research, the last frame of the video sequence is designated as the key frame $I^k$ to compute the difference from the predicted frame, while all other frames are concurrently fed into the dual branch encoder. For the convenience of introducing the model structure, we take encoding one frame as an example.

3.1.1. CNN Local Branch

The local feature branch is designed to automatically learn spatial hierarchies within frame through adaptive feature extraction. Furthermore, the convolutional layers apply learnable filters across local regions, enabling effective capture of local patterns.

This module is organized into three stages, where each stage includes two convolutional layers, an activation layer, and a pooling layer. The output feature at the n-th stage is defined as the following:

$$X_n^C=CN{N}_n\left(X_{n-1}^C\right), n=1,2,3$$

(1)

where $X_0^C\in I\setminus \left\{I_P\right\}$ is an input frame, $X_n^C$ represents the output feature of the n-th stage.

3.1.2. Swin Transformer Global Branch

The swin transformer is utilized for capturing the global information for frame by a window-based self-attention mechanism. Its shifted window design enables cross-window feature integration, effectively modeling long-range dependencies while maintaining computational efficiency.

The swin transformer module likewise comprises three stages. The first stage consist of patch partition, swin embedding, and a swin transformer block, while the second and third stages each include a patch merging operation and a swin transformer block. Specifically, each input frame $I_t\in {\mathbb{R}}^{C\times H\times W}$ is first partitioned into 2D patches of size $I_t^p\in {\mathbb{R}}^{p^{2}C\times (⌈{\displaystyle \frac{H}{p}}⌉\times ⌈{\displaystyle \frac{W}{p}}⌉)}$, where P denotes the patch size (typically set to four), as defined in Equation (3). These patches are then fed into the swin transformer block through a linear embedding layer, which corresponds to the first stage. As illustrated in Figure 2, the block incorporates window-based multi-head self-attention (W-MSA) and shifted window-based multi-head self-attention (SW-MSA) to facilitate inter-window feature propagation. Subsequently, the output features are downsampled via a patch merging operation, and the next stage features are generated through the swin transformer block. the patch merging operation can reduce spatial resolution while enlarging the receptive field, thereby enabling the model to capture broader contextual information at deeper stages. The output of the swin transformer module at the n-th stage, $X_n^{ST}$, can be derived as the following:

$$X_0^{ST}=MLP\left(P^P\left(I_t\right)\right), I_t\in I\setminus \left\{I_P\right\}$$

(2)

$$X_1^{ST}=T_1\left(X_0^{ST}\right)$$

(3)

$$X_n^{ST}=T_n\left(P_n^M\left(X_1^{ST}\right)\right)n=2,3$$

(4)

where $P^P$ is the patch partition operation, $P_n^M$ is the patch merging operation, and $T_n$ is the swin transformer block.

3.2. Decoder

Symmetrically to the encoder part, the decoder of the DBIFF-NET also comprises three stages. The output from each stage of the encoder is fed into the corresponding interactive fusion module (LDIF, MDIF, and HDIF). Each of interactive fusion module in turn is followed by an attention decoder module that performs upsampling. This design implements skip connections between encoder and decoder stages, which preserves richer spatial details, thereby enabling the decoder to restore image details more effectively during upsampling.

3.2.1. Interactive Fusion Module

Following the dual-branch encoder, we obtain multi-scale output features of one frame $X_n^C$ and $X_n^{ST}$. How to fuse these features is a difficult and crucial task. To fully mine and utilize both global and local feature information during fusion, this work designs an interactive fusion module, as shown in the Figure 3. This module employs the global feature from swin transformer layer as weight vectors and those from the CNN layer as value vectors, executing an attention mechanism to facilitate dynamic interaction between features. Based on different encoding stages, the interactive fusion block is divided the low-dimensional interactive fusion block (LDIF), mid-dimensional interactive fusion block (MDIF), and high-dimensional interactive fusion block (HDIF), respectively. The process is described as the following:

$${\sigma }_n=\mathrm{Softmax}\left(ML{P}_n\left(X_n^{ST}\right)\right)$$

(5)

$$X_n^{fused}={\sigma }_n\odot Con{v}_{1\times 1}\left(X_n^C\right)\oplus X_n^{ST} n=3,2,1$$

(6)

where $MLP(·)$ and $Con{v}_{1\times 1}(·)$ are used to adjust the feature size, ⊙ is Hadamard product, and ⊕ is element-addition operation. $X_1^{fused}$, $X_2^{fused}$, and $X_3^{fused}$ represent the low-dimensional fused feature, the mid-dimensional fused feature and the high-dimensional fused feature, respectively.

Furthermore, Section 4.5.2 presents a comprehensive experimental analysis that systematically evaluates multiple other fusion strategies, including element-wise addition, Q-attention, and feature concatenation.

3.2.2. Temporal Shift Module

After HDIF, the high-dimensional fused feature $X_3^{fused}$ is fed into the Temporal Shift Module (TSM) to learn temporal information across video frames. TSM is a technique for efficient video understanding, which performs translation and permutation of partial channel features between adjacent frames, enabling the results of each layer have channel values between adjacent frames. Specifically, the operation process of TSM is shown in the Figure 4. Firstly, consecutive frame features are concatenated along the time dimension to form the input feature of TSM, denoted as ${\widehat{X}}^H\in {\mathbb{R}}^{T\times C\times H\times W}$. Then, a portion of channel features in each frame is shifted to the next frame, while the corresponding portion of the channel features in the first frame is zero-padded. As shown in the figure, after temporal shift, the green part of frame $I_2$ are replaced by the red part of frame $I_1$, while the original first two channel features of frame $I_1$ are zero-padded. This mechanism allows channels shifted backward in time to inject past information into the current frame’s computations, while channels shifted forward introduce information from future frames. Meanwhile, the remaining channels remain unshifted to preserve the spatial characteristics of each frame. This dynamic redistribution of features between adjacent frames is applied prior to the convolution operation within the residual block, thereby enriching temporal context and improving the model’s effectiveness in temporal modeling tasks. The TSM is computed as the following:

$${\widehat{X}}_3^{fused}=concat\left(\left\{X_{3_t}^{fused}|t\in \{1,2,3,4\}\right\}\right)$$

(7)

$$X^{TSM}=TSM({\widehat{X}}_3^{fused})$$

(8)

where $X_{I_t}^H$ is high-dimensional fusion features of $I_t$ frame.

3.2.3. Attention Decoder Module

In order to enhance the sensitivity of neural networks to key features, a channel attention mechanism is integrated into decoder. The detailed structure of this attention decoder module is shown in Figure 5. This process can be represented by the following formula:

$${\widehat{X}}_n^{fused}=concat\left(\left\{X_{n_t}^{fused}|t\in \{1,2,3,4\}\right\}\right) n=3,2,1$$

(9)

$${\widehat{I}}_n^{up}=U({\widehat{I}}_{n+1}^{up}\oplus {\widehat{X}}_n^{fused}) n=3,2,1$$

(10)

$${\sigma }_n^{out}=Sigmod\left(f_{Conv}\left(AvgPool\left({\widehat{I}}_n^{up}\right)\right)\right) n=3,2,1$$

(11)

$${\widehat{I}}_n^{out}={\sigma }_n^{out}\odot {\widehat{I}}_n^{up} n=3,2,1$$

(12)

where U represents the upsampling operation, which includes the rule activation function, deconvolution operation, and normalization operation. $f_{Conv}$ comprises two $1\times 1$ convolution layers and a rule activation function. $AvgPool(·)$ is global average pooling operation. ${\widehat{X}}_n^{fused}$ represents continuous frame concatenated features of n-th stage. When n takes four, ${\widehat{I}}_4^{\mathrm{up}}$ is equivalent to $X^{\mathrm{TSM}}$.

3.2.4. Loss Function

The objective of DBIFF-NET is to predict the next frame from a sequence of consecutive input frames. The discrepancy between predicted and ground truth frames can be quantified using $L_2$ distance:

$$L_{dis}={∥I^k-\widehat{I}∥}_2$$

(13)

where $\widehat{I}$ is the predicted frame by DBIFF-NET output, and $I_P$ is the key frame.

However, the $L_2$ loss function is only calculated based on pixel level differences and cannot effectively measure the structural similarity of images. To address this limitation, Gradient Difference Lossc (GDL) and multi-scale structural similarity are used to compensate for pixel loss and improve the clarity of predicted frames:

$$L_{gdl}=\sum _{i,j}{\left|\left|I_{i,j}^k-I_{i-1,j}^k\right|-\left|{\widehat{I}}_{i,j}-{\widehat{I}}_{i-1,j}\right|\right|}_1+{\left|\left|I_{i,j}^k-I_{i,j-1}^k\right|-\left|{\widehat{I}}_{i,j}-{\widehat{I}}_{i,j-1}\right|\right|}_1$$

(14)

$$L_{mss}=1-\prod _{m=1}^M{({\displaystyle \frac{2{\mu }_I{\mu }_{\widehat{I}}+c_1}{u_I^2+u_{\widehat{I}}^2+c_1}})}^{{\beta }_m}{({\displaystyle \frac{2{\sigma }_{I\widehat{I}}+c_2}{{\sigma }_I^2+{\sigma }_{\widehat{I}}^2+c_2}})}^{{\gamma }_m}$$

(15)

where ${\mu }_I$ and ${\mu }_{\widehat{I}}$ denote the mean of the predicted frames and the key frames, respectively. ${\sigma }_{I\widehat{I}}$ is the covariance between the predicted frame and the real frame. ${\sigma }_I$ and ${\sigma }_{\widehat{I}}$ are the standard deviation between the predicted frame and the key frame, respectively. $c_1$ and $c_2$ denote stabilization constants that ensure numerical stability by preventing division by zero.

Finally, the various loss functions above are integrated into the complete loss function as follows:

$$L_{con}(I,\widehat{I})=L_{dis}+L_{gdl}+L_{mss}.$$

(16)

3.2.5. Anomaly Detection

During the testing phase, the model determines anomalies by calculating the anomaly scores of consecutive frames. According to previous research, the anomaly score is a normalized value in the range [0, 1], formulated as follows:

$$s\left(t\right)=1-{\displaystyle \frac{PSN{R}_t-{\min }_t\left(PSN{R}_t\right)}{{\max }_t\left(PSN{R}_t\right)-{\min }_t\left(PSN{R}_t\right)}}$$

(17)

where $PSNR$ is the peak signal-to-noise ratio, which is an assessment metric for image quality. The calculation of PSNR is as follows:

$$PSNR=10{log}_{10}{\displaystyle \frac{{[{\max }_{{\widehat{I}}_t}]}^2}{\frac{1}{M}{\sum }_{i=1}^M{(I_{t,i}^k-{\widehat{I}}_{t,i})}^2}}$$

(18)

where M is the total number of frame pixels, t denotes the frame index, and ${\max }_{{\widehat{I}}_t}$ is the maximum value of frame pixels.

4. Comparative Experiments

4.1. Datasets

To evaluate the performance of DBIFF-NET, this work conducts a series of experiments on three benchmark datasets widely used in the VAD, including UCSD ped2 dataset [40], CUHK Avenue [41], and ShanghaiTech Campus dataset [42]. The detailed descriptions of these two datasets are as follows:

• UCSD ped2 Dataset. The UCSD ped2 dataset was recorded on the pedestrian walkways of the University of California San Diego. Sample cases are illustrated in Figure 6a. In this dataset, walking is defined as normal behavior, while activities such as cycling and skateboarding are considered anomalies. It contains 16 training videos and 12 testing videos. In experiments, the training videos are extracted into 2550 frames, while the testing videos are extracted into 2010 frames, all with a resolution of 240 × 360 pixels.
• CUHK Avenue. The CUHK Avenue was recorded on the CUHK campus avenue. It consists of 16 training videos and 21 testing videos, covering 47 abnormal events, such as jumping, throwing objects, and dancing. Figure 6b shows the sample cases of this dataset. In experiments, the training videos are extracted into 15,328 frames, while the testing videos are extracted into 15,324, all with a resolution of 360 × 640 pixels.
• ShanghaiTech Campus dataset. The ShanghaiTech Campus dataset dataset was created by ShanghaiTech University, featuring footage captured in diverse environments including the university campus, Shanghai streets, and commercial districts. It contains 130 distinct anomaly categories, such as climbing, cycling, and jumping, thereby presenting considerable challenges for video anomaly detection tasks. Figure 6c shows the sample cases of this dataset. The dataset comprises 330 training videos and 107 testing videos. In experiments, the training videos are extracted into 274,515 frames, and the testing videos are extracted into 42,883, all with a resolution of 480 × 856 pixels.

4.2. Training Details

During training, five consecutive frames are selected, and each frame is resized to 256 × 256 pixels for UCSD Ped2, 192 × 320 pixels for CUHK Avenue, and 192 × 288 pixels for ShanghaiTech Campus. The first four frames are input into the model to predict the fifth frame. Subsequently, the predicted fifth frame is compared with the ground-truth fifth frame to calculate the loss. Finally, the Adam optimizer is employed to optimize the model, with the learning rate set to $2\times {10}^{-4}$. All experiments are conducted on four 4090 GPUs. The hyperparameters of DBIFF-NET are shown in the

Table 1. The hyperparameters of DBIFF-NET.

Hyperparameter	Setting
Epoch	120
Batch size	16
Learning rate	$2\times {10}^{-4}$
Optimizer	Adam
The length of input video frame (t)	5
The stage N of convolution module and swin transformer module	3
The size of patch	4
The depth of swin transformer block	[2, 2, 2]
The attention head of swin transformer block	[4, 8, 16]
Dropout	0.5
The Number of channels for temporal shift	32

.

4.3. Evaluation Metric

This work uses the commonly used performance metric ROC AUC in VAD to evaluate the performance of the model. The range of AUC values is [0, 1], and the higher its value, the better the anomaly detection performance.

4.4. Comparison with State-of-the-Art

To better evaluate the effectiveness of the proposed method, ten state-of-the-art (SOAT) methods are adopted to comparative experiments, including AE-Conv2D [10], Mem-AE [30], MNAD-Recon [43], zxVAD [44], MAMC-Net [45], Dual GroupGAN [46], BDHF [47], CLAE [23], Dang et al. [20], and MemATr [48].

Table 2. Comparison with SOTA methods in term of AUC (%) on UCSD Ped2 and ShanghaiTech Campus datasets.

Methods	UCSD Ped2	CUHK Avenue	ShanghaiTech Campus Dataset
AE-Conv2D [10]	90.0	70.2	60.9
Mem-AE [30]	94.1	71.1	71.2
MNAD-Recon [43]	90.2	82.8	69.8
zxVAD [44]	95.8	83.1	71.6
MAMC-Net [45]	96.7	87.6	71.5
Dual GroupGAN [46]	96.6	85.5	73.1
BDHF [47]	96.4	86.5	73.6
CLAE [23]	90.8	83.1	73.3
Dang et al. [20]	96.0	85.9	73.3
MemATr [48]	95.2	83.8	73.4
DBIFF-Net (Ours)	97.7	84.5	73.8

The bold entries show the best results.

presents the performance of DBIFF-NET, and these comparison methods on the UCSD Ped2, CUHK Avenue, and ShanghaiTech Campus datasets. It can be observed that DBIFF-NET achieves 97.7%, 85.0, and 73.8% on UCSD Ped2, CUHK Avenue, and ShanghaiTech Campus datasets, respectively, which surpasses the most SOAT methods. Compared to traditional autoencoder Method (e.g., AE-Conv2D and Mem-AE), our method has a substantial improvement. This indicates that the proposed model is a more effective modeling for the anomaly detection.

Table 3. The performance of the DBIFF-Net on three datasets.

Dataset	Method	Precision	Recall	F1-Score
UCSD Ped2	Dang et al. [20]	96.0	93.6	94.8
	DBIFF-Net (Ours)	97.1	96.2	96.7
CUHK Avenue	Dang et al. [20]	62.1	71.4	64.4
	DBIFF-Net (Ours)	57.4	75.4	65.2
ShanghaiTech Campus	Dang et al. [20]	53.2	85.6	65.0
	DBIFF-Net (Ours)	58.2	74.1	65.2

The bold entries show the best results.

presents a comparative analysis of our model against the method by Dang et al. (2025) [20] in terms of precision, recall, and F1-score. The results demonstrate that DBIFF-Net consistently surpasses theirs in F1-score across all three datasets. Furthermore, the notably higher recall of DBIFF-Net indicates a heightened sensitivity in detecting anomalies under complex scenarios.

It is worth noting that DBIFF-NET demonstrates more reliable performance on the UCSD Ped2 dataset than on the ShanghaiTech Campus dataset. This can be attributed to the larger scale and the diversity of anomaly events in ShanghaiTech Campus dataset, which makes anomaly detection on this dataset more challenging. It should be emphasized that DBIFF-NET dose not utilize any auxiliary techniques, such as adversarial learning, data augmentation, or memory enhancement methods. DBIFF-NET still outperforms many advanced models, suggesting that well-designed, simple multi-scale methods can achieve performance competitive with or even superior to more complex approaches.

Although DBIFF-Net demonstrates better performance against comparison methods on the UCSD Ped2 and ShanghaiTech Campus datasets, its performance on CUHK Avenue is suboptimal. Further analysis revealed that the CUHK Avenue dataset contains noisy or corrupted video. For instance, the training set includes videos of children running and jumping, which significantly contaminates the training data. Additionally, in some test videos, only partial body segments of dancers are visible. These data quality issues substantially challenge the model’s detection capability. We note that other existing methods also struggle to correctly identify such anomalous clips, and this particular issue remains understudied in these methods. Therefore, this represents a valuable and promising research direction. We plan to prioritize investigating such challenges in our future work. From a modeling standpoint, in complex scenarios, DBIFF-NET demonstrates significant performance improvements on the ShanghaiTech Campus dataset, while the results on the CUHK Avenue dataset are relatively weaker. This disparity is largely due to the use of the swin transformer as the global branch, which requires a larger dataset for effective training. The training set for the ShanghaiTech Campus dataset is ten times larger than that of CUHK Avenue, contributing to the superior detection performance observed.

4.5. Ablation Study and Analysis

To evaluate the impact of each module in the proposed method on the performance, this section conducts ablation experiments on the dual-stream network structure, the fusion method within the DBIFF-NET model, channel attention mechanism, and temporal shift module (TSM) on the UCSD Ped2 dataset.

4.5.1. The Impact of Dual-Branch Networks on Performance

To investigate the impact of the dual-branch network on model performance, this experiment replaces the encoding module in the DBIFF-NET with a single-stream CNN and a single-stream swin transformer, respectively. The results are presented in

Table 4. Ablation experiment of different branches on the UCSD Ped2 dataset.

Methods	UCSD Ped2
Single-CNN	97.2
Single-swin transformer	84.1
DBIFF-NET (dual-branch)	97.7

The bold entries show the best results.

. While both single-stream networks achieve AUC scores exceeding 80%, the model attains its best performance only when the dual-branch network is utilized. This improvement can be attributed to the multi-scale encoding approach, which enables the model to integrate local and global features for a comprehensive understanding of video frames, thereby enhancing its anomaly detection capability.

4.5.2. Impact of Fusion Methods on Performance

Multi-scale feature fusion is often challenging. In constructing the DBIFF-NET, this study considers several fusion strategies, including element-wise addition (SUM), Q-attention (QA), and feature concatenation (CON). To illustrate these fusion methods, we take two features, $X_i^C$ and $X_i^{S}T$, as examples in the following descriptions:

$$SUM(·)=X_n^C\oplus X_n^{ST}$$

(19)

$$QA(·)=Soft\max \left(q{X}_n^C\right)X_n^C\oplus Soft\max \left(q{X}_n^{ST}\right)X_n^{ST}$$

(20)

$$CON(·)=concat(X_n^C,X_n^{ST})$$

(21)

where q represents a learnable weight. Comparative results of fusion strategies are presented in

Table 5. Ablation experiment of fusion methods on the UCSD Ped2 dataset.

Fusion Methods	UCSD Ped2
SUM	97.2
QA	97.7
CON	96.9
Interactive Fusion	97.7

The bold entries show the best results.

. The table indicates that the performance of QA strategy is on par with that of interactive fusion strategy while outperforming other strategies. Further experiments on the ShanghaiTech Campus dataset revealed that the interactive fusion strategy achieved an AUC of 73.8%, significantly higher than 72.8% of the QA strategy. Therefore, the interactive fusion strategy is adopted in the DBIFF-NET model.

4.5.3. Effectiveness of Each Component in Decoder

In

Table 6. Ablation experiment of TSM and channel attention on the UCSD Ped2 dataset.

Interactive Fusion Module	TSM	Channel Attention	ROC AUC
✓			96.4
✓	✓		96.5
✓		✓	94.1
	✓	✓	97.2
✓	✓✓	✓	97.0
✓	✓	✓	97.7

The bold entries show the best results. ✓ denotes that the module is used. ✓✓ represents more TSM.

, we conducted ablation studies on each component in decoder, including the interactive fusion module, the TSM module and the channel attention module. Both the TSM module and the channel attention module effectively enhance anomaly detection capability of DBIFF-NET. Especially for the addition of TSM, it boosts the performance by 3.6%. This shows that the construction of TSM module and the channel attention module play a critical role in contributing to video event restoration. In addition, “more TSM” represents that TSM is applied after the LDIF, MDIF, and HDIF modules. In the final model, TSM is applied only after HDIF. Results demonstrate that randomly increased TSM modules do not achieve the best performance of the model. This proves the rationality of the DBIFF-NET model structure.

4.5.4. Visualization

In this section, we visualize the anomaly detection results of DBIFF-Net. Figure 7 and Figure 8 visualize the anomaly score generated by DBIFF-Net on the test sets of the UCSD Ped2 and ShanghaiTech Campus datasets, respectively. The x-axis in the figure represents video frames, and the y-axis denotes the anomaly score. From the curve of anomaly score, both test sets contain lot of normal frames, presenting a significant challenge for anomaly detection. However, our model successfully captures anomalies with accuracy, demonstrating the effectiveness of our approach.

Specifically, Figure 7 (UCSD Ped2 dataset) shows consistently low anomaly score in the first 80 frames. At approximately frame 90, the anomaly score begins to rapidly increase. Upon examining the data around frame 90, we observe that a bicycle starts to enter the video. By frame 100, when the bicycle becomes fully visible, the anomaly score stabilizes at elevated levels and continue rising as the object becomes more distinct.

The anomaly score curve shown in Figure 8 exhibits significant fluctuations, reflecting the complexity of anomalous behavior in this ShanghaiTech Campus video segment. From this figure, the curve fluctuates but remains consistently low before to 250 frames. However, the anomaly score rapidly increased starting around 280 frames. When it peaks at approximately frame 290 (marked (c) Anomaly frame), it is seen that a black backpack enters the video. Subsequently, the anomaly score rapidly drops again as the backpack completely disappears from the video, reaching a low point. The score then climbs sharply again as the backpack descended. When the backpack is thrown up for the second time, the anomaly score curve shows the same trend, and the anomaly score spikes when the anomalous objects appear and drops sharply when the anomalies conclude. This alignment between score peaks and anomalies clearly demonstrate the model’s ability to effectively distinguish anomalies from normal activity.

5. Conclusions and Future Work

This article introduces a novel video anomaly detection model, DBIFF-NET, designed to enhance the accuracy and efficiency of anomaly detection in video surveillance. The model utilizes multi-scale encoding to extract global and local features from video frames, while leveraging residual skip connections, interactive fusion, channel attention, and TSM to better distinguish between normal and anomalous data. DBIFF-NET demonstrates superior anomaly detection performance compared to most state-of-the-art methods across three benchmark datasets. The ablation studies show that multi-scale encoding, interactive fusion module, channel attention, and TSM collectively enhance anomaly detection performance of the model. Despite its strong overall performance, comparisons on the CUHK Avenue and ShanghaiTech Campus datasets reveal a limitation: the model’s representational capacity appears constrained under data-scarce conditions. Additionally, computational overhead remains non-negligible, with a model size of 371.57 MB and a computational cost of 232.4 GFLOPs for a four-frame input. In future work, data augmentation techniques will be further explored and applied to the CUHK Avenue dataset to enhance model robustness and improve performance under limited-data conditions. In addition, the computational overhead is still non-negligible. We also plan to optimize parameters usage and improve the model’s response time to facilitate more efficient deployment.