CNN-Based Spatiotemporal Feature Extraction for Video Processing: A Systematic Review

Abstract

The extraction of spatiotemporal features from video sequences allows for the recognition of actions and the analysis of behaviors in video, making it a key challenge in automated video processing. The literature shows widespread use of deep learning approaches, specifically convolutional neural networks (CNNs). In this context, researchers face the challenge of identifying the advantages, disadvantages, and emerging trends across different architectures, evaluation metrics, and even dataset selection. The objective of this study is to identify the most common CNN architectures, evaluation metrics, datasets, and trends in spatiotemporal feature extraction for video analysis. The selection of articles used the PRISMA methodology and the Joanna Briggs Institute (JBI) methodological framework. From the databases Scopus, Web of Science and the MDPI platform, and based on the inclusion/exclusion criteria, 31 articles that met the criteria were analyzed and synthesized. The search was conducted primarily using the keywords “Convolutional Neural Network,” “video processing,” and “feature extraction,” limiting the selected works to those published between 2020 and the end of 2025. The results mainly show the use of four neural network architectures: 2D CNNs, 3D CNNs, hybrid models (e.g., CNN–RNN, CNN–Transformer, and multi-stream models), and, to a lesser extent, lightweight architectures. Commonly used datasets were identified (e.g., UCF101 and HMDB51). Additionally, standardized evaluation metrics were identified, ranging from accuracy and F1-score to performance measures specific to each case study. The challenges identified center on the heterogeneity of the study datasets, the lack of standardized evaluation metrics, and maintaining a balance between accuracy and computational resource consumption. On the other hand, strong emerging trends toward the use of hybrid models and those integrating transformers have been identified. This systematic review emphasizes the need for clear and robust guidelines that allow for the appropriate selection of CNN architecture, test datasets, and evaluation metrics in applications for extracting spatiotemporal features from video sequences, as well as identifying trends and future lines of research.

1. Introduction

A cornerstone of modern computer vision is the ability to process video automatically [1], with decisive applications in areas such as automated driver assistance systems (ADASs) [2], medical diagnostics [3], intelligent monitoring and surveillance [4], and interactive media content [5]. Automatically scrutinizing and interpreting ever-changing visual information requires advanced feature-extraction methods capable of capturing both static information and inevitable changes in the scene or scenario. Wang et al. [6] stated, “effective video representation is critical for various video processing applications,” emphasizing the importance of investigating and developing robust methods for spatiotemporal feature extraction.

To break it down further, spatial features in video processing are the static or motionless information within individual frames, i.e., edges [7], textures [8,9], shapes [10], and objects [11], which are similar to the information extracted from static or motionless images. However, temporal features capture the dynamics and motion between consecutive frames, including optical flow, object displacement, and scene changes. To ensure a complete understanding of video information, it is necessary to integrate these two methods efficiently [12].

Carreira & Zisserman (2017) [13] identified a key challenge in action recognition: capturing both spatial and temporal components of an action. Extracting these spatiotemporal correlations is complex and has driven the development of many computational approaches. This core challenge has motivated researchers to explore different architectural strategies, ranging from adapting existing models to designing new, specialized architectures.

Architectures combining 2D and RNN [14] are classified as a hybrid model; in this case, the approach integrates both convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The CNN component manages spatial feature extraction, while the RNN addresses temporal dependencies. Previously, video processing with CNNs involved different steps: 2D-CNN models were first pre-trained on static frames, and their features were subsequently combined [15]. Each approach has its advantages and disadvantages. 2D-CNNs are computationally efficient and benefit from transfer learning when using pre-trained models, whereas 3D-CNNs better capture spatiotemporal correlations but demand more computational power and larger datasets for effective training.

1.1. Motivation Behind Creating This Article

Thanks to rapid growth in research, which reflects the area’s potential viability, this growth has also resulted in a highly fragmented area. Specifically, video processing using CNN-based spatiotemporal feature extraction is a major focus due to increased demand for applications and implementations in autonomous systems, human–computer interaction, and, unsurprisingly, surveillance and monitoring. To further demonstrate this point, see Figure 1. This graph illustrates the number of publications related to CNNs, and it is evident that the number of publications continues to increase each year, further reinforcing the viability of the area of study.

Figure 1. Articles published per year (2015–2025) in JCR-indexed journals with titles containing “Convolutional Neural Network”. The upward trend confirms the growing research activity in CNN-based methods for video and image analysis.

During the review and analysis of the works that met the inclusion criteria for this systematic literature review, it became clear that there is no one-size-fits-all method or approach. On the contrary, there is a wide variety of methods, ranging from foundational methods, e.g., 2D-CNNs and 3D-CNNs, to sophisticated hybrid models (e.g., LSTM and GRU) and transformer-based models. This has, unfortunately, yielded an overwhelming number of architecture types. This diversity further complicates the ability to discern fundamental trade-offs, such as comparative advantages and particular application contexts for each architectural family. The recent and existing literature tends to focus on a single perspective, e.g., 3D-CNNs [16], but does not present a unified categorization that methodically classifies the full range of 2D, 3D, hybrids, and emerging trends.

Throughout this systematic review, the authors identified several previous reviews with a similar scope. A detailed analysis revealed a need for comprehensive, unbiased cross-study comparisons to assess the robustness and transferability of these methods [17,18]. As mentioned earlier, these issues have also contributed to an environment with divided evaluations. The articles included in this review utilize various datasets, from well-known ones like UCF101 and HMDB51 to more specialized datasets. Private datasets were also employed, mainly for anomaly detection or gait recognition. Although accuracy remains a key metric, its limitations under class imbalance are well known, prompting the use of diverse metrics such as precision, F1-score, recall, and task-specific measures like mAP, PSNR, and SSIM.

During the research for this systematic review, the authors noticed a recurring trend: existing reviews cover accuracy and computational complexity; very few studies thoroughly address pragmatic considerations such as energy efficiency, scalability to long or high-resolution videos, and inference speed on edge devices. Energy efficiency is usually assessed by measuring the amount of power consumed during inference using hardware power meters or software APIs (e.g., NVIDIA Management Library), and reports metrics like frames per second per watt (FPS/W) or inference per joule. Fortunately, there exist standardized benchmarks like MLPerf inference and the principles of Green AI, which offer frameworks for these measurements [19,20]. That said, it was evident that existing reviews lack technical depth. These aspects are increasingly vital for real-world applications, as highlighted by the advent of lightweight architectures, e.g., GP3D [21] and MDNet-LBPC [22]; these architectures were specifically designed for resource-constrained environments.

This further underscores the need for reviews that adhere to established norms and guidelines such as PRISMA [23] to ensure systematic rigor. In the course of this review, the authors came across reviews in this area [24,25] that had a narrative style and also lacked the rigorous, reproducible methodology required to ensure an unbiased analysis. On the contrary, the absence of established, rigorous systematic search parameters, clear and explicit inclusion criteria, and quality assessment of the selected literature can actually result in selection bias, which, in turn, limits the credibility of their conclusions.

Now that gaps have been identified (i.e., updating, fragmentation, evaluation, and methodology), this systematic review aims to provide a consolidated, thorough, and relevant analysis of CNN-based spatiotemporal feature extraction methods. This will produce a valuable resource for investigators, researchers, and novice practitioners trying to navigate this complex and intricate field. The motivation for this systematic review is the urgent need to synthesize this varied and ever-evolving information.

1.2. Contribution

This systematic review will present key contributions to CNN-based spatiotemporal feature extraction for video processing and directly address gaps identified in the literature. The primary contributions of this systematic review are the following:

• Consolidated classification of CNN architectures: After reviewing and analyzing all 31 selected studies, we classified and categorized the predominant CNN architectures for spatiotemporal feature extraction into four groups: 2D-CNN, 3D-CNN, hybrid models (e.g., CNN–RNN, CNN–Transformer, and two-stream), and, lastly, emerging lightweight architectures. Unlike current systematic reviews, which often focus only on a single paradigm (2D, 3D, or hybrid architectures [16,24]), the categorization presented in this review is thorough and unified. This review directly counters the trends of current reviews by presenting a clear and well-structured overview of the field, thereby filling gaps in previous reviews.
• Quantitative benchmarking and qualitative appraisal of architectures: This review provides both a quantitative analysis of crucial evaluation metrics (e.g., accuracy and F1-score) and a qualitative study of the strengths and weaknesses of each architectural category. The resulting summary highlights the current state of the art and the trade-offs between accuracy, computational cost, and model complexity. This was achieved through rigorous adherence to the PRISMA-based systematic review methodology, considering the most recent literature published between 2020 and 2025.
• Identification of key datasets and evaluation protocols: A crucial gap in the existing literature is the absence of the ability to have a fair cross-study comparison, because there are no standardized evaluation datasets to evaluate. This review will address that gap by first producing a thorough summary of the datasets that are used most frequently (e.g., UCF101 and HMDB51), as well as the predominant evaluation metrics. Analyzing the inconsistency between reported metrics and experimental protocols highlights the need for coherence in comparative evaluation.
• Analysis of emerging trends and open challenges: This systematic review will provide a clear and concise path forward for future investigators and researchers. It will identify and discuss the most notable emerging trends, for example, how there is a growing shift toward the use of hybrid models and transformer-based architectures, the increasing integration of attention mechanisms, and the advent of lightweight models for edge deployment. The authors will also point out the main obstacles, i.e., handling complex or cluttered scenes, ensuring real-time inference on resource-constrained devices, and grappling with privacy and ethical issues.

This review offers four novel contributions beyond prior fragmented surveys: (i) a hierarchical taxonomy of CNN architectures for spatiotemporal feature extraction that provides clearer categorization than previous overlapping classifications, establishing a practical framework for understanding predominant architectural paradigms; (ii) cross-category quantitative performance comparisons using consistent evaluation protocols, enabling meaningful architectural comparisons that prior reviews have lacked; (iii) a systematic adaptation of the JBI risk-of-bias assessment framework to deep-learning benchmark studies, providing a methodological quality evaluation that has not been previously applied in this domain; and (iv) a comprehensive extraction of future research priorities directly from the literature itself, identifying consensus directions that emerge from the reviewed studies rather than relying on subjective author perspectives.

This review will close with a discussion, possible implications, and the limitations of this review (see Section 5), preceded by the conclusion that also outlines future work (see Section 6). Prior to that, the answers to the posed research question are presented in Section 4. In Section 3, the methodology is covered, and it also includes the search strategy, inclusion and exclusion criteria, and data extraction. Section 2 contains all the necessary theoretical foundations through a thorough review of the most recent literature.

2. Foundations and Related Work

The interdisciplinary nature of the research in this area is worth underscoring. Figure 2a demonstrates what percentage of the articles reviewed came from which dataset, while Figure 2b shows the main subject categories they belong to; this helps to contextualize and properly frame the discussion that will follow.

Figure 2. (a) Breakdown of the 31 selected studies by academic database (Scopus, Web of Science, and the MDPI platform), showing the proportion sourced from each. (b) Distribution of these studies across subject areas, highlighting the multidisciplinary nature of spatiotemporal feature extraction research.

2.1. Fundamentals of Artificial Neural Networks

At the center of modern video-processing deep learning systems are artificial neural networks (ANNs). They were inspired by how the human brain functions, consisting of layers of interconnected artificial neurons. During training, each connection between artificial neurons has an adjustable weight that is updated using optimization algorithms, with backpropagation being the prime example [26]. This structure defines the computational paradigm.

2.1.1. Fundamental Architectures of ANN

The evolution of the ANN architecture has paved the path for the emergence of several categories based on their structure and applications. One of the most recent is transformers, which are architectures that rely on self-attention mechanisms that have completely transformed sequence processing—initially with Natural Language Processing (NLP) but then later in computer vision [27]. CNNs are designed to specialize in processing grid-like data (e.g., images and videos), using stacked convolutional, pooling, and fully connected layers to extract and learn hierarchical spatial and spatiotemporal features [28]. RNNs enable feedback loops and have an internal state that helps handle temporal sequences, but variants such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) address the gradient issue [29]. Feedforward networks transfer information unidirectionally, from input to output, without cycles or feedback loops, and can range from rudimentary perceptrons to Multilayer Perceptrons (MLPs) [30].

The equations that follow govern the LSTM unit. The first two gates, the forget gate and the input gate, described in Equations (1) and (2), control how the cell state is updated. The forget gate decides which information from the previous cell state $C_{t-1}$ should be discarded, while the input gate determines which new candidate values ${\tilde{C}}_t$ should be stored in the cell state. The output gate (Equation (5)) then filters the updated cell state to produce the hidden state $h_t$.

$$\begin{array}{cc}\hfill f_t& =\sigma (W_f·[h_{t-1},x_t]+b_f),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill i_t& =\sigma (W_i·[h_{t-1},x_t]+b_i),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill {\tilde{C}}_t& =tanh(W_C·[h_{t-1},x_t]+b_C),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill C_t& =f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill o_t& =\sigma (W_o·[h_{t-1},x_t]+b_o),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill h_t& =o_t\odot tanh\left(C_t\right)\hfill \end{array}$$

(6)

where $x_t$ is the input at time t, $h_t$ is the hidden state, $C_t$ is the cell state, $\sigma$ is the sigmoid function, ⊙ denotes element-wise multiplication, and the W and b terms are learned weights and biases. The forget gate $f_t$ and the input gate $i_t$ work together: $f_t$ controls the extent to which the previous cell state $C_{t-1}$ is retained, while $i_t$ governs the contribution of the new candidate ${\tilde{C}}_t$. This complementary mechanism allows the LSTM to selectively remember or forget information over long sequences.

The GRU simplifies the LSTM architecture using update and reset gates:

$$\begin{array}{cc}\hfill z_t& =\sigma (W_z·[h_{t-1},x_t]),\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill r_t& =\sigma (W_r·[h_{t-1},x_t]),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\tilde{h}}_t& =tanh(W·[r_t\odot h_{t-1},x_t]),\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill h_t& =(1-z_t)\odot h_{t-1}+z_t\odot {\tilde{h}}_t\hfill \end{array}$$

(10)

The core self-attention mechanism in transformers is the scaled dot-product attention,

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d_k}}}\right)V$$

(11)

where Q, K, and V are the query, key, and value matrices, and $d_k$ is the dimension of the keys.

2.1.2. Applications of ANN in Computer Vision

ANNs have demonstrated exceptional performance in various computer vision applications, forming the technological basis for advanced systems of object classification and detection [31], semantic and instance segmentation [32], depth estimation and 3D reconstruction [33], video processing and analysis [15], and vision systems for autonomous vehicles [34].

2.2. Convolutional Neural Networks for Video Processing

CNNs are a fundamental pillar for extracting spatiotemporal features in video, having undergone notable evolution in their adaptation to the temporal domain.

2.2.1. 2D-CNN-Based Approaches

2D-CNN-based approaches leverage pre-trained architectures on static image datasets (such as ImageNet) to extract spatial features, which are then temporally integrated using various mechanisms: Two-Stream Architectures—These have two separate streams for spatial (RGB frames) and temporal (optical flow) information, with late fusion of predictions [15]. CNN + RNN, It combines the power of CNNs for spatial extraction with the ability of RNNs to model temporal dependencies. LRCN [14] uses frame-wise CNNs and LSTMs for temporal integration, effectively capturing long-range dependencies. Temporal Segment Networks. It introduces temporal consensus mechanisms to aggregate information across extended video segments [35], enabling the modeling of longer-duration actions.

For each position $(i,j)$, a value is computed by a 2D convolution $S(i,j)$. This is carried out by adding the element-wise products of a small kernel K and the overlapping region of an input image or feature map I. To be more specific, for an input I of height H and width W and a kernel K of size $m\times n$, the convolution will produce an output feature map S, which is defined by the double sum below. This operation is at the center of every CNN. Its mathematical representation is:

$$S(i,j)=(I\ast K)(i,j)=\sum _m\sum _{n}I(i+m,j+n)K(m,n)$$

(12)

The convolution extracts local, small patterns (i.e., edges, textures, and shapes) by sliding the kernels across the input. At each possible position $(i,j)$, a single output value is computed. The double summation works as follows (Algorithm 1):

Algorithm 1 2D Convolution at position $(i,j)$

Require:  Input feature map I, kernel K of size $m\times n$ Ensure:  Output value $S(i,j)$   1: $S\leftarrow 0$   2: for  $r=0$ to  $m-1$  do   3:    for $c=0$ to $n-1$ do   4:      $S\leftarrow S+I(i+r,j+c)\times K(r,c)$   5:    end for   6: end for   7: return S

2.2.2. 3D-CNN-Based Approaches

3D-CNNs extend convolutional operations to the temporal domain, learning spatiotemporal filters jointly and end-to-end: C3D—Showed that $3\times 3\times 3$ kernels with homogeneous architecture can learn effective spatiotemporal features, setting an important baseline for 3D approaches [12]. I3D—Inflates pre-trained 2D filters in 3D, enabling transfer learning from ImageNet and achieving state-of-the-art performance on multiple benchmarks. This approach demonstrated the importance of large-scale pre-training [13]. R(2 + 1)D—Factors 3D convolutions into 2D spatial and 1D temporal components, improving optimization and generalization by reducing the number of parameters and facilitating training [36]. SlowFast Networks—Proposes two pathways with different temporal frequencies to capture semantic (slow) and motion (fast) information simultaneously, offering an efficient and effective solution [37].

For a 3D input volume I (height, width, and depth/time) and a 3D kernel K, the convolution is given by

$$S(i,j,k)=(I\ast K)(i,j,k)=\sum _m\sum _n\sum _{p}I(i+m,j+n,k+p)K(m,n,p)$$

(13)

where the triple sum runs over the spatial and temporal dimensions of the kernel.

2.2.3. Hybrid and Emerging Models

The most recent approaches combine multiple paradigms to overcome the limitations of unimodal or single-mode approaches: CNN–Transformer—These use a CNN for local feature extraction and transformers to model long-range global dependencies [38]. Lightweight Architectures—Lightweight models for real-time applications that optimize the accuracy–speed balance [39]. Self-Supervised Learning—Unsupervised learning of spatiotemporal representations that reduces dependence on annotated data [40]. Multimodal Approaches—Integrating visual information with audio, text, and other modalities for richer or more complete representations [41].

2.3. Previous Systematic Reviews

There are numerous reviews in the area of computer vision and video processing, but they have significant limitations that justify the present systematic review. For instance, Herath et al. [24] presented a comprehensive review of human action recognition but focused mainly on traditional methods and early deep learning, which have become obsolete in the face of recent rapid advances in transformer architectures and self-supervised learning. Similarly, several reviews have focused on specific applications, such as action recognition [42] and video captioning [17]. Although specialized, these works do not comparatively address spatiotemporal feature extraction techniques across different tasks and domains, limiting their usefulness for researchers seeking a general perspective.

Limitations of Existing Systematic Reviews

Critical analysis of the existing literature reveals several significant gaps:

• Lack of updating. This is one significant shortcoming of many systematic reviews; for example, the works by Herath et al. [24] and Sargano et al. [43], which, in full disclosure, were published before transformer-based architectures and self-supervised learning became widely used. Nevertheless, they overlooked key breakthroughs like Video Vision Transformers (ViViT) [38] and masked autoencoders. This leaves a critical gap in our understanding of modern spatiotemporal feature extractors. So, in short, reviews quickly become outdated due to how fast this field of study is progressing.
• Fragmented approach. Because of this, it is quite difficult for researchers and investigators to properly position their contribution to the field because of how the literature is split. For instance, the review conducted by Ji et al. [16], although quite thorough, only examines 3D-CNN but never takes into consideration the large body of work on 2D + RNN hybrid models and two-stream methods. Similarly, reviews on video action recognition in the wild [25] do a good job of covering many techniques but fail to provide a single classification that properly groups 2D, 3D, hybrid models, and emerging architectures together. To put it simply, existing reviews are narrow in scope, each only focusing on one piece of the puzzle.
• Incomplete evaluation. As a result, this limits confidence in any general conclusions about robustness or the transferability of spatiotemporal feature extraction techniques across distinct problems. For example, a model that works extremely well for action recognition on the UCF101 dataset might fail if tasked with anomaly detection on the UCF Crime dataset. Still, cross-domain comparisons are rarely made, let alone reported. This creates a need for comprehensive comparative evaluations across multiple tasks, datasets, and evaluation metrics. Many systematic reviews, like those that focus on video anomaly detection [17] or video segmentation [18], provide a thorough and detailed analysis of their methods and datasets, but only within their narrow perspective of the field.
• Limited technical perspective. Existing reviews, including those that focus on lightweight architectures [39], do not provide a comprehensive comparison of the speed of inference on edge devices, or how well (or not) methods scale to long-form videos, given that this is crucial for real-world deployment. Yet all these factors are essential for the actual implementation of these models. This is why there is a need for systematic reviews that go beyond accuracy and FLOPs to include model size, latency, and hardware compatibility and present information that would aid implementation and deployment on devices with limited resources. Most current reviews narrowly focus on raw performance, put simply.
• Non-systematic methodology. Due to this, the conclusions of many noteworthy and influential reviews like Herath et al. [24] and Yu et al. [25] could be biased and, due to this, may not be fully reliable, given that there is no explicit systematic search protocol, inclusion criteria, or a method for ensuring the quality of selected literature. If these details are not reported, then it is impossible to recreate or reproduce their work or findings in order to ensure all relevant studies were covered. Therefore, it is essential to strictly adhere to the well-established and structured guidelines of the PRISMA framework. This leads to a more transparent and rigorous review, reinforcing its reliability.

The authors noted a need for a systematic review that addresses the full spectrum of architecture types, evaluation protocols, and issues in practical deployment. It should provide an up-to-date, unified, and thorough analysis of CNN-based spatiotemporal feature extraction methods for video processing. The identified gaps clearly emphasize and reinforce the need for this systematic review.

2.4. Evaluation Metrics

Once researchers conclude their investigation and present their results, they must evaluate the performance and effectiveness of the algorithms/models used. Some of the most common evaluation metrics are accuracy, sensitivity, specificity, and F1-score.

2.4.1. Accuracy

It is one of the most common evaluation metrics used in classification tasks. It is defined as the proportion of correctly labeled samples relative to the total number of entries in the dataset. Essentially, accuracy assesses the overall validity of a model’s prediction. It is defined by the following equation,

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(14)

where TP is the true positives, indicating that the model correctly predicts a positive case; TN is the true negatives, when the model correctly predicts a negative case; FN is the false negatives, which occur when the model incorrectly predicts a negative case; and, finally, FP refers to the false positives, which occur when the system predicts something that is not actually present.

2.4.2. Sensitivity

This evaluation metric, also known as “recall” or “true positive”, measures a model’s ability to correctly identify actual positive cases in the dataset. Specifically, it represents the percentage of positives that were correctly identified. A high percentage indicates accurate detection of negatives, while a low percentage suggests the model is missing positive cases. The following equation describes this relationship.

$$\mathrm{Sensitivity}={\displaystyle \frac{TP}{TP+FN}}$$

(15)

Eventhough models with high sensitivity limit false negatives, it must be taken into account that overly sensitive models may increase the likelihood of false positives.

2.4.3. Specificity

Evaluates the model’s ability to distinguish true negatives correctly and is expressed by the following equation:

$$\mathrm{Specificity}={\displaystyle \frac{TN}{TN+FP}}$$

(16)

Again, it must be noted that models streamlined for high specificity may be overly conservative, leading to true positives being missed or overlooked.

2.4.4. F1-Score

The F1-score is the harmonic mean of precision and sensitivity (recall), providing a single metric that balances false positives and false negatives. It is particularly useful when the class distribution is imbalanced, because it penalizes models that achieve high precision at the expense of recall, or vice versa. The F1-score is defined as follows,

$$\mathrm{F}1-\mathrm{score}={\displaystyle \frac{2·(\mathrm{Precision}·\mathrm{Recall})}{(\mathrm{Precision}+\mathrm{Recall})}}$$

(17)

where precision (positive predictive value, PPV) is $\mathrm{TP}/(\mathrm{TP}+\mathrm{FP})$, and recall (sensitivity, SEN) is $\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$. An F-score of 1 is only achieved when both precision and recall are perfect, while an F-score of 0 indicates that the model is incapable of correctly detecting a single positive instance.

2.4.5. Why the Need for This Systematic Review

This review does three things: first, it introduces a consolidated classification system; second, it compares performance across multiple datasets and evaluation metrics; third, it identifies emerging trends and future research directions. Once completed, it directly fills the gaps that have been identified. The result is a thorough and comprehensive, up-to-date, methodologically rigorous summary of CNN-based spatiotemporal feature extraction for video processing.

3. Materials and Methods

The authors of this review have organized this systematic review into four phases: (i) formulation and definition of research questions. (ii) clearly defining the criteria for inclusion and exclusion, (iii) a systematic search for literature across three academic databases, and (iv) extraction and analysis of data from the selected studies. This review was conducted following the PRISMA 2020 statement [23]. No review protocol was registered prior to conducting this study. The following subsections will detail each stage further.

3.1. Research Objectives

The questions that will guide this investigation are listed below; see Table 1. The main objective of the systematic review is to analyze, categorize, and critically assess the state of the art for CNN-based spatiotemporal feature extraction in video processing.

Table 1. Research questions addressed in this systematic review.

Code	Research Question
RQ1	What CNN architectures are most commonly employed for spatiotemporal feature extraction from video, and how can they be categorized?
RQ2	Which datasets and benchmark datasets are most frequently used to evaluate these architectures?
RQ3	Which evaluation metrics are most commonly employed to assess spatiotemporal feature extraction methods?
RQ4	What are the main strengths and limitations of each architectural category?
RQ5	What are the current challenges, emerging trends, and future research directions in CNN-based spatiotemporal feature extraction for video processing?

3.2. Criteria for Article Selection and Keywords

Three academic databases were selected: (i) Scopus, (ii) Web of Science (Clarivate), and (iii) the MDPI platform. These databases were chosen for their complementary coverage of peer-reviewed literature across multiple disciplines. To ensure rigor and reproducibility, the authors first clearly established inclusion and exclusion criteria prior to conducting any literature search.

To stay at the vanguard of the latest developments in CNN-based spatiotemporal feature extraction, particularly the emergence of Transformer-based models and lightweight architectures, the authors made a conscious decision to limit the search to 2020 through 2025. Scopus and Web of Science (Clarivate) have been widely recognized as the most complete multidisciplinary citation indexes that cover numerous high-impact journals. Although the MDPI platform may be considered a subset of one or both of the larger databases mentioned earlier, it was still chosen because it is a specialized open-access platform, so as not to overlook any relevant studies. Thanks to the combination of these three databases, we ensure broad disciplinary coverage in areas with significant research, i.e., Asia, Europe, and the Americas, see Section 4.1, while also targeting open-access publishers, thereby minimizing selection bias and improving search reproducibility.

Regarding database selection, Scopus and Web of Science were chosen due to their established status as the most comprehensive multidisciplinary citation indexes, covering the vast majority of peer-reviewed literature in computer science and engineering. The inclusion of the MDPI platform, while potentially overlapping with these larger databases, served a specific purpose: ensuring representation of open-access literature that may not be consistently indexed in proprietary databases. This tripartite database selection strategy follows recommendations for systematic reviews in rapidly evolving fields, where multiple complementary sources are necessary to minimize publication bias. While databases such as IEEE Xplore and the ACM Digital Library offer valuable content, their coverage overlaps substantially with Scopus and Web of Science for the specific time period and subject areas under consideration. The subject area restrictions applied (Computer Science, Engineering, Mathematics, Physics and Astronomy, Decision Science) further refined the search to fields where quantitative, experimentally validated CNN-based video analysis work is predominantly published [4,5].

It was also decided to limit the search to specific subject areas, e.g., Computer Science, Engineering, Mathematics, Physics and Astronomy, and Decision Science, because these fields typically produce quantitative, experimentally validated work relevant to video analysis. Only peer-reviewed journal articles and conference papers written in English were included. To preserve academic rigor and ensure reproducibility, all non-empirical publications (e.g., editorials, technical reports, books) were excluded. To avoid deviating from the main focus of the core topic, the search terms were limited to a combination of “Convolutional Neural Network,” “video processing,” and “feature extraction.” The complete criteria for inclusion and exclusion, as well as the justification for those decisions, can be found in Table 2.

Table 2. Eligibility criteria applied for study selection during the systematic review.

Criterion	Inclusion Criteria	Exclusion Criteria	Justification
Language	Studies published in English.	Publications in languages other than English.	English is the predominant language for high-impact scientific publications, ensuring consistency and accessibility of the reviewed literature.
Publication year	Studies published between 2020 and 2025.	Studies published before 2020.	To capture recent advances and current trends in CNN-based video processing and feature extraction.
Publication type	Peer-reviewed journal articles and conference papers.	Review papers, books, book chapters, theses, dissertations, technical reports, editorials, and other non-peer-reviewed publications.	To ensure methodological quality, scientific rigor, and reproducibility of the reviewed evidence.
Study focus	Studies proposing, evaluating, or applying CNN-based methods for feature extraction in video processing tasks.	Studies not related to CNNs, video processing, or feature extraction, or whose primary focus falls outside the scope of this review.	To ensure consistency with the research objectives and the review protocol.
Research area	Computer Science, Artificial Intelligence, Engineering, Mathematics, Physics and Astronomy, Robotics, Imaging Science, and related disciplines.	Studies belonging to unrelated research areas without a direct contribution to CNN-based video processing.	To restrict the review to disciplines directly relevant to the proposed research topic.
Full-text availability	Studies with full-text access available.	Publications with unavailable or inaccessible full text.	To enable a complete methodological assessment and reliable data extraction.

The scope of this review is defined by three core methodological criteria: (1) convolutional neural networks as the primary architectural paradigm, (2) video processing as the application domain, and (3) feature extraction as the central methodological task. This tripartite framework ensures that all included studies are directly relevant to the intersection of these concepts. Studies addressing only a subset—such as action recognition without explicit feature extraction methods, or 3D convolutions without CNN-based processing—fall outside the defined scope regardless of their individual merit. Importantly, the eligibility criteria do not restrict the specific video analysis task. The screening procedure employed examples such as “object detection,” “security,” and “surveillance” as illustrative references to common application domains within the broader field of video-based feature extraction; these examples were not intended as exclusionary criteria. The final set of included applications—spanning action recognition, anomaly detection, sign language recognition, gait recognition, fire and smoke detection, video quality assessment, frame interpolation, and others—emerged naturally from the application of the inclusion criteria, which required only that studies address video processing with CNNs and explicit feature extraction. This breadth of applications reflects the methodological focus of the review rather than a narrow emphasis on any single domain. The resulting sample of 31 studies represents the complete set of works published within the specified timeframe that satisfy these eligibility criteria, providing sufficient breadth across architectural categories, application domains, and evaluation protocols to support the synthesis presented in this review.

The sample size of 31 studies reflects the deliberate methodological focus of this review rather than a limitation of the field’s breadth. The tripartite scope—requiring explicit mention of convolutional neural networks, video processing, and feature extraction—concentrates the analysis on works that directly address the intersection of these three methodological components. This targeted approach ensures that all included studies are directly relevant to answering the research questions posed in Table 1. Studies that do not explicitly address feature extraction methodologies, even if they address related topics such as action recognition or video understanding, fall outside the defined scope. The resulting sample provides sufficient breadth across architectural categories (2D-CNN, 3D-CNN, hybrid models, lightweight architectures), application domains (action recognition, anomaly detection, sign language recognition, gait recognition, fire and smoke detection, video quality assessment, frame interpolation), and evaluation protocols to support the synthesis presented in this review. While a larger corpus could be obtained by broadening the search terms, such an approach would introduce studies of tangential relevance and would not fundamentally alter the conclusions regarding the predominant architectural trends, datasets, and evaluation practices identified in this review.

3.3. Search Strategy

The first step was defining the exact search terms through an initial exploratory phase; they were then refined and adjusted to find an acceptable balance between sensitivity and specificity. Adhering to PRISMA 2020 guidelines [23], in order to capture the core concepts, a search string was constructed using a combination of keywords (e.g., convolutional neural networks, video processing, and feature extraction) and Boolean operators.

The deliberate selection of the tripartite keyword combination—“Convolutional Neural Network,” “video processing,” and “feature extraction”—was driven by the specific focus of this review on methodological contributions that explicitly address the intersection of these three concepts. This approach aligns with established best practices for systematic reviews in computer vision, where targeted search strings are recommended to ensure relevance and manageability of the literature corpus. Expanding the search to include broader terms such as “action recognition” or “video understanding” would have introduced studies that, while related, do not necessarily address explicit feature extraction methodologies—A central requirement for answering RQ1 through RQ4 [5,17]. Furthermore, the selected combination reflects the terminology used in foundational works on CNN-based spatiotemporal feature learning [12,13,14], ensuring alignment with the established literature. This targeted approach yielded a focused corpus of 208 initial records, which, after rigorous screening, provided sufficient breadth across architectural categories and application domains to support the synthesis presented in this review.

To ensure no relevant studies were omitted, no additional filters for study design or outcome were applied at this stage. Table 3 lists the final search strings for each individual database (e.g., Scopus, Web of Science (Clarivate), and the MDPI journal platform) mentioned in Section 3.2.

Table 3. Search strategies and filtering criteria applied to each academic database.

Database	Search Query	Filters Applied
MDPI	“Convolutional Neural Network” AND “video processing” AND “feature extraction”	Subjects: Engineering, Computer Science & Mathematics, Physics & Astronomy; Research Articles; Publication years: 2020–2025.
Scopus	TITLE-ABS-KEY(“Convolutional Neural Network” AND “video processing” AND “feature extraction”)	Publication years: 2020–2025; Subject areas: Computer Science, Engineering, Mathematics, Physics, and Decision Sciences; Document types: Articles and Conference Papers; Language: English.
Web of Science	“Convolutional Neural Network” AND “video processing” AND “feature extraction”	Publication years: 2020–2025; Document type: Article; Language: English; Categories related to Computer Science, Artificial Intelligence, Software Engineering, Engineering, Robotics, Imaging Science, Optics, Applied Physics, Remote Sensing, and Mathematics.

The remaining selection criteria (e.g., time period, document types, subject areas, language, etc.) have already been established in Section 3.2. The search was carried out in the second half of 2025, and the results were exported as a CSV file and then manually organized in a spreadsheet for further analysis.

3.4. Screening of Relevant Papers

The initial search yielded 208 articles, of which 23 were identified as duplicates. The remaining records were screened using a three-stage process. The first stage involved title screening, where peer-reviewed research articles addressing video processing with CNNs and explicit feature extraction were retained regardless of the specific application domain. The second stage required that articles provide quantifiable performance data for their proposed application. The final stage was full-text screening, which excluded articles that did not specify the CNN architecture or feature-extraction techniques. After applying these criteria, 31 articles were selected for inclusion. The distribution of articles across the three databases is shown in Figure 2, while the complete screening workflow is illustrated in Figure 3, and the number of articles at each stage is shown in Figure 4.

Figure 3. PRISMA 2020 flow diagram of the study selection process. Initial identification yielded 208 records, of which 23 duplicates were removed; after title, abstract, and full-text screening, 31 articles remained for the systematic review.

Figure 4. Number of articles retained at each screening phase: 208 initially identified, 185 after duplicate removal, 59 after title screening, 36 after abstract screening, and finally 31 after full-text assessment.

3.5. Feature Extraction Methods

For each of the 31 selected studies, we systematically collected parameters from every stage of the processing pipeline: acquisition, feature extraction, classification architecture, and evaluation metrics. Below, we summarize the extracted information. Figure 5 shows the types of data and the specific points we collected (for reference, see Section 3).

Figure 5. Categories of information extracted from each selected study, organized by processing stage (acquisition, feature extraction, classification, and performance evaluation). This structured extraction supports the quantitative and qualitative synthesis in this review.

• Acquisition and preprocessing: We recorded parameters that influence the model’s capacity and computational cost, e.g., the type of pooling used (e.g., max, average, or global).
• Feature extraction methods: We kept track of whether pre-trained models were used. For each article, we recorded the main techniques used for feature extraction—2D CNNs (e.g., VGG, ResNet, and Inception), 3D CNNs (e.g., C3D, I3D, and SlowFast), and hybrid models (CNNs combined with RNNs, LSTMs, GRUs, or transformers). We also extracted temporal modeling methods, such as optical flow and 3D convolutions. Finally, we recorded the kinds of spatial features extracted–edges, textures, and object parts.
• Classification architectures: For hybrid models, we recorded the specific building blocks, e.g., CNN + RNN, CNN + transformer, or two-stream fusion. We then assigned each model to one of the following categories based on the classification from Section 2: 2D-CNN, 3D-CNN, hybrid models, or emerging. We also collected additional information such as the number of fully connected layers, pooling layer type (e.g., Max, global average, or implied), and whether attention mechanisms were present.
• Performance evaluation: For detection or localization tasks, we recorded the mean Average Precision (mAP). For regression-based tasks, e.g., frame interpolation, quality assessment, we noted metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). We also collected standard classification metrics: accuracy, F1-score, precision, and recall. When available, we also collected inference speed (frames per second, FPS) and model size (number of parameters in millions).

All the extracted data were gathered into a structured spreadsheet (see Table S32), which allowed us to compare studies side by side. This systematic extraction provides the foundation for the quantitative and qualitative synthesis that we present in Section 4.

3.6. Classification of Hybrid Models

In this review, a hybrid model is described as a neural network that intentionally combines two or more fundamentally different components or processing paradigms into a single end-to-end framework, with each part providing unique capabilities to address a complex task [44]. This definition aids in classifying the architectures discussed in the literature.

4. Results

This section reviews the studies that fulfill the inclusion and exclusion criteria outlined in Table 2, following the PRISMA 2020 standard. Figure 3 displays the 31 qualifying studies, which are thoroughly analyzed to address the research questions listed in Table 1. Additionally, Figure 6 illustrates the most frequently observed architectural features in these studies.

Figure 6. Most common architectural characteristics in the reviewed studies: (a) types of pooling layers used (max pooling dominates), and (b) number of fully connected layers (most architectures use one or two; the multiple category represents studies where the exact integer count was not explicitly specified by the authors). These characteristics reflect common design choices in CNN-based spatiotemporal architectures.

Generally, the methods examined performed well, though with some variability. Most results clustered around 90-95%, with the red line showing a median of 92.28%. There was one outlier, with the lowest value at 60.23%, but overall, the range was from 72.1% to 100%; see Figure 7. In this section, each initial research question, listed in Table 1, is answered with the findings derived from the synthesis of the works.

Figure 7. Violin plot of the 28 accuracy values reported in the reviewed studies (three studies used alternative primary metrics). The median accuracy is 92.28%; values range from 60.23% to 100%, indicating variable performance across different tasks and architectures.

4.1. Overview of Selected Studies

Following the screening process described in Section 3, 31 articles published between 2020 and 2025 met our inclusion criteria. These studies address a diverse range of applications, including human action recognition [45,46], anomaly detection [47,48], sign language recognition [49], gait recognition [50], fire and smoke detection [51,52], video quality assessment [53], and video frame interpolation [54]. The authors of the selected articles represent a geographically diverse group, with contributions from Asia, Europe, and the Americas (see Figure 8), underscoring the global interest in spatiotemporal feature extraction. A summary of the selected works, including their architecture type, application, and dataset, is provided in Table 4.

Figure 8. Geographical distribution of the 31 selected studies by country of the first author. The map illustrates the global interest in CNN-based spatiotemporal feature extraction, with contributions from Asia, Europe, and the Americas.

Table 4. Summary of the studies included in the systematic review, grouped according to publication year, architecture, application, and experimental dataset.

Study	Year	Architecture	Application	Dataset
[45]	2025	ConvRBM–LSTM	Action recognition	KTH, UCF Sports, HMDB51
[46]	2025	CNN–BiLSTM	Human activity recognition	Custom
[55]	2025	CNN–LSTM	Motion prediction	UCF101
[56]	2024	ResNet50V2–GRU	Violence detection	Crowd, Movies, Hockey
[57]	2024	InceptionV3–MobileNetV2	Abnormal behavior detection	UTI student interactions
[58]	2024	MobileNetV2–Transformer	Video frame prediction	Vimeo90K
[48]	2023	Two-stream CNN	Anomaly detection	UCF Crime
[49]	2023	CNN–BiLSTM	Sign-language recognition	Arabic Sign Language
[59]	2023	CNN + Traditional CV	Action recognition	UAVGesture, DroneAction, UAVHuman
[60]	2022	VGG19–GRU/LSTM	Human activity recognition	TRECVID
[61]	2022	TSCNN–3DCNN–LSTM	Volleyball action recognition	Volleyball, UCF101, HMDB51
[62]	2022	Two-stream CNN	Activity recognition	HQ Fall Simulation, UP Fall Detection
[63]	2021	CNN–FlowNet	Action detection	UCF-24, JHMDB-21
[52]	2025	Lightweight CNN	Smoke detection	Custom
[64]	2025	Three-stream Transformer	Infrared action recognition	InfAR, IITR-IAR, UNISV
[51]	2025	Dual-stream CNN	Fire detection	Custom
[65]	2025	InceptionV3–Attention–GRU	Action recognition	UCF101, HMDB51
[66]	2025	CNN–ConvLSTM	Salient object detection	DAVIS, ViSal
[50]	2025	Swin Transformer–3D CNN	Gait recognition	CASIA-B, OUMVLP
[22]	2022	Lightweight Encoder–Decoder	Motion detection	CDNet2014
[67]	2021	3D CNN	Multimedia event detection	Columbia Consumer Video
[21]	2021	Efficient Pseudo-3D CNN	Action classification	UCF101
[68]	2022	CNN–GRU	Shoplifting detection	Custom
[69]	2022	CNN–BiLSTM	Excavator activity classification	Custom
[70]	2023	Causal CNN–LRCN	Data fusion	UCF101
[71]	2024	CNN–Deep BiLSTM	Human activity recognition	UCF11, UCF Sports, JHMDB
[47]	2024	Two-stream CNN	Anomaly detection	UCF Crime
[53]	2024	Two-stream Attention CNN	Video quality assessment	LIVE, CSIQ
[54]	2024	Transformer	Video frame interpolation	Vimeo90K, Middlebury, UCF101, SNU-FILM
[72]	2024	CNN–Autoencoder–LSTM/GRU	Video tampering detection	COMSATS
[73]	2025	Two-stream CNN–LSTM	Excavator activity recognition	Custom

The diversity of applications across these 31 studies—spanning action recognition, anomaly detection, sign language recognition, gait recognition, fire and smoke detection, video quality assessment, and frame interpolation—demonstrates that the sample captures the multidisciplinary nature of CNN-based spatiotemporal feature extraction research. Although the number of studies is modest, it reflects this review’s focused methodological scope rather than an exhaustive survey of the entire video-processing literature. The selected studies provide a representative cross-section of the field’s predominant architectural paradigms and evaluation practices.

4.2. RQ1. CNN Architectures for Spatiotemporal Feature Extraction

Throughout the review of the works, it was found that spatiotemporal feature extraction architectures based on CNNs can be grouped into four categories: (1) 2D CNN-based, (2) 3D CNN-based, (3) hybrid models that leverage multiple architectures, and (4) emerging architectures such as CNN–Transformer hybrids, which apply the classification method depending on the case study.

It is important to note that the proposed taxonomy is designed to capture the primary architectural paradigm emphasized by each study, rather than to establish strictly mutually exclusive mathematical categories. The inherent overlap between categories reflects the reality of modern deep learning research, where architectures increasingly combine multiple paradigms. For example, a model may incorporate both CNN and transformer components (hybrid) while also being optimized for edge deployment (lightweight), or a two-stream architecture may also be considered a hybrid model. In such cases, studies were classified according to their authors’ stated primary contribution and the dominant architectural component emphasized in their methodology. This approach follows established practices in systematic literature reviews where overlapping categories are acknowledged and addressed through clear classification criteria. The resulting taxonomy provides a practical framework for understanding the predominant trends in CNN-based spatiotemporal feature extraction, while recognizing the interconnected nature of modern architectural design.

• 2D-CNN-Based Approaches: A known shortcoming of this method is its difficulty in capturing long-range temporal dependencies [14], despite being computationally efficient. These methods rely on pre-trained 2D CNNs (e.g., VGG, Inception, and ResNet) for spatial feature extraction, followed by temporal aggregation via late fusion, early fusion, or recurrent networks. For instance, ref. [45] used a 2D Conv-RBM combined with an LSTM for temporal modeling, while [57] employed InceptionV3 and MobileNetV2, and then concatenated the features for classification.
• 3D-CNN-Based Approaches: These models apply 3D convolutions to learn spatiotemporal features jointly. Ref. [67] proposed a 3D-CNN framework for multimedia event detection, achieving a mean average precision of 83.05%. Ref. [61] incorporated an MS-3DCNN module within a larger fusion network for volleyball action recognition. Ref. [51] designed a dual-stream network with ShuffleNet V2 backbones, effectively combining spatial and temporal streams. Despite their ability to capture motion information, 3D-CNNs are computationally expensive and require large-scale training data [12].
• Hybrid Models: This is the most prevalent category (over 60% of the reviewed studies). Hybrid models integrate CNNs with other architectures to exploit complementary strengths.

  –

  CNN–RNN Hybrids: Ref. [46] introduced a three-fold model combining CNN, optimized RNN, and Bi-LSTM with an Artificial Ecosystem Customized Bald Eagle Optimization (AECBEO) feature-selection mechanism. Ref. [55] proposed CLGA-Net, which fuses CNN and LSTM with Genetic Algorithm and AdamW optimization. Ref. [60] used VGG-19 for sub-activity recognition, followed by a hybrid GRU-LSTM RNN for final classification. These models excel at capturing long-term temporal dependencies but are prone to overfitting and have high training complexity.

  –

  CNN–Transformer Hybrids: Ref. [58] combined MobileNetV2 with a transformer for video frame embedding prediction. Ref. [50] fused Swin Transformer with 3D-CNN for gait recognition, achieving state-of-the-art accuracy on CASIA-B. Ref. [64] developed a three-stream IFormer (hybrid CNN-ViT) network for infrared action recognition. Transformers effectively model global dependencies but increase the parameter count and may require more data.

  –

  Two-Stream/Multi-Stream Hybrids: Ref. [48] designed a two-stream architecture with YOLO-V4/VGG-16 for spatial features and FlowNet/VGG-16 for temporal features, followed by CNN-LSTM classification. Ref. [47] proposed TempoFuseNet, a two-stream framework with ResNet50 and a novel SG3I approach for temporal modeling. Ref. [69] used GoogLeNet for feature extraction and BiLSTM for activity classification of excavators. These methods achieve high accuracy by explicitly separating spatial and temporal processing, but they often require pre-computed optical flow, increasing inference time.

• Lightweight Architectures: This category includes models designed for efficiency or specialized tasks. Ref. [22] proposed a lightweight encoder-decoder based on a single linear bottleneck and pooling compensation, achieving real-time motion detection. Ref. [21] introduced an efficient group pseudo-3D network (GP3D) that reduces parameters while maintaining accuracy. Ref. [52] developed WISNet, a lightweight framework with BFBlocks for robust smoke detection. Ref. [53] used a two-stream CNN with attention for video quality assessment. Ref. [54] employed a transformer with parallel spatiotemporal attention for video frame interpolation. These architectures balance accuracy and computational cost, making them suitable for edge deployment.

4.3. RQ2. Benchmark Datasets Used for Evaluation

To address RQ2, we refer to Table 5, which summarizes the most frequently used datasets for validation in the studies. From this overview, it is evident that the UCF101 dataset [74] is the authors’ preferred option. This is due to its large number of 13,320 videos and 101 classes, which allow for robust and meaningful experimentation. It is most commonly found in works on the recognition of human actions. In the same field, we find, in second place, the HMDB51 dataset [75], with 6766 videos and 51 classes, reflecting complex and challenging real-world conditions for the proposed models. UCF Crime [76] is a dataset oriented towards real-world anomaly detection in surveillance videos, containing 1900 videos across 13 classes. Specialized datasets were also identified for action recognition using infrared (InfAR) [77], CASIA-B [78], a specialized dataset used in human gait recognition algorithms, volleyball [79] in sports applications for recognizing group activities, among others that use their own and customized datasets such as [46,69,72]. This heterogeneity demonstrates the particularity and complexity of each case study, highlighting the need for a standardized reference for dataset use that enables meaningful and valid comparisons between new proposals.

Table 5. Most frequently used datasets in the reviewed studies.

Dataset	Application Domain	Number of Videos	No. of Classes	References	Website
UCF101	Action recognition	13,320	101	[55,61,65]	UCF101, https://www.crcv.ucf.edu/research/data-sets/ucf101/ (accessed on 5 March 2026)
HMDB51	Action recognition	6766	51	[45,61,65]	HMDB51, https://serre.lab.brown.edu/hmdb51.html (accessed on 11 February 2026)
UCF Crime	Anomaly detection	1900	13	[47,48]	UCF Crim, https://www.crcv.ucf.edu/projects/real-world/ (accessed on 11 February 2026)
InfAR	Infrared action recognition	1000	12	[64]
CASIA-B	Gait recognition	13,640	124	[50]	CASIA-B, http://www.cbsr.ia.ac.cn/english/Gait%20Databases.asp (accessed on 5 March 2026)
Volleyball	Group activity recognition	4830	9	[61]	Volleyball, https://github.com/mostafa-saad/deep-activity-rec (accessed on 21 December 2025)
Vimeo90K	Frame interpolation	89,800 sequences	–	[54]	Vimeo90K, https://toflow.csail.mit.edu/ (accessed on 18 January 2026)
CDNet2014	Motion detection	31 video sequences	–	[22]	CDNet2014, http://changedetection.net/ (accessed on 21 December 2025)

Note: Only the most frequently used datasets are shown. Studies using custom, private, or less common datasets (e.g., KTH, UCF Sports, JHMDB, TRECVID, LIVE, CSIQ, etc.) are not included in this table; the full dataset mapping for all 31 studies is provided in the Supplementary Materials.

4.4. RQ3. Evaluation Metrics

The 31 studies analyzed in this review use common evaluation metrics, with accuracy reported in 94% of studies. Accuracy is not the only metric used in classification tasks; it is also reported alongside precision, recall, and F1-score analyses to assess the current class imbalance in tasks where anomalous activity detection is performed [47,56]. Work where the central application focuses on detection and localization usually employs mean Average Precision (mAP) as an evaluation metric [51,67]. Refs. [54,58], where regression-based frame interpolation is used, implement alternative metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Ref. [53] evaluates the performance, accuracy, and consistency of models using correlation metrics such as Pearson Linear Correlation Coefficient (PLCC) and Spearman’s Rank Order Correlation Coefficient (SROCC) in tasks related to video quality assessment.

Several studies also report computational metrics, such as frames per second (FPS) and parameter count (in millions), to demonstrate efficiency [22,52]. The distribution of accuracy values reported in the reviewed studies is shown in Figure 7, with a median accuracy close to 92.28%.

Table 6 summarizes the observed accuracy ranges for each architectural category, along with the number of studies in each category and their corresponding references. Table 7 summarizes the evaluation metrics used in the reviewed studies. The variability indicates the need for standardized evaluation protocols; this is a well-documented limitation in the literature of other systematic reviews.

Table 6. Accuracy ranges, number of studies, and references per architectural category.

Architecture Type	Min (%)	Max (%)	No. of Studies	References
3D-CNN	78.85	95.6	3	[51,61,67]
CNN-RNN Hybrid	83.33	100	9	[46,56,60,63,65,68,69,70,72]
CNN–Transformer Hybrid	86.07	99.25	5	[50,55,58,64,65]
Two-Stream/Multi-Stream	84.55	97.4	8	[47,48,50,53,61,62,66,73]
Lightweight/Efficient	90.16	96.69	4	[21,22,45,52]

Table 7. Evaluation metrics used in the reviewed studies, with the number of works and all corresponding references.

Metric	No. of Studies	References
Accuracy	27	[21,22,45,46,47,48,49,50,51,52,55,56,57,59,60,61,62,63,64,65,67,68,69,70,71,72,73]
Precision	8	[47,51,55,56,60,65,68,69]
Recall	8	[47,51,55,56,60,65,68,69]
F1-score	9	[46,47,51,55,56,60,65,68,73]
mAP	3	[51,63,67]
PSNR/SSIM	1	[54]
FPS/Parameters	4	[21,22,52,63]

Note: The counts are based on the 31 studies analyzed. Several studies report on multiple metrics used. Representative references are provided; the full list is available in the Supplementary Materials.

4.5. RQ4. Advantages and Limitations of CNN Architectures

Table 8 shows a comparison of the main advantages and limitations of the CNN-based architecture families identified in the reviewed studies. In general, it illustrates the difficulty of choosing an architecture that is superior in all video-processing scenarios; instead, each approach involves a trade-off between spatiotemporal representation capabilities, computational cost, and applicability to different real-world problems. While classic architectures, such as 2D CNNs and lightweight models, stand out for their efficiency and ease of implementation, 3D CNNs, hybrid CNN-RNN architectures, and CNN–Transformer architectures offer more advanced spatiotemporal representation, although they require more memory, computing power, and training data. Ultimately, the choice of architecture should be based on the characteristics of the problem, the available resources, and the balance between accuracy, efficiency, and temporal modeling capabilities.

Table 8. Comparison of the main advantages and limitations of the most common CNN-based architectures used for video processing.

Architecture	Advantages	Limitations	Evidence from Reviewed Studies
2D CNN	Low computational cost; benefits from ImageNet pre-training; simple implementation and fine-tuning.	Limited capability to capture long-range temporal dependencies; performance decreases on videos with complex motion patterns.	[45] (97.3% on KTH, 81.5% on HMDB51).
3D CNN	Learns spatial and temporal information jointly; excellent for motion representation; strong action-recognition performance.	High computational cost; requires large annotated datasets; limited transfer-learning flexibility. Lightweight variants alleviate these drawbacks.	[51,67].
CNN–RNN            (Hybrid)	Combines CNN spatial representations with temporal sequence modeling; suitable for long-term temporal dependencies.	Training instability due to vanishing/exploding gradients; high computational complexity; sensitive to limited training data.	[46].
CNN–Transformer (Hybrid)	Captures global context and long-range dependencies; achieves state-of-the-art performance in several applications.	Quadratic computational complexity; high memory consumption; requires large-scale datasets for training.	[50,58,64].
Two-Stream            Architectures	Separates spatial and temporal information; easily incorporates complementary modalities such as RGB, optical flow and depth.	Optical-flow computation is computationally expensive and limits real-time deployment.	[47].
Lightweight             Architectures	Designed for edge devices and real-time applications; excellent balance between accuracy and inference speed.	Reduced performance in highly dynamic or complex scenes due to lower model capacity.	[22,52].

Note: Representative studies are included to illustrate how recent works address the main limitations of each architectural family.

4.6. RQ5. Challenges, Emerging Trends, and Future Research Directions

Based on the reviewed literature, several clear trends and persistent challenges emerge.

4.6.1. Emerging Trends

The most prominent trend is the widespread hybridization of CNN-based architectures. Over 60% of the reviewed studies propose hybrid models that combine CNNs with other paradigms, such as CNN–RNN, CNN–Transformer, and two-stream designs. This shift reflects the growing recognition that no single architecture can adequately capture all aspects of spatiotemporal feature extraction. Closely related to hybridization is the increasing integration of attention mechanisms. Spatial and temporal attention are now commonly employed to help models focus on the most relevant regions and frames. For instance, Ref. [50] incorporated a Swin Transformer with 3D-CNN, while [53] embedded SE-blocks into a two-stream network. Attention not only improves interpretability but also often boosts accuracy.

It is evident that new, innovative trends are gaining traction; studies show a shift towards multimodal fusion analysis. Ref. [73] is one such trend, proposing combinations of RGB data with point clouds, RGB with audio, or RGB with optical flow, with RGB camera data held constant. Approaches that leverage redundant data sources prove robust in scenarios with complex illumination and occlusion. In contrast, approaches that prioritize edge deployment execute processes locally rather than relying on centralized services; this helps reduce latency, ensure privacy, and minimize dependence on an internet connection. In this sense, architectures such as GP3D [21], single-line bottleneck networks [22], and BFBlock [52] show high performance with fewer parameters, resulting in balanced accuracy and computational cost.

Finally, there is a growing trend toward self-supervised learning approaches as a promising avenue that minimizes the need for labeled datasets. This potential was observed in the proposal presented at [40], and despite its limitations, studies show a preference for these innovative trends.

4.6.2. Open Challenges

Although the literature shows an active area of study, there are still areas for improvement in state-of-the-art approaches. A wide variety of approaches exist; however, open questions and limitations remain, posing challenges in video processing, such as handling scenarios with high dynamic complexity and the presence of occlusions. In this regard, Ref. [54] reports limitations of its approach with respect to certain natural human body movements, particularly finger joint movements, highlighting the need for robust temporal models that compensate for heterogeneous joint motion. On the other hand, the low quality of video sources, coupled with the cost of professional equipment, is a recurring challenge in many studies, compounded by aggressive compression algorithms for efficient storage and by lighting conditions at different times of day. This is despite attempts such as the one proposed in [47], which addresses super-resolution video preprocessing and achieves good performance under adverse conditions.

Another current challenge is achieving generalization across domains, which is a complex task. In this regard, models trained on one dataset cannot be directly tested on another; adjustments must be made to the models, resulting in a lack of homogeneity across comparisons. The work presented in [72] shows that their manipulation detection method is difficult to generalize to videos with drastic scene changes, highlighting the need for domain adaptation and learning with limited data—both open and relatively unexplored challenges. Real-time performance on resource-constrained devices is a critical requirement for many applications, yet it is not yet fully met. While lightweight models exist, achieving consistent real-time inference (e.g., 30 FPS) on edge devices without sacrificing accuracy remains difficult; most efficient models are evaluated on high-end GPUs, and deployment on embedded hardware requires further optimization.

Finally, ethical and privacy concerns are increasingly recognized but seldom addressed in depth. Only a few studies mention privacy (e.g., [71]), and techniques such as federated learning or on-device processing are rarely discussed. As video analytics become more pervasive, this remains a critical area for future research.

4.7. Synthesis of Spatiotemporal Feature Extraction Approaches

The reviewed studies reveal several common patterns regarding spatiotemporal feature extraction, dataset selection, and temporal modeling. The following observations summarize the main findings.

Summary of key studies. Table 9 offers a curated overview of key studies, detailing the context of application, architecture, feature extraction approaches, datasets, performance metrics, and limitations. The authors categorize these limitations into four groups: (1) Dataset, (2) Acquisition, (3) Feature extraction, and (4) Results. The most common limitations relate to issues with the feature extraction methods and techniques used, as well as the way results are derived. Fewer limitations are associated with dataset quality and data acquisition processes.

Table 9. Summary of key studies included in the systematic review. Limitations categories: (1) Dataset, (2) Acquisition, (3) Feature extraction, and (4) Results.

Year	Application	Architecture Type	Feature Extraction	Dataset	Performance (Accuracy)	Limitations
2025	Action recognition	2D Conv-RBM + LSTM	2D Conv-RBM, smart frame selection	KTH, UCF Sports, HMDB51	97.3%, 94.8%, 81.5%	(1) small sample size; (3) limited temporal modeling for complex actions [45]
2025	Human activity recognition	CNN + optimized RNN + Bi-LSTM	MBC, LGTrP, BPPC, C-MBP with AECBEO selection	Custom (Dataset 1,2)	97%	(2) high computational complexity; (4) real-time not demonstrated [46]
2025	Motion prediction	CNN + LSTM + GA + AdamW	CNN (spatial), LSTM (temporal)	UCF101	99.25%	(2) requires high-performance hardware; (3) hyperparameter sensitivity [55]
2024	Violence detection	ResNet50V2 + GRU	ResNet50V2 (transfer learning)	Crowd, Movies, Hockey	100% (Movies), 96.3% (Hockey)	(2) increased test time; (4) limited to weapon-independent violence [56]
2024	Abnormal behavior detection	Hybrid InceptionV3 + MobileNetV2	InceptionV3, MobileNetV2	UTI student interactions	92.7%	(2) limited to single dataset; (4) no real-time evaluation [57]
2024	Gait recognition	Swin Transformer + 3D-CNN	3D CNN (local), Swin Transformer (global)	CASIA-B, OUMVLP	97.4%, 97.1%	(3) high model complexity; (4) struggles with significant gait changes [50]
2023	Anomaly detection	Two-stream YOLO-V4/ VGG-16 + FlowNet/ VGG-16 + CNN-LSTM	YOLO-V4, VGG-16, FlowNet	UCF Crime	95.6%	(2) multiple hyper-parameters; (3) requires optical flow pre-computation [48]
2022	Volleyball action recognition	Fusion CNN (TSCNN +3DCNN +LSTM)	MS-3DCNN, spatial/ temporal ResNet101	Volleyball, UCF101, HMDB51	89.9%, 85.5%, 78.3%	(3) feature fusion ratio must be tuned; (4) accuracy drop on complex backgrounds [61]
2022	Real-time motion detection	Lightweight encoder-decoder (single linear bottleneck)	Modified VGG16, pooling compensation	CDNet2014	95.74%	(3) still relies on datasets; (4) not completely unsupervised [22]
2021	Multimedia event detection	3D-CNN	3D convolutions	Columbia Consumer Video	mAP 83.05%	(1) small dataset; (3) requires hyperparameter tuning [67]
2024	Video tampering detection	2D-CNN + autoencoder + LSTM/GRU	Spatiotemporal average pooling, autoencoder	COMSATS (custom)	90.73%	(1) dataset not public; (3) cannot detect multiple tampering in one video [72]
2025	Multi-modal excavator activity	Two-stream CNN-LSTM (RGB + point cloud)	CNN on RGB and point cloud, Farneback optical flow	Custom (12 sites)	94.67%	(2) limited to single excavator; (4) performance degrades with multiple machines [73]

Feature extraction methods. To provide a consolidated view of the feature extraction techniques employed across the reviewed literature, Table 10 categorizes the primary feature extraction methods, the number of studies that adopt each method, and the corresponding references. This summary underscores the diversity of approaches used for spatiotemporal feature learning.

Table 10. Feature extraction methods used in the 31 reviewed studies, with counts and references.

Feature Extraction Method	No. of Studies	References
2D CNN (spatial only)	6	[22,45,57,60,69,72]
3D CNN (spatiotemporal)	3	[51,61,67]
CNN + RNN/LSTM/GRU	11	[46,49,55,56,60,63,65,69,71,72,73]
CNN + Transformer	4	[50,58,64,65]
Two-Stream (RGB + Optical Flow)	5	[47,48,53,62,73]
Lightweight/Efficient CNNs	4	[21,22,45,52]
Hybrid (CNN + Attention + RNN)	3	[50,65,66]
Transformer-only	1	[54]

Note: Some studies employ multiple feature extraction methods and are counted in more than one category.

Spatial features explicitly extracted. Beyond the architectural choices, we also recorded what kinds of spatial features the studies explicitly extracted. Table 11 summarizes the findings. Most studies implicitly learn spatial features via CNNs, but a few explicitly name edges, textures, or object parts. As shown in Table 11, texture descriptors (e.g., MBC and LGTrP) appear in [46], while YOLO-V4 in [48,62] explicitly extracts object parts. Only one study [45] mentions “edges” as a spatial feature.

Table 11. Spatial feature types explicitly extracted in the reviewed studies.

Spatial Feature	No. of Studies	References
Edges	1	[45]
Textures	1	[46]
Object parts	2	[48,62]

Temporal modeling methods. In addition to studying spatial features, it is necessary to examine temporal modeling techniques. Table 12 summarizes the two methods for extracting temporal features: the first is optical flow, which focuses on understanding the calculated movement—the displacement between each subsequence of frames—and the second is 3D convolution, which learns spatiotemporal patterns obtained directly from the data in a unified way. Furthermore, Table 10 explores feature extraction methods in general, from classical models to hybrid models such as CNN + RNN/LSTM/GRU, among others, which are present in 11 of the 31 studies analyzed. As shown in Table 12, optical flow is the most frequently explicitly used temporal modeling technique, appearing in five studies (e.g., [48,62]). Three studies employ 3D convolutions for joint spatiotemporal feature learning, including [61,67]. Recurrent units (LSTM and GRU) are not included here because they are already accounted for in the hybrid category of Table 10.

Table 12. Temporal modeling methods explicitly employed in the reviewed studies.

Temporal Method	No. of Studies	References
Optical flow	5	[47,48,53,62,73]
3D convolutions	3	[51,61,67]

Overall findings. To provide a consolidated visual overview of the main findings from this systematic review, Figure 9 presents a schematic that connects the four identified CNN architecture categories, the most frequently used datasets, and the predominant evaluation metrics. This diagram summarizes the key relationships uncovered in the analysis of the 31 reviewed studies.

Figure 9. Schematic summary of the main findings from the systematic review. The diagram connects the four identified CNN architecture categories (2D-CNN, 3D-CNN, hybrid models, and lightweight architectures) to the most frequently used datasets (UCF101, HMDB51, UCF Crime, and specialized datasets) and the predominant evaluation metrics (accuracy, F1-score, mAP, PSNR/SSIM, and computational metrics). The bottom panel highlights key insights derived from the 31 analyzed studies.

4.8. Statistical Considerations for Cross-Study Performance Comparisons

This section presents additional statistical analyses to support the interpretation of the reported performance metrics. Specifically, it examines the sources of heterogeneity among the reviewed studies, assesses the distribution characteristics of the reported accuracies, and provides a statistical basis for their descriptive interpretation.

4.8.1. Heterogeneity of Reported Performance

While performance metrics such as accuracy are commonly reported in the reviewed literature, direct comparisons between studies should be interpreted with caution, as the included works differ substantially in terms of datasets, application domains, class distribution, evaluation protocols, and application case complexity. Therefore, the analyses presented in this section aim to characterize the distribution of reported results, rather than to establish statistical superiority among architectural categories.

Table 7 shows the performance metrics reported in the selected studies, providing valuable insight into the effectiveness of CNN-based architectures. However, directly comparing them requires careful analysis, as the studies vary in their experimental setups. It is important to note that, unlike benchmark assessments with a uniform protocol, systematic reviews gather results under largely heterogeneous conditions, making direct numerical comparisons difficult.

The included studies consider a variety of application domains, including tasks such as action recognition, anomaly detection, video surveillance, gait analysis, frame interpolation, video quality assessment, and other video compression tasks (see Table 4). Furthermore, they employ different reference datasets, class distributions, preprocessing strategies, evaluation protocols, training configurations, and experimental objectives. Consequently, the reported accuracy values reflect not only the intrinsic capabilities of a particular CNN architecture but also the characteristics, configurations, researcher preferences, and the specific complexity of each case study.

Table 13 summarizes the main sources of methodological heterogeneity identified in the reviewed studies. These factors demonstrate that the reported performance metrics were obtained under substantially different experimental conditions. Consequently, the statistical analyses presented in the following sections aim to characterize the distribution of reported results, rather than to establish a direct superiority among the different CNN architectures.

Table 13. Main sources of heterogeneity identified among the reviewed studies and their implications for cross-study performance comparisons.

Source of Heterogeneity	Description and Implications
Application domain	The reviewed studies cover diverse application areas, including action recognition, anomaly detection, video surveillance, gait recognition, frame interpolation, video quality assessment, and other video understanding tasks. Each application presents different objectives and levels of complexity, making direct performance comparisons inherently challenging.
Benchmark datasets	Different public datasets were employed (e.g., UCF101, HMDB51, Kinetics, Something-Something, and domain-specific datasets), each characterized by distinct scene complexity, number of samples, recording conditions, and class distributions. Consequently, reported performance is strongly influenced by dataset characteristics.
Class distribution	The reviewed studies addressed problems ranging from binary classification to large-scale multi-class recognition. Dataset imbalance and varying numbers of target classes directly affect the difficulty of the learning task and the resulting performance metrics.
Evaluation metrics	Although classification accuracy was the most frequently reported metric, several studies evaluated performance using F1-score, precision, recall, mAP, IoU, PSNR, SSIM, or task-specific measures. Therefore, performance should be interpreted within the context of the evaluation criterion employed.
CNN architecture	The analyzed studies include conventional 2D CNNs, 3D CNNs, hybrid architectures (e.g., CNN-LSTM and CNN–Transformer), and lightweight models. These architectural families differ substantially in temporal modeling capability, computational complexity, and design objectives.
Experimental protocols	Training strategies, validation procedures, preprocessing methods, data augmentation techniques, and hyperparameter optimization varied considerably across studies. Such methodological differences further limit the direct comparability of reported results.
Performance reporting	Several studies reported only the best experimental configuration, whereas others presented average performance across multiple runs or cross-validation folds. Consequently, variability estimates are not consistently available throughout the literature.

Consequently, the statistical analyses presented in this section were designed to describe the distribution of reported performance, not to determine definitive superiority among architectural families. Therefore, robust descriptive statistics and distribution analysis were used to provide a more reliable interpretation of the reported accuracies, considering the diversity of the reviewed studies.

In that regard, the following subsections evaluate the statistical properties of the reported precision values and analyze the suitability of the descriptive summaries of performance in the reviewed studies.

4.8.2. Descriptive Statistical Analysis

Given the heterogeneity described above, descriptive statistics were used to summarize the accuracy values reported in the reviewed studies. Since extreme values (lower/upper) can significantly affect conventional statistics such as the mean and standard deviation, robust measures of central tendency and dispersion were considered more appropriate tools for characterizing the performance reported in the literature.

Table 14 summarizes the statistics for the reported accuracy values. The median accuracy (0.9228), along with the interquartile range (IQR) of 0.0853, provides a representative summary of central tendency and dispersion while reducing the influence of extreme observations. Based on the IQR criterion, only one study was identified as a potential outlier (accuracy = 0.6023), while the remaining observations were distributed within the expected range. These findings support the use of robust descriptive statistics to characterize the reported performance of heterogeneous studies.

Table 14. Robust descriptive statistics of the reported accuracy values extracted from the reviewed studies.

Statistic	Value	Interpretation
Number of observations (n)	27	Studies reporting classification accuracy.
Mean	0.9014	Arithmetic average of the reported accuracy values.
Median	0.9228	Robust estimate of the central tendency, minimally affected by extreme observations.
Standard deviation	0.0836	Overall variability of the reported accuracies.
Minimum	0.6023	Lowest reported accuracy.
Maximum	1.0000	Highest reported accuracy.
First quartile (Q1)	0.8707	Twenty-five percent of the reported accuracies are below this value.
Third quartile (Q3)	0.9560	Seventy-five percent of the reported accuracies are below this value.
Interquartile range (IQR)	0.0853	Robust measure of dispersion corresponding to the middle 50% of the observations.
Lower IQR bound	0.7428	Threshold used to identify potential lower outliers.
Upper IQR bound	1.0839	Threshold used to identify potential upper outliers.
Detected outliers	1	One study (accuracy = 0.6023) was identified as a potential outlier using the IQR criterion.

4.8.3. Distributional Analysis of Reported Accuracy Values

To determine whether the parametric descriptive measures were appropriate, the distribution of the reported precision values was analyzed in greater detail. In addition to visual inspection, four complementary normality tests were conducted to assess whether the data followed a Gaussian distribution. The analysis also examined measures of skewness and kurtosis to characterize the overall shape of the distribution.

Table 15 summarizes all the statistical tests, which consistently rejected the null hypothesis of normality. These results indicate that the reported precision values do not follow a Gaussian distribution. The multiple normality tests are consistent in their results, increasing confidence that the observed distribution differs from normality, rather than reflecting the characteristics of a single statistical procedure.

Table 15. Normality assessment of the reported accuracy values.

Test	Statistic	p-Value	Decision
Shapiro–Wilk	0.8312	0.00050	Reject normality.
D’Agostino–Pearson	23.0471	$9.89\times {10}^{-6}$	Reject normality.
Jarque–Bera	35.8758	$1.62\times {10}^{-8}$	Reject normality.
Anderson–Darling	1.2044	–	Distribution exceeds the critical value, indicating non-normality.

This non-normal behavior is further supported by the distribution’s shape descriptors. The reported precision values showed a negative skewness of −1.988, indicating a longer left tail, associated with a small number of lower-performing studies. Additionally, the positive kurtosis, with a value of 5.381, suggests a peaked distribution, with heavier tails than expected under a normal distribution, reflecting the presence of extreme observations.

The reported performance metrics were analyzed using robust descriptive statistics, such as the median and IQR, which are less susceptible to skewed distributions and isolated extreme values compared to the mean and standard deviation. Finally, the statistical analyses in this review are intended to describe the overall distribution of the reported performance, not to determine statistical superiority among the CNN architectural categories.

4.9. Statistical Implications and Sensitivity Analysis

Statistical analyses show that the reported accuracy values come from very different experimental conditions across studies and do not follow a Gaussian distribution. Therefore, performance metrics should be interpreted descriptively, rather than indicating a clear statistical superiority among CNN architectures. Variability is influenced by differences in datasets, domains, evaluation protocols, and task complexity. Consequently, robust statistics such as the median and IQR offer more reliable summaries than isolated minimum, maximum, or average accuracy values.

To verify the robustness of these results, an additional sensitivity analysis was carried out that temporarily excluded the single outlier, detected by the IQR criterion.

A further sensitivity analysis was performed after temporarily excluding the single outlier detected by the interquartile range criterion (precision = 0.6023). As shown in Table 16 and Table 17, removing this outlier significantly improved the distribution characteristics of the dataset. All normality tests failed to reject the null hypothesis of normality, and skewness and kurtosis decreased markedly. However, the robust descriptive statistics remained fairly stable, with the median changing from 0.9228 to 0.9249 and the IQR decreasing slightly from 0.0853 to 0.0732. These results suggest that the outlier primarily affected the distribution assumptions, not the central tendency of the performance. Therefore, it was decided to retain the full dataset in all subsequent analyses to ensure the integrity of the review, even though the sensitivity analysis provides additional evidence supporting the reliability of the reported descriptive statistics.

Table 16. Comparison of descriptive statistics between the complete dataset and the sensitivity analysis.

Statistic	Complete Dataset ($\mathit{n}=27$)	Without Outlier ($\mathit{n}=26$)
Mean	0.9014	0.9129
Median	0.9228	0.9249
Standard deviation	0.0836	0.0596
Minimum	0.6023	0.7630
Maximum	1.0000	1.0000
First quartile (Q1)	0.8707	0.8828
Third quartile (Q3)	0.9560	0.9560
Interquartile range (IQR)	0.0853	0.0732
Skewness	−1.9879	−0.8857
Kurtosis	8.3814	3.4870

Table 17. Normality tests before and after sensitivity analysis.

Test	Statistic	p-Value	Statistic	p-Value
	Complete Dataset		Without Outlier
Shapiro–Wilk	0.8312	0.0005	0.9411	0.1424
D’Agostino–Pearson	23.0471	<0.0010	4.2725	0.1181
Jarque–Bera	35.8758	<0.0010	3.0457	0.2181
Anderson–Darling	1.2044	—	0.4914	—

5. Discussion

Section 4 offers a comprehensive overview of CNN-based methods for spatiotemporal feature extraction, along with relevant datasets and evaluation metrics. Building on these results, the subsequent discussion interprets major architectural trends, assesses the methodological quality of the included studies, and suggests future research directions. The architecture classification introduced here provides a unified taxonomy, filling a notable gap identified in earlier surveys [16,24].

5.1. Architectural Trends and Hybrid Models

In Section 4.2, the results supporting the answer to RQ1 are presented, positioning 3D CNNs as a well-founded approach used in proposals that are strengthened by adding the spatiotemporal component, thus efficiently improving the analysis of video sequences. However, evidence shows that hybrid models are gaining ground in recent studies. Hybrid models based on CNN-RNNs, such as CLCA-Net [55] and the one proposed by Deotale et al. [60], combine the advantages of spatial compression offered by CNNs with the integration of RNNs for the analysis of temporal dynamics. Another hybrid approach observed in the review is the CNN–Transformer model, which arises from the need to detect local details using CNNs and, for global details, using transformers. A clear example is the Swin Transformer-3DCNN model [50] and TIFAR-Net [64], which have demonstrated stable (see Table 6) and computationally balanced performance well above average. Transformer integration is becoming increasingly evident in computer vision-related tasks, particularly in sequence modeling [27,38].

Similarly, the analysis of advantages and limitations described in RQ4 (see Section 4.5) broadens the perspective on these trends. Although 3D CNNs and hybrid models offer robust yet complex approaches with greater accuracy ranges compared to classical approaches, they come at the cost of high computational costs and the need for a large number of parameters. This calls into question their implementation in applications where real-time analysis is imperative or where processing resources are limited. Conversely, lightweight classical models (such as [22,52]) remain a viable alternative for maintaining a balanced relationship between accuracy and speed, with lower performance in complex scenarios with high motion dynamics. This dilemma highlights the latent challenges that future research must continue to address.

5.2. Datasets and Evaluation Practices

Regarding RQ2 (see Section 4.3), the predominance of datasets like UCF101 [74] and HMDB51 [75] is consistent with the field’s focus on human action recognition. However, the use of specialized datasets, such as UCF Crime [76] for anomaly detection and infrared datasets, such as InFAR [77] for specific modalities, indicates a diversification of application domains. A significant limitation is the lack of standardized benchmarks for tasks beyond action recognition, making cross-study comparisons difficult.

The analysis of evaluation metrics for RQ3 (see Section 4.4 and Section 4.8) shows that accuracy remains the most commonly reported metric. Still, its limitations, especially in multi-class or imbalanced datasets, are well recognized. Studies increasingly report precision, recall, F1-score, and specificity to provide a more nuanced view of model performance [46,56]. For regression-based tasks, domain-specific metrics like PSNR and SSIM are essential [54,58]. The variability in reported evaluation metrics underscores the need for more standardized evaluation protocols, as highlighted in the limitations of previous reviews [23].

5.3. Applications and Future Outlook

The emerging trends and open challenges identified in RQ5 (see Section 4.6) point to promising future directions. The move toward multimodal fusion [73] and the integration of attention mechanisms [53] are likely to yield more robust and context-aware models. However, challenges such as handling low-quality video [47], domain generalization [72], and privacy concerns remain significant obstacles requiring innovative solutions.

The findings of this review have implications across a wide range of video-based applications, including surveillance, autonomous driving, healthcare monitoring, and human–computer interaction. As spatiotemporal feature extraction methods continue to evolve, their integration into real-world systems will depend on addressing the trade-offs between accuracy, computational efficiency, and scalability.

5.4. Risk of Bias Assessment

To evaluate the methodological quality of the included studies and to identify potential sources of bias that could influence the interpretation of the evidence, we employed a critical appraisal tool adapted from the Joanna Briggs Institute (JBI) SUMARI Checklist for Quasi-Experimental Studies [80]. Rather than serving as a definitive measure of overall study quality, the checklist was used as a structured framework to assess key methodological domains, including internal validity, participant selection, confounding factors, outcome measurement, follow-up completeness, and statistical analysis. This approach enabled a transparent and systematic assessment of methodological strengths and potential limitations across the included studies. The appraisal criteria are summarized in Table 18.

Table 18. Scoring criteria for the JBI SUMARI-inspired risk of bias assessment.

Answer	Value	Description
Yes	1	The criterion was fully met.
Unclear	0.5	Insufficient information to judge.
No	0	The criterion was not met.
Not applicable	–	The item did not apply to the study design (excluded from average).

Following the recommendations of the JBI methodology, the appraisal was interpreted at the item level to identify methodological strengths and potential risks of bias, rather than relying solely on an overall numerical score.

Although an overall score was calculated to facilitate comparison between studies, the main interpretation was based on individual assessment domains rather than the total score alone, in accordance with the intended use of JBI’s critical appraisal framework.

In this regard, an average score is determined for each study (the sum of the scores divided by the number of applicable items). The overall risk of bias is then classified as low, moderate, or high according to the thresholds shown in Table 19.

Table 19. Risk of bias classification based on average score.

Risk Level	Average Score	Description
Low risk	>0.8	The study meets most criteria with minimal risk of bias.
Moderate risk	0.5–0.8	The study shows some methodological weaknesses; interpretation should consider potential bias.
High risk	<0.5	The study has substantial flaws; findings may be unreliable.

Because the JBI SUMARI Checklist was originally developed for quasi-experimental studies, each appraisal item was operationalized to evaluate the methodological rigor of primary deep learning studies while preserving its original intent [80]. The adapted criteria focused on experimental design, fairness of comparisons, consistency of evaluation, reliability of reported outcomes, completeness of reporting, and appropriateness of statistical analyses. The items were adapted as follows:

• Temporal precedence: Is it clear what the “cause” (the proposed model/architecture) and the “effect” (the reported performance) are, with no confusion about which variable comes first?
• Control group: Was there a baseline or alternative method used as a control for comparison?
• Comparability of groups: Were the datasets and evaluation conditions (e.g., train/test splits and data pre-processing) similar across compared groups?
• Similar treatment/care: Did the groups receive the same experimental conditions except for the intervention of interest (the proposed method)?
• Multiple outcome measurements: Were multiple evaluation metrics (e.g., accuracy, F1-score, and precision) reported, and were results presented for both training and test sets?
• Outcomes measured consistently: Were the same metrics and evaluation protocols used for all compared methods?
• Reliability of outcome measurement: Were the reported metrics standard in the field (e.g., accuracy, mAP, and FPS) and clearly defined?
• Follow-up completeness: If a study reported results for multiple runs or splits (e.g., cross-validation), were the results fully reported and any dropouts explained?
• Appropriate statistical analysis: Were appropriate statistical tests (e.g., confidence intervals and significance tests) used to compare methods, or at least descriptive statistics (mean and standard deviation) provided?

Table S32 of the Supplementary Materials to this work details the scores obtained for each study in the modified risk of bias analysis of the JBI SUMARI-proposed framework. The scores for this adapted version of the risk of bias analysis are based on 9 questions (Q). For questions Q1–Q4 and Q8, a score of 1 is given for “Yes,” 0.5 for “Not Clear,” 0 for “No,” or ‘–’ for cases where none of the above options apply (see Table 18). Questions Q5–Q7 consist of segmented sub-responses, the arithmetic mean of which is calculated to determine the score. The final score obtained for each of the 31 analyzed studies is defined as the Average, which is the arithmetic mean of questions Q1–Q9. Table S32 also indexes each study with its respective analysis, as defined in Tables S1–S31, for greater transparency regarding the analysis performed using the guidelines proposed by the JBI SUMARI. It is imperative to mention that Q8 refers to exhaustive follow-up, a characteristic that does not apply to any of the analyzed studies; therefore, it is shown as zero. It was not excluded from the analysis for transparency, and this does not affect the average value reported by the studies. A heatmap visualization (see Figure 10) shows that most studies performed well on items related to question clarity (Q1), consistency in outcome measurement (Q6), and reliability of metrics (Q7). Lower scores were observed on items related to the control group (Q2), multiple outcome measurements (Q5), and statistical analysis (Q9). Notably, Q8 (Follow-up completeness) did not apply to any of the computational studies. However, it is displayed as 0 in the heatmap because the visualization tool does not support null entries; it was excluded from the average risk score and does not influence the overall assessment. The average risk score across all 31 studies was 0.76, placing the overall body of evidence in the moderate-risk category. A moderate risk score (0.76) suggests that the evidence from the studies is mostly reliable and adequate to underpin the main conclusions of this review (see Table 19). However, there are some methodological weaknesses, especially in areas like baseline comparisons, multiple performance metrics, and statistical testing. For researchers, this means that while the overall trends and findings are likely to be robust, individual study results should be interpreted with caution. The moderate score also underscores the need for future work to adopt more rigorous experimental designs and transparent reporting to increase the certainty of the evidence. This indicates that while the included studies generally adhered to sound methodological practices, improvements in reporting—especially regarding baseline comparisons, multiple evaluation metrics, and statistical testing—would strengthen future research. The uniform distribution of scores across the heatmap suggests that a few low-quality papers do not unduly influence the review’s conclusions.

Figure 10. Heatmap summarizing the risk of bias assessment of the 31 included studies based on the nine JBI-adapted quality criteria. Individual study scores range from low (purple) to high (pink) risk of bias. The overall mean score of 0.76 indicates moderate methodological quality across the reviewed studies [21,22,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73].

5.5. Future Research Directions Identified in the Literature

The review of the 31 papers reveals consistent priorities for improving how spatiotemporal features are extracted with CNNs in video processing. These priorities can be broadly categorized into four classes:

• Larger and more diverse datasets. At least 15 studies call for collecting data across varied environments, lighting conditions, and scenarios to improve generalizability [21,45,46,49,51,67,68,69,73]. Several authors also advocate for publicly available benchmarks that would enable fairer cross-study comparisons.
• Real-time and edge deployment. Reducing inference latency and model size for deployment on resource-constrained devices (edge, IoT, and mobile) remains a high priority [22,47,52,54,55,61,63,65]. Proposed strategies include lightweight operator design, pruning, and hardware-aware optimization.
• Multimodal fusion and auxiliary signals. Integrating complementary modalities—audio [58], point clouds [73], infrared [64], weather information [52], or contextual data [47]—is seen as a promising route to improve robustness under adverse conditions.
• Self-supervised, unsupervised, and domain-adaptation methods. Several works highlight the need for techniques that reduce reliance on manual annotations, such as self-supervised learning, domain adaptation, and fully unsupervised detection [22,66,71,72].

On the other hand, research lines are emerging that focus on extending classical models for application in complex scenarios with multiple people or moving objects [59,72,73], proposing improvements in the design of temporal models for long-range dependency analysis [60,71], and, though less frequent, increasingly prominent approaches that address ethical and privacy concerns [71]. This evidence shows that efforts are shifting beyond simply improving model accuracy to focus on practical, implementable, and reliable proposals, all while maintaining a balance.

5.6. Review Limitations

Given the heterogeneity of the analyzed studies, both in their methodological approaches and the evaluation metrics employed, this review was conducted using a narrative approach, without incorporating meta-analyses or sensitivity studies.

According to the inclusion and exclusion criteria defined in Table 2, this systematic review included 31 studies out of a total of 208 identified through search queries applied to three academic databases considered references in the field: Scopus, the MDPI platform, and Web of Science. However, relevant research hosted in other repositories was not included due to factors such as publication language, access restrictions, questionable publisher practices, or inclusion in low-impact indexes. This situation could create gray areas or introduce bias into the results.

Furthermore, the search was limited to studies published between 2020 and the end of 2025 to focus the analysis on recent publications. However, this time frame may affect the coverage of the state of the art by excluding fundamental studies published before that period (Table 20).

Table 20. Summary of future research directions explicitly mentioned in the reviewed studies.

Ref.	Future Direction(s) Proposed
[45]	Improve model performance and efficiency; extend the approach to other application domains.
[46]	Incorporate additional texture and shape features; evaluate on larger databases and real-time datasets.
[55]	Improve real-time performance; scale to large datasets; optimize for edge computing and IoT.
[56]	Explore other deep-learning methods and feature-extraction techniques; obtain additional datasets and annotations.
[58]	Increase training-data diversity; integrate MFCC audio features; investigate multimodal transformer frameworks.
[48]	Improve detection accuracy and efficiency; incorporate attention mechanisms and GANs; compare multiple deep-learning architectures.
[49]	Extend to signer-independent recognition; support other sign languages; collect larger sentence-level datasets; reduce computational cost for real-time applications.
[59]	Improve performance on complex datasets; investigate multi-human feature representations; model interactions among individuals.
[60]	Automate sub-activity training; apply transfer learning; extend to group action recognition.
[61]	Improve robustness on low-quality and complex-background videos; extend to real-time recognition.
[62]	Improve handling of partial occlusions; evaluate alternative feature extractors; validate in smart-home environments.
[63]	Investigate lightweight spatiotemporal architectures; perform temporal action localization; optimize for edge devices.
[52]	Integrate satellite imagery and sensor data; include additional environmental variables; investigate transfer and online learning.
[64]	Combine RGB and infrared information to improve recognition accuracy.
[51]	Optimize network architectures; develop lightweight optical-flow estimation; establish larger fire-video datasets.
[65]	Reduce processing time for real-time surveillance; develop application-specific models.
[66]	Investigate domain adaptation and human-centric methods to improve generalization and interpretability.
[50]	Improve computational efficiency; evaluate additional attention mechanisms and feature extractors.
[22]	Develop a fully unsupervised real-time motion-detection algorithm; further reduce model size.
[67]	Evaluate on larger datasets; investigate RNN/LSTM and attention-based models.
[21]	Train on larger-scale datasets; optimize implementation to reduce latency.
[68]	Focus on practical deployment; collect a larger and more diverse shoplifting dataset; evaluate additional architectures.
[69]	Improve robustness under varying illumination; incorporate multiple cameras; collect data from diverse construction sites.
[70]	Investigate more advanced CNN architectures; analyze the influence of hyperparameters.
[71]	Improve temporal-context fusion; develop privacy-preserving methods; investigate 3D CNN and transformer backbones.
[47]	Integrate multimodal contextual information; enable real-time processing.
[53]	Extend the approach to no-reference video quality assessment; optimize practical video systems.
[54]	Support multiple input frames; interpolate arbitrary time steps; improve performance on human-body motion, particularly finger joints.
[72]	Detect multiple inter-frame tampering types simultaneously; address frame-duplication attacks.
[73]	Recognize activities of multiple excavators simultaneously; expand data collection to diverse sites; improve occlusion handling and contextual understanding.

Note: Only the 30 studies that provided explicit, non-generic statements about future work are included. Ref. [57] did not mention any specific future direction and is, therefore, omitted.

The initial search in the selected repositories was conducted using strict criteria, considering only works that explicitly mentioned the use of convolutional neural network-based techniques in video-processing applications aimed at feature extraction. As a result, a preliminary set of 208 works that met the established criteria was identified. Although this process ensured the relevance of the included studies, it may also have excluded research based on other deep learning architectures or related applications that were not specifically focused on feature extraction from video sequences. The search strategy was intentionally focused on studies explicitly combining the three core concepts of convolutional neural networks, video processing, and feature extraction. This targeted approach, while ensuring methodological relevance, may have excluded related work that uses alternative terminology or addresses adjacent research questions (e.g., action recognition without explicit feature extraction methods, or video understanding without CNN-based processing). As acknowledged in the literature, systematic reviews in rapidly evolving fields must balance comprehensiveness with relevance to avoid overwhelming the synthesis with marginally related works. Researchers should consider this scope delimitation when interpreting the findings and applying them to their specific contexts.

The screening procedure intentionally did not restrict the final set of applications, as the review’s methodological focus on spatiotemporal feature extraction with CNNs is applicable across diverse video-analysis tasks. The examples used during screening (e.g., object detection, security, and surveillance) were illustrative of common application domains within the broader field, not exclusionary criteria. The resulting diversity of included applications—from action recognition to frame interpolation—reflects the breadth of the eligibility criteria and the methodological focus of the review.

The sample of 31 studies included in this review reflects the application of the defined inclusion criteria and the targeted nature of the research questions. While the field of CNN-based video processing is extensive, the requirement that studies explicitly address all three core concepts—convolutional neural networks, video processing, and feature extraction—necessarily limits the sample size. This methodological focus was intentionally chosen to ensure relevance and to avoid overwhelming the synthesis with marginally related works. Researchers should consider this scope delimitation when interpreting the findings and applying them to their specific contexts. The sample, though limited in number, provides sufficient breadth across architectural categories and application domains to support the conclusions drawn in this review.

The taxonomy presented in this review, while useful for identifying predominant architectural trends, necessarily involves some degree of categorical overlap. This reflects the hybrid nature of many contemporary deep learning architectures, which increasingly combine multiple paradigms. The classification of studies into the four groups (2D-CNN, 3D-CNN, hybrid models, and lightweight architectures) was based on each study’s primary architectural contribution, as stated by the authors. Researchers should be aware that some studies could reasonably belong to more than one category, and the taxonomy should be interpreted as a practical framework for summarizing trends rather than as a rigid mathematical classification.

The review also revealed that a considerable number of works do not report the evaluation metrics used to validate their results in sufficient detail. This situation highlights certain limitations in the quantitative robustness of some proposals and reinforces the need to move towards more standardized evaluation protocols.

Finally, it is important to consider the potential bias associated with studies whose experiments are difficult to replicate or whose experimental conditions are insufficiently described. This lack of information limits the reproducibility of the results and can lead to an overestimation of the performance reported for different models in the literature. Despite these limitations, this systematic review provides a solid and useful synthesis of the current state of research on CNN-based spatiotemporal feature extraction for video processing, constituting a benchmark for future work in this area.

Based on the analysis and discussion above, the following section summarizes the main conclusions of this systematic review and outlines directions for future work.

6. Conclusions

This systematic study on spatiotemporal feature extraction using CNNs in video processing identifies four main findings:

The first relates to the classification of architectures, showing the consolidation of four families—2D-CNN, 3D-CNN, hybrid models (CNN-RNN, CNN–Transformer, and two-stream), and lightweight architectures—offering a unified framework compared to previous fragmented reviews. The second focuses on the analysis of evaluation methods and metrics used in the studies. Heterogeneity in protocols and metrics (accuracy, F1-score, mAP, SSIM, etc.) is confirmed, hindering direct comparisons. The need to standardize evaluation criteria is emphasized.

The third refers to the use of datasets such as UCF101 and HMDB51, which are the most frequent. However, the dependence on private datasets persists for specific tasks, limiting the reproducibility of the proposed approaches. The fourth section focuses on emerging trends and challenges, highlighting the shift towards hybrid and transformer-based models, as well as lightweight architectures for edge devices. Challenges include computational efficiency, scalability to long videos, and ethical considerations.

In summary, this work provides a framework that helps researchers and practitioners select appropriate architectures, metrics, and datasets while also pointing to avenues for future research: protocol standardization, multimodal integration, and optimization for resource-constrained environments. Limitations include the fact that the analysis is restricted to 31 articles published between 2020 and 2025 and to three databases (Scopus, Web of Science, and MDPI), which may introduce selection bias.

Finally, future work suggests exploring machine learning approaches, standardized benchmarks, and the integration of additional modalities (audio and text) to enrich spatiotemporal feature extraction. It is also necessary to broaden the search to include academic databases and patents, since the study of innovative approaches for spatiotemporal feature extraction is still emerging, with potential in surveillance, emergencies, and autonomous systems.